docs: design for 2023/24 KS4/KS2 data and DfE LA averages (C2, H2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/superpowers/specs/2026-10-06-ks4-2023-24-and-la-averages-design.md

@@ -0,0 +1,300 @@
+# 2023/24 Results and DfE LA Averages — Design
+
+**Date:** 2026-10-06
+**Status:** approved design, not yet implemented
+**Scope:** `tap_uk_ees` (KS4 results, KS4 information, new LA stream),
+`safe_numeric`, new `fact_ks4_la_averages` mart, annual EES DAG selector,
+`/api/la-averages`, secondary search rows and map cards
+**Fixes:** audit findings C2 and H2, from the 3 Oct 2026 accuracy audit
+
+## Goal
+
+Show every 2023/24 result and school-information figure DfE published, and
+compare each state-funded secondary school with DfE's own local-authority
+average.
+
+## The problem
+
+### C2: 2023/24 is empty
+
+Bishop Stopford School (137086) shows Attainment 8 60.7 for 2022/23, nothing
+for 2023/24 and 58.7 for 2024/25. DfE's 2023/24 figures are Attainment 8 64.1,
+Progress 8 +1.02 and English and maths grade 4+ 91.7%. 2023/24 is the last year
+DfE published Progress 8 (2024/25 has no KS2 baseline), so the site shows no
+recent Progress 8 for any school. Site-wide, 4,170 listed schools have a DfE
+2023/24 Attainment 8 and 3,384 a Progress 8.
+
+There are three separate causes.
+
+1. **KS4 results.** The 2024/25 release's
+   `202425_performance_tables_schools_final.csv` is a time series: it holds
+   2022/23, 2023/24 and 2024/25 under the current column names, with the right
+   values (Bishop Stopford 2023/24: 64.1, 1.02, 91.7). The 2023/24 release's own
+   `202324_performance_tables_schools_final.csv`, re-issued on 10 March 2026,
+   uses the older names (`t_pupils`, `avg_att8`, `avg_p8score`,
+   `pt_l2basics_94` …). `EESDatasetStream` reads releases in the API's order,
+   newest first, so the old file is read last. Its rows carry none of the
+   declared fields, and target-postgres upserts on the stream's primary key
+   (`append_only = not key_properties` in meltanolabs-target-postgres 0.8.0),
+   so they overwrite the good 2023/24 rows with nulls.
+
+   The two files share the keys of every "Total" row, but 34,254 of the 57,090
+   2023/24 sub-group rows use different labels ("Low prior" against "Low prior
+   attainment"). Those old-label rows sit in `raw.ees_ks4_performance` with
+   null measures. Nothing reads them.
+
+2. **KS4 school information.** 2023/24 information exists only in the 2023/24
+   release, in `202324_information_about_schools_final.csv`, with the older
+   names. `EESKS4InfoStream` declares the newer ones, so prior attainment,
+   SEN percentages, disadvantage gaps and Progress 8 banding are null for
+   2023/24.
+
+3. **KS2 school information.** The 2023/24 file
+   `ks2_school_information_data.csv` uses the declared names, and pupil counts
+   load (school 147411: 818 pupils, 112 eligible). Its percentages are written
+   with a sign (`ptfsm6cla1a = "34%"`). `safe_numeric` accepts only
+   `^-?[0-9]+(\.[0-9]+)?$`, so disadvantaged, EAL, SEN and mobility percentages
+   are null for every school in 2023/24.
+
+DfE published no school-level KS2 information file for 2022/23 (the 2023/24
+release's attainment file carries 2022/23 attainment rows only). Those nulls are
+a gap in the source, not a defect.
+
+### H2: "vs LA avg" uses the wrong average
+
+`/api/la-averages` (`backend/app.py`) takes an unweighted mean of every school
+in the LA with an Attainment 8 score, independent and special schools included.
+Independent schools score low because DfE measures exclude IGCSEs, and special
+schools score low for other reasons, so the average is too low almost
+everywhere. Kensington and Chelsea's "LA avg" is 35.2: the mean of 6 state
+schools (54.9) and 8 independent schools (20.4). DfE's figure is 54.5. Of the
+151 LAs the audit compared, ours was lower in 147, by 7.1 points on average and
+by up to 19.3, so most secondary schools look better than their area.
+
+DfE's LA averages are already in the file the pipeline downloads for the
+England averages: the "summary, all state-funded" data set
+(`data-catalogue/data-set/1b649e16-01e8-435b-a814-56be2faf9054/csv`). Its
+`Local authority` / `All state-funded` / Total rows match DfE's published
+performance-table LA averages (RECTYPE 4) for all 152 LAs in 2024/25, with no
+difference. `EESKs4NationalStream` keeps the England row and discards them.
+"All state-funded" is the same population as the England benchmark the site
+already shows.
+
+Computing the average ourselves from state-funded schools, weighted by pupils,
+was tested and rejected: it differs from DfE's figure by 1.1 points on average
+and is never exact.
+
+## Non-goals
+
+- 2022/23 KS2 school information (DfE published none).
+- An "excludes IGCSEs" note wherever an independent school's Attainment 8
+  appears. This change only stops comparing independent schools with the LA.
+- Showing 2023/24 Progress 8 in the GCSE section's headline. The history section
+  shows it once the data loads; the 2024/25 banner stays true.
+- Updating the fixed EES data-set id when DfE publishes 2025/26. It already
+  feeds the England averages; the LA stream shares it.
+- LA comparisons for other measures or on other pages.
+- Deleting the leftover old-label raw rows automatically.
+
+## The rules
+
+### The newest release owns every year it contains (KS4 results only)
+
+`EESDatasetStream` gets an opt-in class attribute,
+`_newest_release_owns_period: bool = False`. `EESKS4PerformanceStream` sets it
+to `True`. When it is on:
+
+- releases are processed newest first by `time_period` (from the release slug),
+  not in the API's order;
+- the stream records each `time_period` it has emitted;
+- in each older release, rows whose `time_period` a newer release already
+  emitted are dropped, and the stream logs how many it skipped and for which
+  years;
+- years only an older release contains are emitted as before.
+
+The filter is a pure function, testable without a download. Other streams keep
+the current behaviour. A general rule would be wrong: the 2024/25 KS2 file holds
+98,448 of the 955,956 rows the 2023/24 release has for 2023/24.
+
+### KS4 information: old names
+
+`EESKS4InfoStream._column_renames` maps the 2023/24 names onto the declared
+fields:
+
+| 2023/24 column | Declared field |
+|---|---|
+| `t_allks_pupils` | `allks_pupil_count` |
+| `t_allks_boys` | `allks_boys_count` |
+| `t_allks_girls` | `allks_girls_count` |
+| `t_pupils` | `endks4_pupil_count` |
+| `avg_ks2_scaledscore` | `ks2_scaledscore_average` |
+| `pt_sen_with_ehcp` | `sen_with_ehcp_pupil_percent` |
+| `pt_sen` | `sen_pupil_percent` |
+| `pt_sen_no_ehcp` | `sen_no_ehcp_pupil_percent` |
+| `diffn_att8` | `attainment8_diffn` |
+| `diffn_p8mea` | `progress8_diffn` |
+| `p8_banding` | `progress8_banding` |
+
+Newer files contain none of the old names, so they are unaffected.
+
+### `safe_numeric` accepts a trailing `%`
+
+The pattern becomes `^-?[0-9]+(\.[0-9]+)?%?$` and the cast reads
+`rtrim(col, '%')`. A value such as `34%` can only mean 34. Suppression codes
+(`c`, `z`, `x` …) still become null.
+
+### LA averages
+
+A new stream, `ees_ks4_la`, reads the same CSV as `ees_ks4_national` and keeps
+rows where `geographic_level = 'Local authority'`,
+`establishment_type_group = 'All state-funded'`, `breakdown_topic = 'Total'` and
+`breakdown = 'Total'` (case-insensitive, as the national stream compares). It
+emits `time_period`, `old_la_code`, `new_la_code`, `la_name` and the 8 headline
+measures the national stream emits (`_KS4_NATIONAL_COL_MAP`). Primary key:
+(`time_period`, `old_la_code`). The two streams share one download-and-filter
+helper.
+
+`old_la_code` is the GIAS LA code (`local_authority_code`), so schools join on
+the code, not the name. Names match today for every LA DfE publishes; DfE
+publishes no figure for City of London.
+
+### Which schools get a gap
+
+The search row and the map card show "vs LA avg" only when the school has an
+Attainment 8 score, is neither special (`isSpecialSchool`) nor independent
+(`isIndependentSchool`: "independent" in the GIAS type, which covers "Other
+independent school" and "Other independent special school"), and its LA has a
+DfE figure for the year the endpoint serves.
+
+## Delivery
+
+Two PRs, as for C1.
+
+### PR 1: pipeline
+
+- `pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py`: release
+  precedence, KS4 information renames, `ees_ks4_la` stream registered in
+  `discover_streams`, shared national/LA helper.
+- `pipeline/transform/macros/safe_numeric.sql`: trailing `%`.
+- `pipeline/transform/models/staging/`: source `raw.ees_ks4_la` and
+  `stg_ees_ks4_la` (view): `cast(old_la_code as integer) as la_code`,
+  `cast(time_period as integer) as year`, `la_name`, measures via
+  `safe_numeric`, named as in `stg_ees_ks4_national`.
+- `pipeline/transform/models/marts/fact_ks4_la_averages.sql` (table): one row
+  per (`year`, `la_code`) with `la_name` and the columns of
+  `fact_ks4_national_averages`. Schema tests: unique (`year`, `la_code`);
+  `year`, `la_code` and `la_name` not null.
+- Data tests in `pipeline/transform/tests/`:
+  - `assert_ks4_years_have_results`: every year in `stg_ees_ks4` has a non-null
+    Attainment 8 for at least 50% of its rows. DfE's files reach 82% each year;
+    2023/24 loads at 0% today. Pre-2019 years come from the legacy model and are
+    not tested.
+  - `assert_ks2_info_percentages_loaded`: for every year in `stg_ees_ks2` where
+    at least 1,000 rows have `total_pupils`, at least 90% of those rows have
+    `disadvantaged_pct`. DfE's files reach 96–97%. 2022/23 has no pupil counts
+    and is skipped.
+  - `assert_ks4_la_averages_cover_las`: the latest year in
+    `fact_ks4_la_averages` has at least 145 LAs (DfE: 152).
+- `pipeline/dags/school_data_pipeline.py`: the annual EES build selects
+  `stg_ees_ks4_la+`.
+- `docs/ARCHITECTURE.md`: LA averages come from DfE's data set.
+
+### PR 2: backend and UI (after the EES DAG has run on PR 1)
+
+- `backend/models.py`: `Ks4LaAverage` for `marts.fact_ks4_la_averages`.
+- `backend/app.py` `/api/la-averages`: the year is the latest with any school
+  Attainment 8 (as now). It reads that year's rows from the mart and keys each
+  `attainment_8_score` by our LA name, through the `local_authority_code` →
+  `local_authority` pairs in the school data. The response shape is unchanged.
+  No rows for that year, a missing table or a query error give an empty map,
+  logged, never another year's figures and never a computed mean.
+- `nextjs-app/lib/utils.ts`: `isIndependentSchool(school)`.
+- `nextjs-app/components/SecondarySchoolRow.tsx` and
+  `nextjs-app/components/LeafletMapInner.tsx`: the rule in "Which schools get a
+  gap".
+- `e2e/tests/journeys.spec.ts`: the two journeys under Testing.
+
+## Testing
+
+**Extractor (pytest, `pipeline/tests/`, new):**
+
+- precedence: with the flag on, a year in a newer release is emitted once, from
+  the newer release; a year only an older release has is kept; with the flag
+  off, every row passes;
+- releases arriving oldest first are still processed newest first;
+- KS4 information: an old-format row yields `endks4_pupil_count`,
+  `ks2_scaledscore_average`, `progress8_banding` and the rest of the table;
+- LA filter: a small CSV with national, regional, LA and sub-group rows yields
+  one row per LA and year, with the declared fields.
+
+**dbt (local `pgserver`, as for C1):**
+
+- unit test on `stg_ees_ks2`: `34%` → 34, `34` → 34, `c` → null;
+- unit test on `stg_ees_ks4_la` or the mart: codes and years cast, measures
+  carried;
+- the three data tests and the schema tests above;
+- `pipeline/tests/test_dag_selectors.py` passes with the new selector.
+
+**Backend (pytest):**
+
+- the response gives the mart's figure where the fixture's plain mean differs;
+- matching works by code when the mart's `la_name` differs from ours;
+- a mart without the served year, and a missing table, give an empty map.
+
+**Front end (Jest):**
+
+- `isIndependentSchool` for "Other independent school", "Other independent
+  special school", "Academy converter" and null;
+- `SecondarySchoolRow` and the map card show no gap for an independent school
+  and show one for a state school with an LA figure.
+
+**E2E (`journeys.spec.ts`, PR 2):**
+
+- C2: Bishop Stopford (137086) history shows 2023/24 Attainment 8 64.1 and
+  Progress 8 +1.02. These are final figures, so the journey stays stable.
+- H2: a search returning an independent and a state secondary: the independent
+  row has no "vs LA avg", the state row has one. The exact gap is not asserted:
+  DfE's 2025/26 provisional KS4 data is due and would change it.
+
+## Rollout and verification
+
+1. PR 1 merges to staging. Tudor runs `school_data_annual_ees` on staging.
+2. Through the staging API: Bishop Stopford 2023/24 Attainment 8 64.1 and
+   Progress 8 1.02; school 147411 2023/24 disadvantaged 34; the LA mart has 152
+   LAs for 2024/25. If a data test fails, the build stops before search sync;
+   investigate before going further.
+3. PR 2 merges, so the post-merge E2E gate runs against loaded data.
+4. Production, Tudor's decision, in order: promote PR 1, run the EES DAG on
+   production, promote PR 2. If PR 2 arrives first, the endpoint returns an
+   empty map and rows show no gap, never a wrong one.
+5. Optional cleanup, in the PR 1 description for Tudor:
+   `delete from raw.ees_ks4_performance where time_period = '202324' and pupil_count is null`.
+   New-format rows always have `pupil_count` (suppressed values are `c` or `z`,
+   not null), so this removes exactly the old-label leftovers.
+6. Validation on production against DfE's files: every listed school's 2023/24
+   Attainment 8, Progress 8 and English and maths figures match; the 2023/24 KS4
+   and KS2 information fields match; `/api/la-averages` equals DfE for all 152
+   LAs; independent rows show no gap. Then mark C2 and H2 resolved in the audit
+   report.
+
+## Expected visible change
+
+- Secondary history charts and tables run unbroken from 2022/23 to 2024/25, and
+  2023/24 Progress 8 appears for about 3,400 schools.
+- 2023/24 school information (KS2 and KS4) fills in.
+- "vs LA avg" drops for most state secondaries. In Kensington and Chelsea, a
+  school with Attainment 8 60.0 moves from +24.8 to +5.5.
+- Independent schools and City of London schools show no LA gap.
+
+## Risks
+
+- **Data-test thresholds stop a good build.** They were set from DfE's own
+  files (82% and 96–97% against 50% and 90%). A failure means the data changed
+  shape, which is what they are for.
+- **`safe_numeric` is shared by 14 models.** Values written as `n%` were null
+  and become numbers. No DfE column uses `%` for anything but a percentage.
+  The staging EES run is the check.
+- **Precedence drops data an older release holds more completely.** It is
+  opt-in for KS4 results, where both files hold the same 57,090 2023/24 keys.
+- **The fixed EES data-set id goes stale** when 2025/26 is published. The
+  endpoint's year check then gives no gap rather than a mismatched one.

pipeline/dags/school_data_pipeline.py

@@ -106,9 +106,12 @@ print(f'Validation passed: {{count}} GIAS rows')
         """,
     )
 
+    # Marts fed by annual EES staging models are rebuilt by the EES DAG, even
+    # when they join dim_school. Selecting them here fails in any database
+    # where that DAG hasn't run (pipeline/tests/test_dag_selectors.py).
     dbt_build = BashOperator(
         task_id="dbt_build",
-        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_gias_establishments+ stg_gias_links+ gias_code_names+ --exclude int_ks2_with_lineage+ int_ks4_with_lineage+",
+        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_gias_establishments+ stg_gias_links+ gias_code_names+ --exclude int_ks2_with_lineage+ int_ks4_with_lineage+ stg_ees_ks4_destinations+ stg_ees_ks5_destinations+",
     )
 
     sync_typesense = BashOperator(
@@ -143,7 +146,7 @@ with DAG(
 
     dbt_build_ofsted = BashOperator(
         task_id="dbt_build",
-        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ofsted_inspections+ int_ofsted_latest+ fact_ofsted_inspection+ dim_school+",
+        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ofsted_inspections+ int_ofsted_latest+ fact_ofsted_inspection+ dim_school+ --exclude stg_ees_ks4_destinations+ stg_ees_ks5_destinations+",
     )
 
     sync_typesense_ofsted = BashOperator(

pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/gias_csv.py

@@ -0,0 +1,26 @@
+"""Read a GIAS extract from the raw bytes of the download.
+
+GIAS writes its CSVs in Windows-1252 and sends no charset, so `resp.text`
+leaves requests to guess the codec. On 3 Oct 2026 it guessed windows-1250 and
+"à" became "ŕ". Decode the bytes ourselves instead.
+"""
+
+from __future__ import annotations
+
+import io
+
+import pandas as pd
+
+GIAS_ENCODING = "cp1252"
+
+
+def read_gias_csv(content: bytes, logger=None) -> pd.DataFrame:
+    """Every column as a string; a blank cell stays ''."""
+    # Windows-1252 leaves five bytes undefined. One stray byte must not stop
+    # the daily refresh of every school, so it becomes U+FFFD and is logged.
+    text = content.decode(GIAS_ENCODING, errors="replace")
+    undecodable = text.count("�")
+    if undecodable and logger is not None:
+        logger.warning("%d byte(s) in the GIAS extract could not be decoded as %s",
+                       undecodable, GIAS_ENCODING)
+    return pd.read_csv(io.StringIO(text), dtype=str, keep_default_na=False)

pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py

@@ -7,6 +7,8 @@ from datetime import date, timedelta
 from singer_sdk import Stream, Tap
 from singer_sdk import typing as th
 
+from tap_uk_gias.gias_csv import read_gias_csv
+
 GIAS_URL_TEMPLATE = (
     "https://ea-edubase-api-prod.azurewebsites.net"
     "/edubase/downloads/public/edubasealldata{date}.csv"
@@ -74,9 +76,6 @@ class GIASEstablishmentsStream(Stream):
 
     def get_records(self, context):
         """Download GIAS CSV and yield rows."""
-        import io
-
-        import pandas as pd
         import requests
 
         today = date.today()
@@ -94,12 +93,7 @@ class GIASEstablishmentsStream(Stream):
 
         resp.raise_for_status()
 
-        df = pd.read_csv(
-            io.StringIO(resp.text),
-            encoding="latin-1",
-            dtype=str,
-            keep_default_na=False,
-        )
+        df = read_gias_csv(resp.content, self.logger)
 
         for _, row in df.iterrows():
             record = row.to_dict()
@@ -126,9 +120,6 @@ class GIASLinksStream(Stream):
 
     def get_records(self, context):
         """Download GIAS links CSV and yield rows."""
-        import io
-
-        import pandas as pd
         import requests
 
         today = date.today()
@@ -146,12 +137,7 @@ class GIASLinksStream(Stream):
 
         resp.raise_for_status()
 
-        df = pd.read_csv(
-            io.StringIO(resp.text),
-            encoding="latin-1",
-            dtype=str,
-            keep_default_na=False,
-        )
+        df = read_gias_csv(resp.content, self.logger)
 
         for _, row in df.iterrows():
             record = row.to_dict()

pipeline/tests/test_dag_selectors.py

@@ -0,0 +1,98 @@
+"""Every scheduled dbt build must only build models whose parents exist.
+
+The daily GIAS build selects `stg_gias_establishments+`, so any mart that joins
+dim_school joins the daily build too. When such a mart also reads a staging
+model that only the manually triggered EES DAG builds, the daily build fails in
+any database where that DAG has not run since. Sync and cache invalidation then
+never run either. The destinations marts did this from late August 2026.
+
+The graph is read from the model SQL, because CI has no dbt.
+"""
+import re
+from collections import defaultdict
+from pathlib import Path
+
+import pytest
+
+PIPELINE = Path(__file__).resolve().parents[1]
+MODELS = PIPELINE / 'transform' / 'models'
+DAG_FILE = PIPELINE / 'dags' / 'school_data_pipeline.py'
+
+REF = re.compile(r"ref\(\s*'([a-z0-9_]+)'\s*\)")
+DBT_BUILD = re.compile(r'dbt_build\w*\s*=\s*BashOperator\(.*?build --profiles-dir \. --target production ([^"]+)"', re.S)
+DAG_ID = re.compile(r'dag_id="([a-z0-9_]+)"')
+
+DAILY = 'school_data_daily'
+
+# dim_school reads int_ofsted_latest only when the relation exists
+# (adapter.get_relation), so a missing table is not a failure.
+OPTIONAL_PARENTS = {'int_ofsted_latest'}
+
+
+def model_parents():
+    """{model: models it refs}. Seeds are left out: they are loaded once and always exist."""
+    sql = {p.stem: p.read_text() for p in MODELS.rglob('*.sql')}
+    return {name: set(REF.findall(text)) & set(sql) for name, text in sql.items()}
+
+
+def downstream(node, children):
+    seen, stack = {node}, [node]
+    while stack:
+        for child in children[stack.pop()]:
+            if child not in seen:
+                seen.add(child)
+                stack.append(child)
+    return seen
+
+
+def expand(tokens, children):
+    out = set()
+    for token in tokens:
+        out |= downstream(token[:-1], children) if token.endswith('+') else {token}
+    return out
+
+
+def scheduled_builds():
+    """{dag_id: dbt selection arguments} for every dbt build in the DAG file."""
+    text = DAG_FILE.read_text()
+    starts = [(m.start(), m.group(1)) for m in DAG_ID.finditer(text)]
+    builds = {}
+    for i, (start, dag_id) in enumerate(starts):
+        end = starts[i + 1][0] if i + 1 < len(starts) else len(text)
+        found = DBT_BUILD.search(text, start, end)
+        if found:
+            builds[dag_id] = found.group(1)
+    return builds
+
+
+def selected_models(args, parents):
+    children = defaultdict(set)
+    for model, ps in parents.items():
+        for p in ps:
+            children[p].add(model)
+    select = re.search(r'--select (.+?)(?= --exclude|$)', args).group(1).split()
+    excluded = re.search(r'--exclude (.+)$', args)
+    exclude = excluded.group(1).split() if excluded else []
+    return (expand(select, children) - expand(exclude, children)) & set(parents)
+
+
+PARENTS = model_parents()
+BUILDS = scheduled_builds()
+DAILY_MODELS = selected_models(BUILDS[DAILY], PARENTS)
+
+
+def test_every_dag_with_a_dbt_build_is_parsed():
+    assert set(BUILDS) == {
+        'school_data_daily', 'school_data_monthly_ofsted', 'school_data_annual_ees',
+        'school_data_annual_idaci', 'school_data_annual_distance',
+    }
+
+
+@pytest.mark.parametrize('dag_id', sorted(BUILDS))
+def test_selected_models_only_read_models_that_exist(dag_id):
+    selected = selected_models(BUILDS[dag_id], PARENTS)
+    # The daily build is the base layer: other DAGs may rely on what it builds.
+    available = selected | OPTIONAL_PARENTS | (DAILY_MODELS if dag_id != DAILY else set())
+    missing = {model: sorted(PARENTS[model] - available) for model in sorted(selected)
+               if PARENTS[model] - available}
+    assert missing == {}, f'{dag_id} builds models whose parents it never builds: {missing}'

pipeline/tests/test_gias_encoding.py

@@ -0,0 +1,66 @@
+"""GIAS publishes its extracts in Windows-1252 and declares no charset.
+
+The tap used to hand pandas `resp.text`, so requests guessed the codec.
+On 3 Oct 2026 it guessed windows-1250, and "St Thomas à Becket" was stored
+as "St Thomas ŕ Becket". The `encoding=` passed to read_csv did nothing,
+because the text was already decoded.
+"""
+import importlib.util
+import logging
+from pathlib import Path
+
+import pytest
+
+MODULE = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-gias'
+          / 'tap_uk_gias' / 'gias_csv.py')
+
+
+@pytest.fixture
+def gias_csv():
+    spec = importlib.util.spec_from_file_location('gias_csv', MODULE)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+# Byte for byte as GIAS writes it: 0xE0 à, 0x92 ’, 0xE9 é, 0xB0 °, 0xE7 ç.
+EXTRACT = (
+    b'"URN","EstablishmentName","HeadLastName"\r\n'
+    b'"138950","St Thomas \xe0 Becket Catholic Secondary School","Smith"\r\n'
+    b'"100000","The Dean and Chapter of St Paul\x92s Cathedral","Pr\xe9vert"\r\n'
+    b'"140677","North Star 180\xb0","Fran\xe7ois"\r\n'
+    b'"100001","No head recorded",""\r\n'
+)
+
+
+def test_names_decode_as_windows_1252(gias_csv):
+    df = gias_csv.read_gias_csv(EXTRACT)
+    assert list(df['EstablishmentName']) == [
+        'St Thomas à Becket Catholic Secondary School',
+        'The Dean and Chapter of St Paul’s Cathedral',
+        'North Star 180°',
+        'No head recorded',
+    ]
+    assert list(df['HeadLastName']) == ['Smith', 'Prévert', 'François', '']
+
+
+def test_the_codec_requests_guessed_is_not_used(gias_csv):
+    # What the tap stored on 3 Oct: the same bytes read as windows-1250.
+    assert 'ŕ' in EXTRACT.decode('cp1250')
+    names = ' '.join(gias_csv.read_gias_csv(EXTRACT)['EstablishmentName'])
+    assert 'ŕ' not in names
+
+
+def test_values_stay_strings(gias_csv):
+    df = gias_csv.read_gias_csv(EXTRACT)
+    assert df.loc[0, 'URN'] == '138950'
+
+
+def test_a_byte_windows_1252_leaves_undefined_does_not_stop_the_load(gias_csv, caplog):
+    # 0x81 has no Windows-1252 character. One odd name must not block the daily
+    # refresh of every school, but it must be visible in the log.
+    extract = b'"URN","EstablishmentName"\r\n"100002","Odd \x81 Name"\r\n'
+    with caplog.at_level(logging.WARNING):
+        df = gias_csv.read_gias_csv(extract, logger=logging.getLogger('gias'))
+    assert df.loc[0, 'EstablishmentName'] == 'Odd � Name'
+    assert 'could not be decoded' in caplog.text