# 2023/24 Results and DfE LA Averages — Design **Date:** 2026-10-06 **Status:** approved design, not yet implemented **Scope:** `tap_uk_ees` (KS4 results, KS4 information, new LA stream), `safe_numeric`, new `fact_ks4_la_averages` mart, annual EES DAG selector, `/api/la-averages`, secondary search rows and map cards **Fixes:** audit findings C2 and H2, from the 3 Oct 2026 accuracy audit ## Goal Show every 2023/24 result and school-information figure DfE published, and compare each state-funded secondary school with DfE's own local-authority average. ## The problem ### C2: 2023/24 is empty Bishop Stopford School (137086) shows Attainment 8 60.7 for 2022/23, nothing for 2023/24 and 58.7 for 2024/25. DfE's 2023/24 figures are Attainment 8 64.1, Progress 8 +1.02 and English and maths grade 4+ 91.7%. 2023/24 is the last year DfE published Progress 8 (2024/25 has no KS2 baseline), so the site shows no recent Progress 8 for any school. Site-wide, 4,170 listed schools have a DfE 2023/24 Attainment 8 and 3,384 a Progress 8. There are three separate causes. 1. **KS4 results.** The 2024/25 release's `202425_performance_tables_schools_final.csv` is a time series: it holds 2022/23, 2023/24 and 2024/25 under the current column names, with the right values (Bishop Stopford 2023/24: 64.1, 1.02, 91.7). The 2023/24 release's own `202324_performance_tables_schools_final.csv`, re-issued on 10 March 2026, uses the older names (`t_pupils`, `avg_att8`, `avg_p8score`, `pt_l2basics_94` …). `EESDatasetStream` reads releases in the API's order, newest first, so the old file is read last. Its rows carry none of the declared fields, and target-postgres upserts on the stream's primary key (`append_only = not key_properties` in meltanolabs-target-postgres 0.8.0), so they overwrite the good 2023/24 rows with nulls. The two files share the keys of every "Total" row, but 34,254 of the 57,090 2023/24 sub-group rows use different labels ("Low prior" against "Low prior attainment"). Those old-label rows sit in `raw.ees_ks4_performance` with null measures. Nothing reads them. 2. **KS4 school information.** 2023/24 information exists only in the 2023/24 release, in `202324_information_about_schools_final.csv`, with the older names. `EESKS4InfoStream` declares the newer ones, so prior attainment, SEN percentages, disadvantage gaps and Progress 8 banding are null for 2023/24. 3. **KS2 school information.** The 2023/24 file `ks2_school_information_data.csv` uses the declared names, and pupil counts load (school 147411: 818 pupils, 112 eligible). Its percentages are written with a sign (`ptfsm6cla1a = "34%"`). `safe_numeric` accepts only `^-?[0-9]+(\.[0-9]+)?$`, so disadvantaged, EAL, SEN and mobility percentages are null for every school in 2023/24. DfE published no school-level KS2 information file for 2022/23 (the 2023/24 release's attainment file carries 2022/23 attainment rows only). Those nulls are a gap in the source, not a defect. ### H2: "vs LA avg" uses the wrong average `/api/la-averages` (`backend/app.py`) takes an unweighted mean of every school in the LA with an Attainment 8 score, independent and special schools included. Independent schools score low because DfE measures exclude IGCSEs, and special schools score low for other reasons, so the average is too low almost everywhere. Kensington and Chelsea's "LA avg" is 35.2: the mean of 6 state schools (54.9) and 8 independent schools (20.4). DfE's figure is 54.5. Of the 151 LAs the audit compared, ours was lower in 147, by 7.1 points on average and by up to 19.3, so most secondary schools look better than their area. DfE's LA averages are already in the file the pipeline downloads for the England averages: the "summary, all state-funded" data set (`data-catalogue/data-set/1b649e16-01e8-435b-a814-56be2faf9054/csv`). Its `Local authority` / `All state-funded` / Total rows match DfE's published performance-table LA averages (RECTYPE 4) for all 152 LAs in 2024/25, with no difference. `EESKs4NationalStream` keeps the England row and discards them. "All state-funded" is the same population as the England benchmark the site already shows. Computing the average ourselves from state-funded schools, weighted by pupils, was tested and rejected: it differs from DfE's figure by 1.1 points on average and is never exact. ## Non-goals - 2022/23 KS2 school information (DfE published none). - An "excludes IGCSEs" note wherever an independent school's Attainment 8 appears. This change only stops comparing independent schools with the LA. - Showing 2023/24 Progress 8 in the GCSE section's headline. The history section shows it once the data loads; the 2024/25 banner stays true. - Updating the fixed EES data-set id when DfE publishes 2025/26. It already feeds the England averages; the LA stream shares it. - LA comparisons for other measures or on other pages. - Deleting the leftover old-label raw rows automatically. ## The rules ### The newest release owns every year it contains (KS4 results only) `EESDatasetStream` gets an opt-in class attribute, `_newest_release_owns_period: bool = False`. `EESKS4PerformanceStream` sets it to `True`. When it is on: - releases are processed newest first by `time_period` (from the release slug), not in the API's order; - the stream records each `time_period` it has emitted; - in each older release, rows whose `time_period` a newer release already emitted are dropped, and the stream logs how many it skipped and for which years; - years only an older release contains are emitted as before. The filter is a pure function, testable without a download. Other streams keep the current behaviour. A general rule would be wrong: the 2024/25 KS2 file holds 98,448 of the 955,956 rows the 2023/24 release has for 2023/24. ### KS4 information: old names `EESKS4InfoStream._column_renames` maps the 2023/24 names onto the declared fields: | 2023/24 column | Declared field | |---|---| | `t_allks_pupils` | `allks_pupil_count` | | `t_allks_boys` | `allks_boys_count` | | `t_allks_girls` | `allks_girls_count` | | `t_pupils` | `endks4_pupil_count` | | `avg_ks2_scaledscore` | `ks2_scaledscore_average` | | `pt_sen_with_ehcp` | `sen_with_ehcp_pupil_percent` | | `pt_sen` | `sen_pupil_percent` | | `pt_sen_no_ehcp` | `sen_no_ehcp_pupil_percent` | | `diffn_att8` | `attainment8_diffn` | | `diffn_p8mea` | `progress8_diffn` | | `p8_banding` | `progress8_banding` | Newer files contain none of the old names, so they are unaffected. ### `safe_numeric` accepts a trailing `%` The pattern becomes `^-?[0-9]+(\.[0-9]+)?%?$` and the cast reads `rtrim(col, '%')`. A value such as `34%` can only mean 34. Suppression codes (`c`, `z`, `x` …) still become null. ### LA averages A new stream, `ees_ks4_la`, reads the same CSV as `ees_ks4_national` and keeps rows where `geographic_level = 'Local authority'`, `establishment_type_group = 'All state-funded'`, `breakdown_topic = 'Total'` and `breakdown = 'Total'` (case-insensitive, as the national stream compares). It emits `time_period`, `old_la_code`, `new_la_code`, `la_name` and the 8 headline measures the national stream emits (`_KS4_NATIONAL_COL_MAP`). Primary key: (`time_period`, `old_la_code`). The two streams share one download-and-filter helper. `old_la_code` is the GIAS LA code (`local_authority_code`), so schools join on the code, not the name. Names match today for every LA DfE publishes; DfE publishes no figure for City of London. ### Which schools get a gap The search row and the map card show "vs LA avg" only when the school has an Attainment 8 score, is neither special (`isSpecialSchool`) nor independent (`isIndependentSchool`: "independent" in the GIAS type, which covers "Other independent school" and "Other independent special school"), and its LA has a DfE figure for the year the endpoint serves. ## Delivery Two PRs, as for C1. ### PR 1: pipeline - `pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py`: release precedence, KS4 information renames, `ees_ks4_la` stream registered in `discover_streams`, shared national/LA helper. - `pipeline/transform/macros/safe_numeric.sql`: trailing `%`. - `pipeline/transform/models/staging/`: source `raw.ees_ks4_la` and `stg_ees_ks4_la` (view): `cast(old_la_code as integer) as la_code`, `cast(time_period as integer) as year`, `la_name`, measures via `safe_numeric`, named as in `stg_ees_ks4_national`. - `pipeline/transform/models/marts/fact_ks4_la_averages.sql` (table): one row per (`year`, `la_code`) with `la_name` and the columns of `fact_ks4_national_averages`. Schema tests: unique (`year`, `la_code`); `year`, `la_code` and `la_name` not null. - Data tests in `pipeline/transform/tests/`: - `assert_ks4_years_have_results`: every year in `stg_ees_ks4` has a non-null Attainment 8 for at least 50% of its rows. DfE's files reach 82% each year; 2023/24 loads at 0% today. Pre-2019 years come from the legacy model and are not tested. - `assert_ks2_info_percentages_loaded`: for every year in `stg_ees_ks2` where at least 1,000 rows have `total_pupils`, at least 90% of those rows have `disadvantaged_pct`. DfE's files reach 96–97%. 2022/23 has no pupil counts and is skipped. - `assert_ks4_la_averages_cover_las`: the latest year in `fact_ks4_la_averages` has at least 145 LAs (DfE: 152). - `pipeline/dags/school_data_pipeline.py`: the annual EES build selects `stg_ees_ks4_la+`. - `docs/ARCHITECTURE.md`: LA averages come from DfE's data set. ### PR 2: backend and UI (after the EES DAG has run on PR 1) - `backend/models.py`: `Ks4LaAverage` for `marts.fact_ks4_la_averages`. - `backend/app.py` `/api/la-averages`: the year is the latest with any school Attainment 8 (as now). It reads that year's rows from the mart and keys each `attainment_8_score` by our LA name, through the `local_authority_code` → `local_authority` pairs in the school data. The response shape is unchanged. No rows for that year, a missing table or a query error give an empty map, logged, never another year's figures and never a computed mean. - `nextjs-app/lib/utils.ts`: `isIndependentSchool(school)`. - `nextjs-app/components/SecondarySchoolRow.tsx` and `nextjs-app/components/LeafletMapInner.tsx`: the rule in "Which schools get a gap". - `e2e/tests/journeys.spec.ts`: the two journeys under Testing. ## Testing **Extractor (pytest, `pipeline/tests/`, new):** - precedence: with the flag on, a year in a newer release is emitted once, from the newer release; a year only an older release has is kept; with the flag off, every row passes; - releases arriving oldest first are still processed newest first; - KS4 information: an old-format row yields `endks4_pupil_count`, `ks2_scaledscore_average`, `progress8_banding` and the rest of the table; - LA filter: a small CSV with national, regional, LA and sub-group rows yields one row per LA and year, with the declared fields. **dbt (local `pgserver`, as for C1):** - unit test on `stg_ees_ks2`: `34%` → 34, `34` → 34, `c` → null; - unit test on `stg_ees_ks4_la` or the mart: codes and years cast, measures carried; - the three data tests and the schema tests above; - `pipeline/tests/test_dag_selectors.py` passes with the new selector. **Backend (pytest):** - the response gives the mart's figure where the fixture's plain mean differs; - matching works by code when the mart's `la_name` differs from ours; - a mart without the served year, and a missing table, give an empty map. **Front end (Jest):** - `isIndependentSchool` for "Other independent school", "Other independent special school", "Academy converter" and null; - `SecondarySchoolRow` and the map card show no gap for an independent school and show one for a state school with an LA figure. **E2E (`journeys.spec.ts`, PR 2):** - C2: Bishop Stopford (137086) history shows 2023/24 Attainment 8 64.1 and Progress 8 +1.02. These are final figures, so the journey stays stable. - H2: a search returning an independent and a state secondary: the independent row has no "vs LA avg", the state row has one. The exact gap is not asserted: DfE's 2025/26 provisional KS4 data is due and would change it. ## Rollout and verification 1. PR 1 merges to staging. Tudor runs `school_data_annual_ees` on staging. 2. Through the staging API: Bishop Stopford 2023/24 Attainment 8 64.1 and Progress 8 1.02; school 147411 2023/24 disadvantaged 34; the LA mart has 152 LAs for 2024/25. If a data test fails, the build stops before search sync; investigate before going further. 3. PR 2 merges, so the post-merge E2E gate runs against loaded data. 4. Production, Tudor's decision, in order: promote PR 1, run the EES DAG on production, promote PR 2. If PR 2 arrives first, the endpoint returns an empty map and rows show no gap, never a wrong one. 5. Optional cleanup, in the PR 1 description for Tudor: `delete from raw.ees_ks4_performance where time_period = '202324' and pupil_count is null`. New-format rows always have `pupil_count` (suppressed values are `c` or `z`, not null), so this removes exactly the old-label leftovers. 6. Validation on production against DfE's files: every listed school's 2023/24 Attainment 8, Progress 8 and English and maths figures match; the 2023/24 KS4 and KS2 information fields match; `/api/la-averages` equals DfE for all 152 LAs; independent rows show no gap. Then mark C2 and H2 resolved in the audit report. ## Expected visible change - Secondary history charts and tables run unbroken from 2022/23 to 2024/25, and 2023/24 Progress 8 appears for about 3,400 schools. - 2023/24 school information (KS2 and KS4) fills in. - "vs LA avg" drops for most state secondaries. In Kensington and Chelsea, a school with Attainment 8 60.0 moves from +24.8 to +5.5. - Independent schools and City of London schools show no LA gap. ## Risks - **Data-test thresholds stop a good build.** They were set from DfE's own files (82% and 96–97% against 50% and 90%). A failure means the data changed shape, which is what they are for. - **`safe_numeric` is shared by 14 models.** Values written as `n%` were null and become numbers. No DfE column uses `%` for anything but a percentage. The staging EES run is the check. - **Precedence drops data an older release holds more completely.** It is opt-in for KS4 results, where both files hold the same 57,090 2023/24 keys. - **The fixed EES data-set id goes stale** when 2025/26 is published. The endpoint's year check then gives no gap rather than a mismatched one.