docs: H4 rollout order and where stg_gias_ukprn is built (review)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(ofsted): a gov.uk outage skips the FE streams, not the Ofsted extract (review)
2026-10-13 21:55:49 +01:00 · 2026-10-13 21:55:22 +01:00 · 2026-10-13 21:54:51 +01:00 · 2026-10-13 21:28:51 +01:00 · 2026-10-13 21:26:42 +01:00 · 2026-10-13 21:26:05 +01:00
44 changed files with 5987 additions and 135 deletions
@@ -61,6 +61,12 @@ history from the full DataFrame and supplementary data from marts. Comparisons
 batch supplementary queries across selected URNs. Async routes still contain
 synchronous dependency calls; a fully asynchronous database layer is not present.

+DfE's official benchmarks live in marts: `fact_ks2_national_averages` and
+`fact_ks4_national_averages` for England, and `fact_ks4_la_averages` (all
+state-funded schools, per LA) for the search rows' "vs LA avg".
+`data_loader.compute_benchmarks` computes further state-school benchmarks from
+our own dataset; they are not DfE figures and are labelled as computed.
+
 ## Frontend boundaries

 `app/(frontend)` owns the public root layout and pages. `app/(payload)` owns the
@@ -0,0 +1,256 @@
+# Inspection Status Beyond Ofsted's School Data — Design
+
+**Date:** 2026-10-06
+**Status:** approved design, not yet implemented
+**Scope:** `tap_uk_gias` (two fields), `tap_uk_ofsted` (two FE streams), Ofsted
+and GIAS staging, `int_ofsted_latest` / `fact_ofsted_latest`, new
+`int_ofsted_fe_latest` and `int_ofsted_listed`, `dim_school`, monthly Ofsted DAG
+selector, backend list and detail payloads, search badge, map card, school page
+Ofsted section, compare Ofsted rows
+**Fixes:** audit finding H4, from the 3 Oct 2026 accuracy audit
+
+## Goal
+
+Never label a school "Not yet inspected" unless Ofsted's own data lists it
+without an inspection, and show every inspection result we can source, with its
+date.
+
+## The problem
+
+The search badge and the map card (`buildOfstedListBadge`,
+`nextjs-app/lib/utils.ts`) print "Not yet inspected" whenever we hold no Ofsted
+record. The school page hides its Ofsted section for these schools, and compare
+says "No inspection in our dataset". On production on 6 October 2026 (build
+`2b5e681`), 2,587 listed schools have no Ofsted record. Joined with GIAS
+(6 October 2026) and Ofsted's management information (31 August 2026):
+
+| Group | Schools | Evidence |
+|---|---|---|
+| Independent schools inspected by ISI | 1,341 | GIAS `InspectorateName` = "ISI". GIAS holds no date, outcome or report link for any of them; ISI publishes no bulk data. |
+| Further-education and post-16 providers inspected by Ofsted | 519 | In Ofsted's "Further education and skills" MI: 509 by Provider URN, 10 only by UKPRN through GIAS. 207 FE colleges, 129 special post-16 institutions, 97 higher-education institutions (their FE and apprenticeship provision), 78 16–19 academies and free schools, 8 others. Queen Mary's College (145175): Good, 6 December 2022. |
+| Listed by Ofsted, never inspected | 263+ | New academies and free schools present in Ofsted's school MI with every inspection field `NULL`, e.g. Abbas and Templecombe Church School (150031, opened January 2024). "Not yet inspected" is Ofsted's own position for these. |
+| No usable inspection data | about 460 | Secure units (inspected as children's homes), higher-education institutions outside the FE MI, newer independents, schools opened after the MI date. |
+
+Of the 519 FE providers: 443 have only a grade from the previous framework
+(overall effectiveness 1–4 and the inspection's first day); 45 also have a newer
+report card under the renewed framework (from 10 November 2025); 2 have only a
+report card; 29 are listed with no inspection. 116 FE providers nationally have
+a short inspection after their latest full inspection; the MI gives its date
+but not its outcome.
+
+## Non-goals
+
+- Area grades from the previous FE framework, and the report card's
+  programme-by-theme grid. The FE display is the headline and a link.
+- ISI dates or outcomes (no data source).
+- A per-school ISI link: ISI's pages use ISI's own identifiers, not URNs.
+- Inspections of secure units and children's homes (social care framework).
+- Changing how school-framework records are chosen (the C1 rule stands).
+
+## Data
+
+### Extractors
+
+**GIAS (`tap_uk_gias`).** Declare `UKPRN` and `InspectorateName (name)` on the
+establishments stream. They are plain values, not codes, so the generated GIAS
+dictionaries (backend and pipeline copies, parity-tested) are untouched.
+
+**Ofsted (`tap_uk_ofsted`).** Two streams from Ofsted's "Further education and
+skills inspections and outcomes: management information from …" pages on
+gov.uk. Pages are found through the gov.uk search API by that title prefix; a
+new page appears each academic year.
+
+- `ofsted_fe_providers`: the newest page's newest
+  `…further_education_and_skills_-_as_at_<date>.csv`. One row per provider:
+  `Provider URN`, `Provider UKPRN`, `Provider name`, `Provider type`,
+  `Has the provider received a full inspection under the renewed EIF?`,
+  `First day of inspection of latest inspection before renewed EIF`,
+  `Latest overall effectiveness before renewed EIF`,
+  `Date of latest short inspection before renewed EIF (if relevant)`.
+  Key: `provider_urn`.
+- `ofsted_fe_inspections`: renewed-framework inspections. Each page carries
+  cumulative files `…further_education_and_skills_-_<start>_to_<end>.csv`
+  (`10_November_2025_to_31_January_2026`, `…_to_28_February_2026`, …, sometimes
+  with a doubled underscore); the stream reads the file with the latest end date
+  from every page since November 2025, so a report card survives the next
+  year's page. Files named `new_provider_monitoring` or `monitoring_inspections`
+  are excluded. Fields: `Provider URN`, `Provider UKPRN`, `Inspection number`,
+  `First day of inspection`, `Date published`, `Safeguarding standards`.
+  Key: `inspection_number`.
+
+Page and file selection, and the CSV-to-field mapping, live in a module that
+imports only pandas and the standard library, so CI's pytest (no
+`singer_sdk`) can test them against saved real page HTML and real headers.
+
+The existing school-MI stream is unchanged. It already loads every row Ofsted
+lists, inspected or not; `stg_ofsted_inspections` drops the uninspected rows.
+
+### dbt
+
+- `stg_gias_establishments` gains `ukprn` and `inspectorate_name`.
+- `stg_ofsted_fe_providers` and `stg_ofsted_fe_inspections`: dates parsed as
+  `DD/MM/YYYY`; grades `1`–`4` cast to integers and `-` to null; safeguarding
+  `Met` / `Not met` to a boolean.
+- `int_ofsted_fe_latest`, one row per URN we hold. A provider matches a school
+  by `Provider URN` = URN; failing that, by `Provider UKPRN` = the school's GIAS
+  `ukprn`. The rule, in the terms of the C1 current-status rule:
+
+  | FE record | `current_grade` | `latest_visit` |
+  |---|---|---|
+  | Report card newer than the previous-framework inspection (or no such inspection) | null; `rc_inspection_date`, `rc_safeguarding_met` set | the report card (`report_card`) |
+  | Previous-framework grade 1–4, no newer report card | that grade, dated by the inspection's first day, basis `graded` | the later of that inspection (`graded`) and any short inspection (`short`) |
+  | Grade `-` and no report card | — | no row |
+
+  A short inspection's outcome is not in the data, so it is a visit, never a
+  grade change.
+- `int_ofsted_latest` / `fact_ofsted_latest` take FE rows only for URNs with no
+  school-framework record; a school record always wins. FE rows carry
+  `framework = 'Further education and skills'`; school area and report-card
+  area columns stay null. `latest_visit_kind` gains the value `short`.
+- `int_ofsted_listed`: every URN Ofsted's data includes, inspected or not —
+  all rows of `raw.ofsted_inspections` (before staging's date filter) plus every
+  matched FE provider.
+- `dim_school` gains `inspectorate` (GIAS `inspectorate_name`: "ISI", "Ofsted",
+  "Not applicable" or null) and `ofsted_listed` (boolean), the latter joined by
+  the same optional-relation pattern used for `int_ofsted_latest`.
+  `pipeline/tests/test_dag_selectors.py`'s `OPTIONAL_PARENTS` adds
+  `int_ofsted_listed`.
+- The monthly Ofsted DAG build selects
+  `stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ int_ofsted_listed+` as
+  well.
+- Tests: the existing `fact_ofsted_latest` tests (unique URN,
+  `assert_ofsted_current_grade_consistent`) cover FE rows. A new test asserts no
+  URN draws on both a school and an FE record.
+
+## What each page shows
+
+**Search badge and map card.** `buildOfstedListBadge` may return no badge. In
+order:
+
+1. report card → "Report Card · year";
+2. grade in force → grade word · year;
+3. inspected, no grade in force → "Inspected · year";
+4. GIAS inspectorate "ISI" → "Inspected by ISI" (grey);
+5. listed by Ofsted → "Not yet inspected" (grey);
+6. otherwise no badge.
+
+FE schools reach 1–3 with the same labels: Queen Mary's College "Good · 2022";
+The Autism Project – CareTrade (143526) "Report Card · 2026".
+
+**School page.**
+
+- FE: the existing Ofsted section with "Further education and skills" under the
+  heading. A grade reads as today ("Good", "Graded inspection, 6 December
+  2022"); a report card shows its date and the safeguarding result; a later
+  short inspection gets "Latest visit: Short inspection, [date]". No area or
+  programme grid. The existing "Ofsted reports ↗" link resolves for FE
+  providers (Leyton Sixth Form College 130457 → `/provider/40/130457`).
+- ISI: an inspection section reading "Inspected by the Independent Schools
+  Inspectorate (ISI)" with "ISI reports ↗" linking to
+  `https://www.isi.net/reports/`.
+- Listed, never inspected: an Ofsted section reading "Not yet inspected", with
+  the Ofsted reports link (for a new academy, Ofsted's page links the
+  predecessor school's reports).
+- No data: no section, as today.
+
+**Compare.** FE and listed schools follow the school page's states; ISI schools
+read "Inspected by ISI"; schools with no data keep "No inspection in our
+dataset".
+
+**Backend.** The list payload and the detail `school_info` gain `inspectorate`
+and `ofsted_listed`. FE detail needs no new block: `_ofsted_block` already
+carries `current_grade`, `latest_visit`, the report-card date and `framework`.
+`inspectorate` and `ofsted_listed` join `_EXTRA_OPTIONAL_COLUMNS` in
+`backend/data_loader.py`, so a `dim_school` built before PR 1 makes the main
+query retry without them (no badge for groups 4–5) instead of failing.
+
+Copy follows the site's factual rule: inspectorate names and Ofsted's own terms
+("Short inspection", "Not yet inspected"), no derived sentences.
+
+## Delivery
+
+Two PRs, as for C1 and C2.
+
+**PR 1 — pipeline:** the GIAS fields, the two FE streams and their selection
+module, the staging and intermediate models, `dim_school`'s two columns, the
+DAG selector, `OPTIONAL_PARENTS`, unit and data tests.
+
+**PR 2 — site (after the DAGs have run on PR 1):** the backend fields and the
+optional-column fallback, the badge rule, the school-page states, compare, and
+E2E.
+
+## Testing
+
+- **Extractor (pytest, pandas-only module):** from saved page HTML, the newest
+  "as at" file and the latest cumulative file per page are chosen and
+  monitoring files are ignored; the doubled-underscore file name parses; real
+  headers map to the declared fields.
+- **dbt unit tests:** previous-framework grade only; grade with a later short
+  inspection (grade dated by the full inspection, latest visit `short`); grade
+  with a newer report card (report card, no grade); report card only;
+  UKPRN-only match; a school record wins over an FE record; a listed,
+  uninspected school is in `int_ofsted_listed` and not in `int_ofsted_latest`.
+- **Backend pytest:** both new fields in the list and detail payloads; the main
+  query falls back when either column is missing.
+- **Jest:** one case per badge rule including "no badge"; every badge caller
+  renders nothing when there is no badge; the school page's FE, ISI and
+  listed-uninspected states; compare cells.
+- **E2E:** Queen Mary's College (145175) shows an Ofsted grade, not "Not yet
+  inspected"; Abbey College Cambridge (136083) shows "Inspected by ISI"; Abbas
+  and Templecombe Church School (150031) shows "Not yet inspected".
+
+## Rollout and verification
+
+1. Locally, load real GIAS, school MI and FE MI through the changed extractors,
+   build the models, and check: about 490 FE rows (443 + 45 + 2); 145175 graded
+   2 on 2022-12-06; 143526 report card 2026-05-12, safeguarding met; 1,341
+   schools with inspectorate "ISI"; 150031 listed, not in `fact_ofsted_latest`.
+2. PR 1 merges; Tudor runs the daily DAG (GIAS fields) and the monthly Ofsted
+   DAG on staging; the staging API shows the same examples.
+3. PR 2 merges, so its E2E journeys run against loaded data.
+4. Production, Tudor's decision: promote PR 1, run both DAGs, promote PR 2. If
+   PR 2 arrives first, the fallback query runs and groups 4–5 show no badge
+   until the DAGs have run.
+5. Validate on production: every "Not yet inspected" belongs to a school
+   Ofsted's own data lists without an inspection; every "Inspected by ISI"
+   matches GIAS; every FE grade, date and report card matches Ofsted's FE files;
+   schools with no data show no badge. Then mark H4 resolved in the audit
+   report.
+
+## Risks
+
+- **FE page or file naming changes.** Selection is by title prefix and file-name
+  pattern; if nothing matches, the FE streams emit nothing and FE schools fall
+  back to "no badge", never a false claim. The extractor tests pin the patterns
+  against real HTML.
+- **A provider's UKPRN shared by several URNs.** Each matched URN takes the
+  provider's result; the URN match is always tried first.
+- **"Not yet inspected" disappears briefly** if PR 2 is promoted before the
+  DAGs run (fallback query). It never appears wrongly.
+
+## Amendments (implementation plan, 2026-10-13)
+
+- **UKPRN comes from a new view, `stg_gias_ukprn`,** on the raw GIAS source,
+  not from a `ukprn` column on `stg_gias_establishments`. A `ref` to
+  `stg_gias_establishments` from the Ofsted models would put them in the daily
+  build's `stg_gias_establishments+` selection, where their Ofsted parents are
+  not built (`pipeline/tests/test_dag_selectors.py`). The monthly Ofsted build
+  selects `stg_gias_ukprn+`; the daily build must not rebuild it, because
+  rebuilding a view drops the views on it (CASCADE) and `dim_school` reads
+  them. `stg_gias_establishments` gains only `inspectorate_name`.
+- **First rollout order (each environment, once).** The new models need
+  `raw.gias_establishments."UKPRN"`, which only a daily run with PR 1 creates,
+  and `dim_school`'s unit tests need `int_ofsted_listed`, which only a monthly
+  run creates. So: run the daily DAG (its `dbt_build` fails once on
+  `dim_school`'s unit tests — expected), then `school_data_monthly_ofsted`,
+  then the daily DAG again. From then on both schedules run normally.
+- **A report card is recognised by `rc_inspection_date` as well as by its area
+  labels** (`hasReportCard`), on the school page and in compare. An FE report
+  card carries a safeguarding result and no school areas, so the label test
+  alone read it as "No overall grade".
+- **The compare page's Ofsted link is fixed.** `ofsted_page_url` built
+  `https://reports.ofsted.gov.uk/provider/21/{urn}`; category 21 is primary
+  schools, so the link 404s for every secondary, special and FE school
+  (checked: 137086, 101099, 130457, 145175). It now uses
+  `…/inspection-reports/find-inspection-report/provider/ELS/{urn}`, which
+  redirects to the right category and is what the school page already uses.
@@ -0,0 +1,300 @@
+# 2023/24 Results and DfE LA Averages — Design
+
+**Date:** 2026-10-06
+**Status:** approved design, not yet implemented
+**Scope:** `tap_uk_ees` (KS4 results, KS4 information, new LA stream),
+`safe_numeric`, new `fact_ks4_la_averages` mart, annual EES DAG selector,
+`/api/la-averages`, secondary search rows and map cards
+**Fixes:** audit findings C2 and H2, from the 3 Oct 2026 accuracy audit
+
+## Goal
+
+Show every 2023/24 result and school-information figure DfE published, and
+compare each state-funded secondary school with DfE's own local-authority
+average.
+
+## The problem
+
+### C2: 2023/24 is empty
+
+Bishop Stopford School (137086) shows Attainment 8 60.7 for 2022/23, nothing
+for 2023/24 and 58.7 for 2024/25. DfE's 2023/24 figures are Attainment 8 64.1,
+Progress 8 +1.02 and English and maths grade 4+ 91.7%. 2023/24 is the last year
+DfE published Progress 8 (2024/25 has no KS2 baseline), so the site shows no
+recent Progress 8 for any school. Site-wide, 4,170 listed schools have a DfE
+2023/24 Attainment 8 and 3,384 a Progress 8.
+
+There are three separate causes.
+
+1. **KS4 results.** The 2024/25 release's
+   `202425_performance_tables_schools_final.csv` is a time series: it holds
+   2022/23, 2023/24 and 2024/25 under the current column names, with the right
+   values (Bishop Stopford 2023/24: 64.1, 1.02, 91.7). The 2023/24 release's own
+   `202324_performance_tables_schools_final.csv`, re-issued on 10 March 2026,
+   uses the older names (`t_pupils`, `avg_att8`, `avg_p8score`,
+   `pt_l2basics_94` …). `EESDatasetStream` reads releases in the API's order,
+   newest first, so the old file is read last. Its rows carry none of the
+   declared fields, and target-postgres upserts on the stream's primary key
+   (`append_only = not key_properties` in meltanolabs-target-postgres 0.8.0),
+   so they overwrite the good 2023/24 rows with nulls.
+
+   The two files share the keys of every "Total" row, but 34,254 of the 57,090
+   2023/24 sub-group rows use different labels ("Low prior" against "Low prior
+   attainment"). Those old-label rows sit in `raw.ees_ks4_performance` with
+   null measures. Nothing reads them.
+
+2. **KS4 school information.** 2023/24 information exists only in the 2023/24
+   release, in `202324_information_about_schools_final.csv`, with the older
+   names. `EESKS4InfoStream` declares the newer ones, so prior attainment,
+   SEN percentages, disadvantage gaps and Progress 8 banding are null for
+   2023/24.
+
+3. **KS2 school information.** The 2023/24 file
+   `ks2_school_information_data.csv` uses the declared names, and pupil counts
+   load (school 147411: 818 pupils, 112 eligible). Its percentages are written
+   with a sign (`ptfsm6cla1a = "34%"`). `safe_numeric` accepts only
+   `^-?[0-9]+(\.[0-9]+)?$`, so disadvantaged, EAL, SEN and mobility percentages
+   are null for every school in 2023/24.
+
+DfE published no school-level KS2 information file for 2022/23 (the 2023/24
+release's attainment file carries 2022/23 attainment rows only). Those nulls are
+a gap in the source, not a defect.
+
+### H2: "vs LA avg" uses the wrong average
+
+`/api/la-averages` (`backend/app.py`) takes an unweighted mean of every school
+in the LA with an Attainment 8 score, independent and special schools included.
+Independent schools score low because DfE measures exclude IGCSEs, and special
+schools score low for other reasons, so the average is too low almost
+everywhere. Kensington and Chelsea's "LA avg" is 35.2: the mean of 6 state
+schools (54.9) and 8 independent schools (20.4). DfE's figure is 54.5. Of the
+151 LAs the audit compared, ours was lower in 147, by 7.1 points on average and
+by up to 19.3, so most secondary schools look better than their area.
+
+DfE's LA averages are already in the file the pipeline downloads for the
+England averages: the "summary, all state-funded" data set
+(`data-catalogue/data-set/1b649e16-01e8-435b-a814-56be2faf9054/csv`). Its
+`Local authority` / `All state-funded` / Total rows match DfE's published
+performance-table LA averages (RECTYPE 4) for all 152 LAs in 2024/25, with no
+difference. `EESKs4NationalStream` keeps the England row and discards them.
+"All state-funded" is the same population as the England benchmark the site
+already shows.
+
+Computing the average ourselves from state-funded schools, weighted by pupils,
+was tested and rejected: it differs from DfE's figure by 1.1 points on average
+and is never exact.
+
+## Non-goals
+
+- 2022/23 KS2 school information (DfE published none).
+- An "excludes IGCSEs" note wherever an independent school's Attainment 8
+  appears. This change only stops comparing independent schools with the LA.
+- Showing 2023/24 Progress 8 in the GCSE section's headline. The history section
+  shows it once the data loads; the 2024/25 banner stays true.
+- Updating the fixed EES data-set id when DfE publishes 2025/26. It already
+  feeds the England averages; the LA stream shares it.
+- LA comparisons for other measures or on other pages.
+- Deleting the leftover old-label raw rows automatically.
+
+## The rules
+
+### The newest release owns every year it contains (KS4 results only)
+
+`EESDatasetStream` gets an opt-in class attribute,
+`_newest_release_owns_period: bool = False`. `EESKS4PerformanceStream` sets it
+to `True`. When it is on:
+
+- releases are processed newest first by `time_period` (from the release slug),
+  not in the API's order;
+- the stream records each `time_period` it has emitted;
+- in each older release, rows whose `time_period` a newer release already
+  emitted are dropped, and the stream logs how many it skipped and for which
+  years;
+- years only an older release contains are emitted as before.
+
+The filter is a pure function, testable without a download. Other streams keep
+the current behaviour. A general rule would be wrong: the 2024/25 KS2 file holds
+98,448 of the 955,956 rows the 2023/24 release has for 2023/24.
+
+### KS4 information: old names
+
+`EESKS4InfoStream._column_renames` maps the 2023/24 names onto the declared
+fields:
+
+| 2023/24 column | Declared field |
+|---|---|
+| `t_allks_pupils` | `allks_pupil_count` |
+| `t_allks_boys` | `allks_boys_count` |
+| `t_allks_girls` | `allks_girls_count` |
+| `t_pupils` | `endks4_pupil_count` |
+| `avg_ks2_scaledscore` | `ks2_scaledscore_average` |
+| `pt_sen_with_ehcp` | `sen_with_ehcp_pupil_percent` |
+| `pt_sen` | `sen_pupil_percent` |
+| `pt_sen_no_ehcp` | `sen_no_ehcp_pupil_percent` |
+| `diffn_att8` | `attainment8_diffn` |
+| `diffn_p8mea` | `progress8_diffn` |
+| `p8_banding` | `progress8_banding` |
+
+Newer files contain none of the old names, so they are unaffected.
+
+### `safe_numeric` accepts a trailing `%`
+
+The pattern becomes `^-?[0-9]+(\.[0-9]+)?%?$` and the cast reads
+`rtrim(col, '%')`. A value such as `34%` can only mean 34. Suppression codes
+(`c`, `z`, `x` …) still become null.
+
+### LA averages
+
+A new stream, `ees_ks4_la`, reads the same CSV as `ees_ks4_national` and keeps
+rows where `geographic_level = 'Local authority'`,
+`establishment_type_group = 'All state-funded'`, `breakdown_topic = 'Total'` and
+`breakdown = 'Total'` (case-insensitive, as the national stream compares). It
+emits `time_period`, `old_la_code`, `new_la_code`, `la_name` and the 8 headline
+measures the national stream emits (`_KS4_NATIONAL_COL_MAP`). Primary key:
+(`time_period`, `old_la_code`). The two streams share one download-and-filter
+helper.
+
+`old_la_code` is the GIAS LA code (`local_authority_code`), so schools join on
+the code, not the name. Names match today for every LA DfE publishes; DfE
+publishes no figure for City of London.
+
+### Which schools get a gap
+
+The search row and the map card show "vs LA avg" only when the school has an
+Attainment 8 score, is neither special (`isSpecialSchool`) nor independent
+(`isIndependentSchool`: "independent" in the GIAS type, which covers "Other
+independent school" and "Other independent special school"), and its LA has a
+DfE figure for the year the endpoint serves.
+
+## Delivery
+
+Two PRs, as for C1.
+
+### PR 1: pipeline
+
+- `pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py`: release
+  precedence, KS4 information renames, `ees_ks4_la` stream registered in
+  `discover_streams`, shared national/LA helper.
+- `pipeline/transform/macros/safe_numeric.sql`: trailing `%`.
+- `pipeline/transform/models/staging/`: source `raw.ees_ks4_la` and
+  `stg_ees_ks4_la` (view): `cast(old_la_code as integer) as la_code`,
+  `cast(time_period as integer) as year`, `la_name`, measures via
+  `safe_numeric`, named as in `stg_ees_ks4_national`.
+- `pipeline/transform/models/marts/fact_ks4_la_averages.sql` (table): one row
+  per (`year`, `la_code`) with `la_name` and the columns of
+  `fact_ks4_national_averages`. Schema tests: unique (`year`, `la_code`);
+  `year`, `la_code` and `la_name` not null.
+- Data tests in `pipeline/transform/tests/`:
+  - `assert_ks4_years_have_results`: every year in `stg_ees_ks4` has a non-null
+    Attainment 8 for at least 50% of its rows. DfE's files reach 82% each year;
+    2023/24 loads at 0% today. Pre-2019 years come from the legacy model and are
+    not tested.
+  - `assert_ks2_info_percentages_loaded`: for every year in `stg_ees_ks2` where
+    at least 1,000 rows have `total_pupils`, at least 90% of those rows have
+    `disadvantaged_pct`. DfE's files reach 96–97%. 2022/23 has no pupil counts
+    and is skipped.
+  - `assert_ks4_la_averages_cover_las`: the latest year in
+    `fact_ks4_la_averages` has at least 145 LAs (DfE: 152).
+- `pipeline/dags/school_data_pipeline.py`: the annual EES build selects
+  `stg_ees_ks4_la+`.
+- `docs/ARCHITECTURE.md`: LA averages come from DfE's data set.
+
+### PR 2: backend and UI (after the EES DAG has run on PR 1)
+
+- `backend/models.py`: `Ks4LaAverage` for `marts.fact_ks4_la_averages`.
+- `backend/app.py` `/api/la-averages`: the year is the latest with any school
+  Attainment 8 (as now). It reads that year's rows from the mart and keys each
+  `attainment_8_score` by our LA name, through the `local_authority_code` →
+  `local_authority` pairs in the school data. The response shape is unchanged.
+  No rows for that year, a missing table or a query error give an empty map,
+  logged, never another year's figures and never a computed mean.
+- `nextjs-app/lib/utils.ts`: `isIndependentSchool(school)`.
+- `nextjs-app/components/SecondarySchoolRow.tsx` and
+  `nextjs-app/components/LeafletMapInner.tsx`: the rule in "Which schools get a
+  gap".
+- `e2e/tests/journeys.spec.ts`: the two journeys under Testing.
+
+## Testing
+
+**Extractor (pytest, `pipeline/tests/`, new):**
+
+- precedence: with the flag on, a year in a newer release is emitted once, from
+  the newer release; a year only an older release has is kept; with the flag
+  off, every row passes;
+- releases arriving oldest first are still processed newest first;
+- KS4 information: an old-format row yields `endks4_pupil_count`,
+  `ks2_scaledscore_average`, `progress8_banding` and the rest of the table;
+- LA filter: a small CSV with national, regional, LA and sub-group rows yields
+  one row per LA and year, with the declared fields.
+
+**dbt (local `pgserver`, as for C1):**
+
+- unit test on `stg_ees_ks2`: `34%` → 34, `34` → 34, `c` → null;
+- unit test on `stg_ees_ks4_la` or the mart: codes and years cast, measures
+  carried;
+- the three data tests and the schema tests above;
+- `pipeline/tests/test_dag_selectors.py` passes with the new selector.
+
+**Backend (pytest):**
+
+- the response gives the mart's figure where the fixture's plain mean differs;
+- matching works by code when the mart's `la_name` differs from ours;
+- a mart without the served year, and a missing table, give an empty map.
+
+**Front end (Jest):**
+
+- `isIndependentSchool` for "Other independent school", "Other independent
+  special school", "Academy converter" and null;
+- `SecondarySchoolRow` and the map card show no gap for an independent school
+  and show one for a state school with an LA figure.
+
+**E2E (`journeys.spec.ts`, PR 2):**
+
+- C2: Bishop Stopford (137086) history shows 2023/24 Attainment 8 64.1 and
+  Progress 8 +1.02. These are final figures, so the journey stays stable.
+- H2: a search returning an independent and a state secondary: the independent
+  row has no "vs LA avg", the state row has one. The exact gap is not asserted:
+  DfE's 2025/26 provisional KS4 data is due and would change it.
+
+## Rollout and verification
+
+1. PR 1 merges to staging. Tudor runs `school_data_annual_ees` on staging.
+2. Through the staging API: Bishop Stopford 2023/24 Attainment 8 64.1 and
+   Progress 8 1.02; school 147411 2023/24 disadvantaged 34; the LA mart has 152
+   LAs for 2024/25. If a data test fails, the build stops before search sync;
+   investigate before going further.
+3. PR 2 merges, so the post-merge E2E gate runs against loaded data.
+4. Production, Tudor's decision, in order: promote PR 1, run the EES DAG on
+   production, promote PR 2. If PR 2 arrives first, the endpoint returns an
+   empty map and rows show no gap, never a wrong one.
+5. Optional cleanup, in the PR 1 description for Tudor:
+   `delete from raw.ees_ks4_performance where time_period = '202324' and pupil_count is null`.
+   New-format rows always have `pupil_count` (suppressed values are `c` or `z`,
+   not null), so this removes exactly the old-label leftovers.
+6. Validation on production against DfE's files: every listed school's 2023/24
+   Attainment 8, Progress 8 and English and maths figures match; the 2023/24 KS4
+   and KS2 information fields match; `/api/la-averages` equals DfE for all 152
+   LAs; independent rows show no gap. Then mark C2 and H2 resolved in the audit
+   report.
+
+## Expected visible change
+
+- Secondary history charts and tables run unbroken from 2022/23 to 2024/25, and
+  2023/24 Progress 8 appears for about 3,400 schools.
+- 2023/24 school information (KS2 and KS4) fills in.
+- "vs LA avg" drops for most state secondaries. In Kensington and Chelsea, a
+  school with Attainment 8 60.0 moves from +24.8 to +5.5.
+- Independent schools and City of London schools show no LA gap.
+
+## Risks
+
+- **Data-test thresholds stop a good build.** They were set from DfE's own
+  files (82% and 96–97% against 50% and 90%). A failure means the data changed
+  shape, which is what they are for.
+- **`safe_numeric` is shared by 14 models.** Values written as `n%` were null
+  and become numbers. No DfE column uses `%` for anything but a percentage.
+  The staging EES run is the check.
+- **Precedence drops data an older release holds more completely.** It is
+  opt-in for KS4 results, where both files hold the same 57,090 2023/24 keys.
+- **The fixed EES data-set id goes stale** when 2025/26 is published. The
+  endpoint's year check then gives no gap rather than a mismatched one.
@@ -146,7 +146,7 @@ with DAG(

    dbt_build_ofsted = BashOperator(
        task_id="dbt_build",
-        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ofsted_inspections+ int_ofsted_latest+ fact_ofsted_inspection+ dim_school+ --exclude stg_ees_ks4_destinations+ stg_ees_ks5_destinations+",
+        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ofsted_inspections+ stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ stg_gias_ukprn+ int_ofsted_latest+ fact_ofsted_inspection+ dim_school+ --exclude stg_ees_ks4_destinations+ stg_ees_ks5_destinations+",
    )

    sync_typesense_ofsted = BashOperator(
@@ -193,7 +193,7 @@ with DAG(

    dbt_build_ees = BashOperator(
        task_id="dbt_build",
-        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ees_ks2+ stg_legacy_ks2+ stg_ees_ks4+ stg_legacy_ks4+ stg_ees_census+ stg_ees_admissions+ stg_ees_ks2_national+ stg_ees_ks4_national+ stg_ees_ks4_destinations+ stg_ees_ks5_destinations+ stg_ees_ks4_destinations_national+ stg_ees_ks5_destinations_national+",
+        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ees_ks2+ stg_legacy_ks2+ stg_ees_ks4+ stg_legacy_ks4+ stg_ees_census+ stg_ees_admissions+ stg_ees_ks2_national+ stg_ees_ks4_national+ stg_ees_ks4_la+ stg_ees_ks4_destinations+ stg_ees_ks5_destinations+ stg_ees_ks4_destinations_national+ stg_ees_ks5_destinations_national+",
    )

    sync_typesense_ees = BashOperator(
@@ -0,0 +1,47 @@
+"""KS4 school information: the fields the stream declares, and the older
+names DfE used for them.
+
+2023/24 information exists only in the 2023/24 release, whose
+202324_information_about_schools_final.csv (re-issued 10 March 2026) uses the
+older names. Without the renames every 2023/24 field loaded as null (audit C2).
+Newer files contain none of the older names, so the renames leave them alone.
+
+Free of the Singer SDK so CI's pytest can load it.
+"""
+
+# Declared Singer fields besides the required time_period and school_urn.
+KS4_INFO_FIELDS = (
+    "school_laestab",
+    "school_name",
+    "establishment_type_group",
+    "reldenom",
+    "admpol_pt",
+    "egender",
+    "agerange",
+    "allks_pupil_count",
+    "allks_boys_count",
+    "allks_girls_count",
+    "endks4_pupil_count",
+    "ks2_scaledscore_average",
+    "sen_with_ehcp_pupil_percent",
+    "sen_pupil_percent",
+    "sen_no_ehcp_pupil_percent",
+    "attainment8_diffn",
+    "progress8_diffn",
+    "progress8_banding",
+)
+
+# 2023/24 column name → declared field.
+KS4_INFO_RENAMES = {
+    "t_allks_pupils": "allks_pupil_count",
+    "t_allks_boys": "allks_boys_count",
+    "t_allks_girls": "allks_girls_count",
+    "t_pupils": "endks4_pupil_count",
+    "avg_ks2_scaledscore": "ks2_scaledscore_average",
+    "pt_sen_with_ehcp": "sen_with_ehcp_pupil_percent",
+    "pt_sen": "sen_pupil_percent",
+    "pt_sen_no_ehcp": "sen_no_ehcp_pupil_percent",
+    "diffn_att8": "attainment8_diffn",
+    "diffn_p8mea": "progress8_diffn",
+    "p8_banding": "progress8_banding",
+}
@@ -0,0 +1,56 @@
+"""DfE's KS4 "summary, all state-funded" data set (Key stage 4 performance).
+
+One CSV holds England, regional and local-authority headline rows for every
+year since 2018/19. The England stream (ees_ks4_national) and the LA stream
+(ees_ks4_la) both read it. Its LA rows match DfE's published performance-table
+LA averages exactly (audit H2). Suppressed values ('z', 'x') become NULL in
+dbt; Progress 8 is 'z' in years with no KS2 baseline (2024/25): DfE policy,
+not missing data.
+
+Free of the Singer SDK so CI's pytest can load it.
+"""
+from __future__ import annotations
+
+import pandas as pd
+
+KS4_SUMMARY_CSV_URL = (
+    "https://explore-education-statistics.service.gov.uk/data-catalogue/"
+    "data-set/1b649e16-01e8-435b-a814-56be2faf9054/csv"
+)
+
+# CSV column → Singer field: the same 8 headline measures at every level.
+KS4_HEADLINE_COL_MAP = {
+    "attainment8_average":    "attainment_8_score",
+    "progress8_average":      "progress_8_score",
+    "engmath_94_percent":     "english_maths_standard_pass_pct",
+    "engmath_95_percent":     "english_maths_strong_pass_pct",
+    "ebacc_entering_percent": "ebacc_entry_pct",
+    "ebacc_94_percent":       "ebacc_standard_pass_pct",
+    "ebacc_95_percent":       "ebacc_strong_pass_pct",
+    "ebacc_aps_average":      "ebacc_avg_score",
+}
+
+
+def headline_rows(df: pd.DataFrame, geographic_level: str) -> pd.DataFrame:
+    """All-pupil rows for all state-funded schools at one geographic level
+    ("National" or "Local authority"). Column names are lower-cased first;
+    a filter column the file lacks is not applied."""
+    df = df.copy()
+    df.columns = [c.strip().lower() for c in df.columns]
+    for col, want in (
+        ("geographic_level", geographic_level),
+        ("establishment_type_group", "All state-funded"),
+        ("breakdown_topic", "Total"),
+        ("breakdown", "Total"),
+    ):
+        if col in df.columns:
+            df = df[df[col].str.strip().str.lower() == want.lower()]
+    return df
+
+
+def headline_record(row: pd.Series, keys: tuple[str, ...]) -> dict[str, str]:
+    """A Singer record: the identifying columns, then the headline measures."""
+    record = {key: str(row.get(key, "")).strip() for key in keys}
+    for csv_col, field in KS4_HEADLINE_COL_MAP.items():
+        record[field] = str(row.get(csv_col, "")).strip()
+    return record
@@ -0,0 +1,59 @@
+"""Which release a row comes from when DfE re-publishes a year.
+
+DfE re-publishes earlier years inside later releases: the 2024/25 KS4 results
+file holds 2022/23, 2023/24 and 2024/25. The 2023/24 release's own file,
+re-issued on 10 March 2026 under older column names, was read after it and
+overwrote every 2023/24 row with blanks (audit C2). A stream that opts in
+treats the newest release as the authority for every year it contains.
+
+Free of the Singer SDK so CI's pytest, which installs only the backend's
+requirements, can load it.
+"""
+from __future__ import annotations
+
+import re
+
+import pandas as pd
+
+_SLUG_YEAR = re.compile(r"^(\d{4})-(\d{2})(?:-|$)")
+
+
+def slug_to_time_period(slug: str) -> str | None:
+    """A release slug's academic year as a time_period: '2022-23' → '202223'.
+
+    Suffixed slugs ('2024-25-revised', '2025-26-provisional') give the same
+    year, so newest_first places them by year; read as unknown, a revised
+    release went last and lost its year to the first release.
+    """
+    match = _SLUG_YEAR.match(slug or "")
+    return match.group(1) + match.group(2) if match else None
+
+
+def newest_first(releases: list[dict]) -> list[dict]:
+    """Releases by time_period, newest first. A release whose time_period is
+    unknown goes last, in the order given."""
+    dated = [r for r in releases if r.get("time_period")]
+    undated = [r for r in releases if not r.get("time_period")]
+    return sorted(dated, key=lambda r: r["time_period"], reverse=True) + undated
+
+
+def periods_in(df: pd.DataFrame) -> set[str]:
+    """The years a release's rows cover."""
+    if "time_period" not in df.columns:
+        return set()
+    return set(df["time_period"].astype(str).str.strip()) - {""}
+
+
+def drop_owned_periods(
+    df: pd.DataFrame, owned: set[str]
+) -> tuple[pd.DataFrame, dict[str, int]]:
+    """Drop the rows for years a newer release already supplied.
+
+    Returns the rows kept and, for each year dropped, how many rows went.
+    """
+    if "time_period" not in df.columns or not owned:
+        return df, {}
+    periods = df["time_period"].astype(str).str.strip()
+    dropped = periods.isin(owned)
+    skipped = {str(k): int(v) for k, v in periods[dropped].value_counts().items()}
+    return df[~dropped], skipped
@@ -17,6 +17,20 @@ import requests
 from singer_sdk import Stream, Tap
 from singer_sdk import typing as th

+from tap_uk_ees.ks4_info import KS4_INFO_FIELDS, KS4_INFO_RENAMES
+from tap_uk_ees.ks4_summary import (
+    KS4_HEADLINE_COL_MAP,
+    KS4_SUMMARY_CSV_URL,
+    headline_record,
+    headline_rows,
+)
+from tap_uk_ees.release_precedence import (
+    drop_owned_periods,
+    newest_first,
+    periods_in,
+    slug_to_time_period,
+)
+
 CONTENT_API_BASE = (
    "https://content.explore-education-statistics.service.gov.uk/api"
 )
@@ -31,14 +45,6 @@ def get_content_release_id(publication_slug: str) -> str:
    return resp.json()["id"]


-def _slug_to_time_period(slug: str) -> str | None:
-    """Convert a release slug like '2022-23' to a time_period like '202223'."""
-    parts = slug.split("-")
-    if len(parts) == 2 and len(parts[0]) == 4 and len(parts[1]) == 2:
-        return parts[0] + parts[1]
-    return None
-
-
 def get_all_releases(publication_slug: str) -> list[dict]:
    """Return all releases for a publication as dicts with 'id' and 'time_period'.

@@ -63,7 +69,7 @@ def get_all_releases(publication_slug: str) -> list[dict]:
            total_pages = paging.get("totalPages", 1)

        for r in releases:
-            time_period = _slug_to_time_period(r.get("slug", ""))
+            time_period = slug_to_time_period(r.get("slug", ""))
            result.append({"id": r["id"], "time_period": time_period})

        if page >= total_pages:
@@ -88,6 +94,8 @@ class EESDatasetStream(Stream):
    target CSV path inside the ZIP (substring match, not exact).
    Subclasses may set _column_renames to map messy CSV column names to
    clean Singer field names before yielding records.
+    Subclasses may set _newest_release_owns_period when DfE re-publishes
+    earlier years in later releases and the newest copy is the authority.
    """

    replication_key = None
@@ -96,6 +104,7 @@ class EESDatasetStream(Stream):
    _urn_column: str = "school_urn"  # column name for URN in the CSV
    _encoding: str = "utf-8"  # CSV file encoding (some DfE files use latin-1)
    _column_renames: dict = {}  # CSV column name → Singer field name
+    _newest_release_owns_period: bool = False  # see release_precedence.py

    def get_records(self, context):
        import pandas as pd
@@ -110,6 +119,9 @@ class EESDatasetStream(Stream):
        self.logger.info(
            "Found %d release(s) for %s", len(releases), self._publication_slug
        )
+        if self._newest_release_owns_period:
+            releases = newest_first(releases)
+        owned_periods: set[str] = set()

        for release in releases:
            release_id = release["id"]
@@ -163,6 +175,15 @@ class EESDatasetStream(Stream):
            if urn_col in df.columns:
                df = df[df[urn_col].notna() & (df[urn_col] != "")]

+            if self._newest_release_owns_period:
+                df, skipped = drop_owned_periods(df, owned_periods)
+                for period, count in sorted(skipped.items()):
+                    self.logger.info(
+                        "Skipping %d rows for %s from release %s: a newer release supplied that year",
+                        count, period, release_id,
+                    )
+                owned_periods |= periods_in(df)
+
            self.logger.info("Emitting %d school-level rows from release %s", len(df), release_id)

            for _, row in df.iterrows():
@@ -251,6 +272,9 @@ class EESKS4PerformanceStream(EESDatasetStream):
    primary_keys = ["school_urn", "time_period", "breakdown_topic", "breakdown", "sex"]
    _publication_slug = "key-stage-4-performance"
    _target_filename = "performance_tables_schools"
+    # DfE's 2024/25 file re-publishes 2022/23 and 2023/24 under current names;
+    # the 2023/24 release's own file uses older ones (audit C2).
+    _newest_release_owns_period = True
    schema = th.PropertiesList(
        th.Property("time_period", th.StringType, required=True),
        th.Property("school_urn", th.StringType, required=True),
@@ -310,34 +334,20 @@ class EESKS4PerformanceStream(EESDatasetStream):


 # ── KS4 Information (wide format: one row per school, context/demographics) ──
-# File: 202425_information_about_schools_provisional.csv (38 cols)
+# Files: 202425_information_about_schools_final.csv (38 cols, current names);
+# 202324_information_about_schools_final.csv (60 cols, older names — the only
+# source of 2023/24 information). Field list and renames: ks4_info.py.

 class EESKS4InfoStream(EESDatasetStream):
    name = "ees_ks4_info"
    primary_keys = ["school_urn", "time_period"]
    _publication_slug = "key-stage-4-performance"
    _target_filename = "information_about_schools"
+    _column_renames = KS4_INFO_RENAMES
    schema = th.PropertiesList(
        th.Property("time_period", th.StringType, required=True),
        th.Property("school_urn", th.StringType, required=True),
-        th.Property("school_laestab", th.StringType),
-        th.Property("school_name", th.StringType),
-        th.Property("establishment_type_group", th.StringType),
-        th.Property("reldenom", th.StringType),
-        th.Property("admpol_pt", th.StringType),
-        th.Property("egender", th.StringType),
-        th.Property("agerange", th.StringType),
-        th.Property("allks_pupil_count", th.StringType),
-        th.Property("allks_boys_count", th.StringType),
-        th.Property("allks_girls_count", th.StringType),
-        th.Property("endks4_pupil_count", th.StringType),
-        th.Property("ks2_scaledscore_average", th.StringType),
-        th.Property("sen_with_ehcp_pupil_percent", th.StringType),
-        th.Property("sen_pupil_percent", th.StringType),
-        th.Property("sen_no_ehcp_pupil_percent", th.StringType),
-        th.Property("attainment8_diffn", th.StringType),
-        th.Property("progress8_diffn", th.StringType),
-        th.Property("progress8_banding", th.StringType),
+        *[th.Property(field, th.StringType) for field in KS4_INFO_FIELDS],
    ).to_dict()


@@ -564,37 +574,23 @@ class EESKs2NationalStream(Stream):
            yield record


-# ── KS4 National Headlines (national level only — one row per year) ──────────
-# Dataset: "National characteristics summary data" (Key stage 4 performance).
-# Official England state-funded headline measures, 2018/19 → latest.
-# Suppressed values ('z', 'x') → NULL downstream. Progress 8 is legitimately
-# absent in years with no KS2 baseline (e.g. 2024/25) — that is DfE policy,
-# not missing data.
+# ── KS4 National and LA Headlines (one data set, two streams) ────────────────
+# DfE's "summary, all state-funded" data set: England, regional and LA rows,
+# 2018/19 → latest. URL, measures and filters: ks4_summary.py.

-_KS4_NATIONAL_CSV_URL = (
-    "https://explore-education-statistics.service.gov.uk/data-catalogue/"
-    "data-set/1b649e16-01e8-435b-a814-56be2faf9054/csv"
-)
+def _read_ks4_summary(logger):
+    """Download DfE's KS4 summary data set."""
+    import pandas as pd

-_KS4_NATIONAL_COL_MAP = {
-    "attainment8_average":    "attainment_8_score",
-    "progress8_average":      "progress_8_score",
-    "engmath_94_percent":     "english_maths_standard_pass_pct",
-    "engmath_95_percent":     "english_maths_strong_pass_pct",
-    "ebacc_entering_percent": "ebacc_entry_pct",
-    "ebacc_94_percent":       "ebacc_standard_pass_pct",
-    "ebacc_95_percent":       "ebacc_strong_pass_pct",
-    "ebacc_aps_average":      "ebacc_avg_score",
-}
+    logger.info("Downloading KS4 summary data set: %s", KS4_SUMMARY_CSV_URL)
+    resp = requests.get(KS4_SUMMARY_CSV_URL, timeout=60)
+    resp.raise_for_status()
+    return pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False)


 class EESKs4NationalStream(Stream):
-    """National KS4 headline averages — one row per academic year.
-
-    Filters to geographic_level == 'National', establishment_type_group ==
-    'All state-funded', breakdown_topic == 'Total', breakdown == 'Total'
-    so only the England-wide all-pupils row per year is emitted.
-    """
+    """National KS4 headline averages — one row per academic year (England,
+    all state-funded schools, all pupils)."""

    name = "ees_ks4_national"
    primary_keys = ["time_period"]
@@ -602,34 +598,41 @@ class EESKs4NationalStream(Stream):

    schema = th.PropertiesList(
        th.Property("time_period", th.StringType, required=True),
-        *[th.Property(out, th.StringType) for out in _KS4_NATIONAL_COL_MAP.values()],
+        *[th.Property(out, th.StringType) for out in KS4_HEADLINE_COL_MAP.values()],
    ).to_dict()

    def get_records(self, context):
-        import pandas as pd
-
-        self.logger.info("Downloading KS4 national headlines: %s", _KS4_NATIONAL_CSV_URL)
-        resp = requests.get(_KS4_NATIONAL_CSV_URL, timeout=60)
-        resp.raise_for_status()
-
-        df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False)
-        df.columns = [c.strip().lower() for c in df.columns]
-
-        for col, want in [
-            ("geographic_level", "national"),
-            ("establishment_type_group", "all state-funded"),
-            ("breakdown_topic", "total"),
-            ("breakdown", "total"),
-        ]:
-            if col in df.columns:
-                df = df[df[col].str.strip().str.lower() == want]
-
+        df = headline_rows(_read_ks4_summary(self.logger), "National")
        self.logger.info("Emitting %d national KS4 rows", len(df))
        for _, row in df.iterrows():
-            record = {"time_period": row.get("time_period", "").strip()}
-            for csv_col, field in _KS4_NATIONAL_COL_MAP.items():
-                record[field] = row.get(csv_col, "").strip()
-            yield record
+            yield headline_record(row, ("time_period",))
+
+
+class EESKs4LaStream(Stream):
+    """DfE's KS4 local-authority averages — one row per academic year and LA
+    (all state-funded schools, all pupils), from the same data set as
+    ees_ks4_national. They match DfE's published performance-table LA averages
+    and replace a mean the API took over every school, independent and special
+    included (audit H2). old_la_code is the GIAS LA code: schools join on it,
+    not on the name."""
+
+    name = "ees_ks4_la"
+    primary_keys = ["time_period", "old_la_code"]
+    replication_key = None
+
+    schema = th.PropertiesList(
+        th.Property("time_period", th.StringType, required=True),
+        th.Property("old_la_code", th.StringType, required=True),
+        th.Property("new_la_code", th.StringType),
+        th.Property("la_name", th.StringType),
+        *[th.Property(out, th.StringType) for out in KS4_HEADLINE_COL_MAP.values()],
+    ).to_dict()
+
+    def get_records(self, context):
+        df = headline_rows(_read_ks4_summary(self.logger), "Local authority")
+        self.logger.info("Emitting %d LA KS4 rows", len(df))
+        for _, row in df.iterrows():
+            yield headline_record(row, ("time_period", "old_la_code", "new_la_code", "la_name"))


 # ── Legacy KS2 (pre-COVID wide format from DfE performance tables) ────────────
@@ -972,6 +975,7 @@ class TapUKEES(Tap):
            LegacyKS4Stream(self),
            EESKs2NationalStream(self),
            EESKs4NationalStream(self),
+            EESKs4LaStream(self),
        ]


@@ -72,6 +72,12 @@ class GIASEstablishmentsStream(Stream):
        th.Property("Northing", th.StringType),
        th.Property("StatutoryLowAge", th.StringType),
        th.Property("StatutoryHighAge", th.StringType),
+        # Matches Ofsted's further-education providers, which some FE
+        # records name by UKPRN only (audit H4).
+        th.Property("UKPRN", th.StringType),
+        # "ISI" for independent schools the Independent Schools Inspectorate
+        # inspects; "Ofsted" or "Not applicable" otherwise (audit H4).
+        th.Property("InspectorateName (name)", th.StringType),
    ).to_dict()

    def get_records(self, context):
@@ -0,0 +1,188 @@
+"""Ofsted's further education and skills management information (audit H4).
+
+Ofsted publishes FE inspection outcomes on one gov.uk page per academic year
+("Further education and skills inspections and outcomes: management
+information from November 2025 to August 2026"). Each page carries "as at"
+files (one row per provider, with its latest inspection under the previous
+framework) and cumulative files of renewed-framework inspections, i.e. report
+cards ("…_10_November_2025_to_31_January_2026.csv", "…_to_28_February_2026.csv",
+…). Monitoring-visit files are a different series.
+
+Free of the Singer SDK so CI's pytest, which installs only the backend's
+requirements, can load it.
+"""
+from __future__ import annotations
+
+import io
+import re
+from datetime import date
+from typing import Callable, Iterator
+
+import pandas as pd
+
+SEARCH_URL = "https://www.gov.uk/api/search.json"
+SEARCH_QUERY = "further education and skills inspections and outcomes management information"
+PAGE_TITLE_PREFIX = "Further education and skills inspections and outcomes: management information from "
+# Renewed-framework (report card) inspections began on 10 November 2025.
+RENEWED_FRAMEWORK_START = date(2025, 11, 1)
+
+_MONTHS = {name: number for number, name in enumerate(
+    ["january", "february", "march", "april", "may", "june", "july",
+     "august", "september", "october", "november", "december"], start=1)}
+_TITLE_START = re.compile(r"from ([A-Za-z]+) (\d{4})")
+_CSV_LINK = re.compile(r'https://assets\.publishing\.service\.gov\.uk/[^"\s<>]+\.csv')
+_AS_AT = re.compile(r"further_education_and_skills_-_+as_at_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$")
+_PERIOD = re.compile(
+    r"further_education_and_skills_-_+(\d{1,2})_([A-Za-z]+)_(\d{4})_to_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$")
+
+# Ofsted column → Singer field.
+PROVIDER_COLUMNS = {
+    "Provider URN": "provider_urn",
+    "Provider UKPRN": "provider_ukprn",
+    "Provider name": "provider_name",
+    "Provider type": "provider_type",
+    "Has the provider received a full inspection under the renewed EIF?": "renewed_eif_full_inspection",
+    "First day of inspection of latest inspection before renewed EIF": "graded_inspection_date",
+    "Latest overall effectiveness before renewed EIF": "overall_effectiveness",
+    "Date of latest short inspection before renewed EIF (if relevant)": "short_inspection_date",
+}
+INSPECTION_COLUMNS = {
+    "Provider URN": "provider_urn",
+    "Provider UKPRN": "provider_ukprn",
+    "Inspection number": "inspection_number",
+    "First day of inspection": "first_day_of_inspection",
+    "Date published": "date_published",
+    "Safeguarding standards": "safeguarding_standards",
+}
+
+
+def _date(day: str, month: str, year: str) -> date | None:
+    number = _MONTHS.get(month.lower())
+    return date(int(year), number, int(day)) if number else None
+
+
+def _page_start(title: str) -> date | None:
+    match = _TITLE_START.search(title)
+    number = _MONTHS.get(match.group(1).lower()) if match else None
+    return date(int(match.group(2)), number, 1) if number else None
+
+
+def _dated_pages(results: list[dict]) -> list[tuple[date, str]]:
+    pages = []
+    for result in results:
+        title = result.get("title", "")
+        start = _page_start(title) if title.startswith(PAGE_TITLE_PREFIX) else None
+        if start:
+            pages.append((start, result["link"]))
+    return sorted(pages, reverse=True)
+
+
+def fe_pages(results: list[dict]) -> list[str]:
+    """gov.uk paths of the FE MI pages, newest academic year first."""
+    return [link for _, link in _dated_pages(results)]
+
+
+def renewed_framework_pages(results: list[dict]) -> list[str]:
+    """The pages that can hold report cards: from November 2025 onwards."""
+    return [link for start, link in _dated_pages(results) if start >= RENEWED_FRAMEWORK_START]
+
+
+def _latest(html: str, pattern: re.Pattern, date_groups: slice) -> str | None:
+    best = None
+    for link in set(_CSV_LINK.findall(html)):
+        match = pattern.search(link)
+        when = _date(*match.groups()[date_groups]) if match else None
+        if when and (best is None or when > best[0]):
+            best = (when, link)
+    return best[1] if best else None
+
+
+def latest_as_at_csv(html: str) -> str | None:
+    """The newest "as at" CSV on a page."""
+    return _latest(html, _AS_AT, slice(0, 3))
+
+
+def latest_period_csv(html: str) -> str | None:
+    """The renewed-framework inspections CSV with the latest end date on a page."""
+    return _latest(html, _PERIOD, slice(3, 6))
+
+
+def providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None:
+    """The newest "as at" CSV, from the newest page that has one."""
+    for page in pages:
+        url = latest_as_at_csv(fetch(page))
+        if url:
+            return url
+    return None
+
+
+def read_mi_csv(content: bytes) -> pd.DataFrame:
+    """Ofsted's MI CSVs: UTF-8 with a BOM, or Windows-1252. The header is the
+    line naming "Provider URN"."""
+    try:
+        text = content.decode("utf-8-sig")
+    except UnicodeDecodeError:
+        text = content.decode("cp1252")
+    lines = text.splitlines()
+    start = next((i for i, line in enumerate(lines[:20]) if "Provider URN" in line), 0)
+    return pd.read_csv(io.StringIO("\n".join(lines[start:])), dtype=str, keep_default_na=False)
+
+
+def records(df: pd.DataFrame, columns: dict[str, str], key: str) -> Iterator[dict]:
+    """Singer records: the mapped columns, stripped. Rows without `key` are skipped."""
+    present = {source: field for source, field in columns.items() if source in df.columns}
+    for _, row in df.iterrows():
+        record = {field: str(row[source]).strip() for source, field in present.items()}
+        if record.get(key):
+            yield record
+
+
+# requests' exceptions are OSErrors; a body that is not JSON is a ValueError.
+_UNAVAILABLE = (OSError, ValueError)
+
+
+def provider_records(
+    search: Callable[[], list[dict]],
+    fetch_page: Callable[[str], str],
+    fetch_csv: Callable[[str], bytes],
+    log: Callable[[str], None],
+) -> Iterator[dict]:
+    """The newest "as at" file's provider records. gov.uk being unreachable
+    yields nothing, logged: the FE streams share the monthly Ofsted extract
+    with the school MI and must not fail it."""
+    try:
+        url = providers_csv_url(fe_pages(search()), fetch_page)
+        if not url:
+            log("No Ofsted FE 'as at' file found")
+            return
+        df = read_mi_csv(fetch_csv(url))
+    except _UNAVAILABLE as exc:
+        log(f"Ofsted FE providers MI unavailable: {exc}")
+        return
+    yield from records(df, PROVIDER_COLUMNS, "provider_urn")
+
+
+def inspection_records(
+    search: Callable[[], list[dict]],
+    fetch_page: Callable[[str], str],
+    fetch_csv: Callable[[str], bytes],
+    log: Callable[[str], None],
+) -> Iterator[dict]:
+    """Renewed-framework inspection records from every page since November
+    2025. A page that cannot be read is logged and skipped; the others load."""
+    try:
+        pages = renewed_framework_pages(search())
+    except _UNAVAILABLE as exc:
+        log(f"Ofsted FE inspections MI unavailable: {exc}")
+        return
+    for page in pages:
+        try:
+            url = latest_period_csv(fetch_page(page))
+            if not url:
+                log(f"No renewed-framework FE inspections file on {page}")
+                continue
+            df = read_mi_csv(fetch_csv(url))
+        except _UNAVAILABLE as exc:
+            log(f"Ofsted FE inspections MI unavailable for {page}: {exc}")
+            continue
+        yield from records(df, INSPECTION_COLUMNS, "inspection_number")
@@ -10,6 +10,15 @@ import requests
 from singer_sdk import Stream, Tap
 from singer_sdk import typing as th

+from tap_uk_ofsted.fe_mi import (
+    INSPECTION_COLUMNS,
+    PROVIDER_COLUMNS,
+    SEARCH_QUERY,
+    SEARCH_URL,
+    inspection_records,
+    provider_records,
+)
+
 GOV_UK_PAGE = (
    "https://www.gov.uk/government/statistical-data-sets/"
    "monthly-management-information-ofsteds-school-inspections-outcomes"
@@ -304,6 +313,62 @@ class OfstedInspectionsStream(Stream):
            self.logger.error("Could not discover Ofsted independent MI download URL")


+def _fe_search_results() -> list[dict]:
+    """gov.uk search results for Ofsted's FE MI pages (one per academic year)."""
+    resp = requests.get(SEARCH_URL, params={"q": SEARCH_QUERY, "count": 50}, timeout=60)
+    resp.raise_for_status()
+    return resp.json().get("results", [])
+
+
+def _gov_uk_page(path: str) -> str:
+    url = path if path.startswith("http") else "https://www.gov.uk" + path
+    resp = requests.get(url, timeout=60)
+    resp.raise_for_status()
+    return resp.text
+
+
+def _gov_uk_file(url: str) -> bytes:
+    resp = requests.get(url, timeout=120)
+    resp.raise_for_status()
+    return resp.content
+
+
+class OfstedFEProvidersStream(Stream):
+    """Ofsted's further education and skills providers: the newest "as at"
+    file, one row per provider with its latest inspection under the previous
+    framework (audit H4). Page and file selection: fe_mi.py."""
+
+    name = "ofsted_fe_providers"
+    primary_keys = ["provider_urn"]
+    replication_key = None
+
+    schema = th.PropertiesList(
+        th.Property("provider_urn", th.StringType, required=True),
+        *[th.Property(field, th.StringType) for field in PROVIDER_COLUMNS.values() if field != "provider_urn"],
+    ).to_dict()
+
+    def get_records(self, context):
+        yield from provider_records(_fe_search_results, _gov_uk_page, _gov_uk_file, self.logger.error)
+
+
+class OfstedFEInspectionsStream(Stream):
+    """Ofsted's renewed-framework FE inspections (report cards), from the
+    latest cumulative file on every page since November 2025, so a report
+    card survives the next academic year's page (audit H4)."""
+
+    name = "ofsted_fe_inspections"
+    primary_keys = ["inspection_number"]
+    replication_key = None
+
+    schema = th.PropertiesList(
+        th.Property("inspection_number", th.StringType, required=True),
+        *[th.Property(field, th.StringType) for field in INSPECTION_COLUMNS.values() if field != "inspection_number"],
+    ).to_dict()
+
+    def get_records(self, context):
+        yield from inspection_records(_fe_search_results, _gov_uk_page, _gov_uk_file, self.logger.warning)
+
+
 class TapUKOfsted(Tap):
    """Singer tap for UK Ofsted Management Information."""

@@ -315,7 +380,7 @@ class TapUKOfsted(Tap):
    ).to_dict()

    def discover_streams(self):
-        return [OfstedInspectionsStream(self)]
+        return [OfstedInspectionsStream(self), OfstedFEProvidersStream(self), OfstedFEInspectionsStream(self)]


 if __name__ == "__main__":
@@ -24,9 +24,9 @@ DAG_ID = re.compile(r'dag_id="([a-z0-9_]+)"')

 DAILY = 'school_data_daily'

-# dim_school reads int_ofsted_latest only when the relation exists
-# (adapter.get_relation), so a missing table is not a failure.
-OPTIONAL_PARENTS = {'int_ofsted_latest'}
+# dim_school reads int_ofsted_latest and int_ofsted_listed only when the
+# relations exist (adapter.get_relation), so a missing table is not a failure.
+OPTIONAL_PARENTS = {'int_ofsted_latest', 'int_ofsted_listed'}


 def model_parents():
@@ -96,3 +96,41 @@ def test_selected_models_only_read_models_that_exist(dag_id):
    missing = {model: sorted(PARENTS[model] - available) for model in sorted(selected)
               if PARENTS[model] - available}
    assert missing == {}, f'{dag_id} builds models whose parents it never builds: {missing}'
+
+
+TABLE = re.compile(r"materialized\s*=\s*['\"]table['\"]")
+
+
+def views():
+    """Staging and intermediate models are views unless configured as tables."""
+    out = set()
+    for path in MODELS.rglob('*.sql'):
+        layer = path.relative_to(MODELS).parts[0]
+        if layer in ('staging', 'intermediate') and not TABLE.search(path.read_text()):
+            out.add(path.stem)
+    return out
+
+
+VIEWS = views()
+
+
+@pytest.mark.parametrize('dag_id', sorted(BUILDS))
+def test_rebuilt_relations_do_not_drop_views_the_build_reads(dag_id):
+    """dbt rebuilds a relation by dropping the old one with CASCADE, which drops
+    every view built on it. A view dropped that way and not rebuilt by the same
+    build is gone when that build reads it: the daily GIAS build rebuilt
+    stg_gias_ukprn and dropped the Ofsted views dim_school reads (audit H4)."""
+    selected = selected_models(BUILDS[dag_id], PARENTS)
+    children = defaultdict(set)
+    for model, ps in PARENTS.items():
+        for p in ps:
+            children[p].add(model)
+    dropped, stack = set(), list(selected)
+    while stack:
+        for child in children[stack.pop()]:
+            if child in VIEWS and child not in dropped:
+                dropped.add(child)
+                stack.append(child)
+    dropped -= selected
+    broken = {m: sorted(PARENTS[m] & dropped) for m in sorted(selected) if PARENTS[m] & dropped}
+    assert broken == {}, f'{dag_id} drops views it then reads: {broken}'
@@ -0,0 +1,67 @@
+"""KS4 school information for 2023/24 exists only in the 2023/24 release,
+whose file uses DfE's older column names. The stream declared only the newer
+ones, so every 2023/24 field loaded as null (audit C2). The headers below are
+DfE's, copied from 202324_information_about_schools_final.csv and
+202425_information_about_schools_final.csv.
+"""
+import importlib.util
+from pathlib import Path
+
+import pytest
+
+MODULE = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-ees'
+          / 'tap_uk_ees' / 'ks4_info.py')
+
+HEADER_2023_24 = (
+    'time_period', 'time_identifier', 'geographic_level', 'country_code', 'country_name',
+    'school_laestab', 'school_urn', 'school_name', 'old_la_code', 'new_la_code', 'la_name',
+    'version', 'establishment_type_group', 'full_address', 'telnum', 'pcon_code', 'pcon_name',
+    'contflag', 'iclose', 'reldenom', 'admpol_pt', 'egender', 'feeder', 'agerange',
+    't_allks_pupils', 't_allks_boys', 't_allks_girls', 't_pupils', 't_boys', 'pt_boys',
+    't_girls', 'pt_girls', 'avg_ks2_scaledscore', 't_prior_lo', 'pt_prior_lo', 't_prior_av',
+    'pt_prior_av', 't_prior_hi', 'pt_prior_hi', 't_disadvantaged', 'pt_disadvantaged',
+    't_not_disadvantaged', 'pt_not_disadvantaged', 't_language_not_english',
+    'pt_language_not_english', 't_language_english', 'pt_language_english',
+    't_language_unknown', 'pt_language_unknown', 't_not_mobile', 'pt_not_mobile',
+    't_sen_with_ehcp', 'pt_sen_with_ehcp', 't_sen', 'pt_sen', 't_sen_no_ehcp',
+    'pt_sen_no_ehcp', 'diffn_att8', 'diffn_p8mea', 'p8_banding',
+)
+
+HEADER_2024_25 = (
+    'time_period', 'time_identifier', 'geographic_level', 'country_code', 'country_name',
+    'school_laestab', 'school_urn', 'school_name', 'old_la_code', 'new_la_code', 'la_name',
+    'version', 'establishment_type_group', 'full_address', 'telnum', 'pcon_code', 'pcon_name',
+    'contflag', 'iclose', 'reldenom', 'admpol_pt', 'egender', 'feeder', 'agerange',
+    'allks_pupil_count', 'allks_boys_count', 'allks_girls_count', 'endks4_pupil_count',
+    'ks2_scaledscore_average', 'sen_with_ehcp_pupil_count', 'sen_with_ehcp_pupil_percent',
+    'sen_pupil_count', 'sen_pupil_percent', 'sen_no_ehcp_pupil_count',
+    'sen_no_ehcp_pupil_percent', 'attainment8_diffn', 'progress8_diffn', 'progress8_banding',
+)
+
+
+@pytest.fixture
+def ks4_info():
+    spec = importlib.util.spec_from_file_location('ks4_info', MODULE)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_every_declared_field_is_in_the_2023_24_file_once_renamed(ks4_info):
+    renamed = {ks4_info.KS4_INFO_RENAMES.get(c, c) for c in HEADER_2023_24}
+    assert set(ks4_info.KS4_INFO_FIELDS) - renamed == set()
+
+
+def test_every_declared_field_is_in_the_2024_25_file(ks4_info):
+    assert set(ks4_info.KS4_INFO_FIELDS) - set(HEADER_2024_25) == set()
+
+
+def test_the_renames_cannot_collide_with_either_file(ks4_info):
+    # An old name in the current file, or a new name already in the old file,
+    # would let a rename overwrite a real column.
+    assert set(ks4_info.KS4_INFO_RENAMES) & set(HEADER_2024_25) == set()
+    assert set(ks4_info.KS4_INFO_RENAMES.values()) & set(HEADER_2023_24) == set()
+
+
+def test_every_rename_names_a_declared_field(ks4_info):
+    assert set(ks4_info.KS4_INFO_RENAMES.values()) <= set(ks4_info.KS4_INFO_FIELDS)
@@ -0,0 +1,72 @@
+"""DfE's KS4 "summary, all state-funded" data set holds England, regional and
+LA rows. The England stream kept only the England row; its LA rows match
+DfE's published LA averages exactly, where the API's own mean was 7 points
+low (audit H2). Values below are DfE's for Kensington and Chelsea (207) and
+Wandsworth (212).
+"""
+import importlib.util
+import io
+from pathlib import Path
+
+import pandas as pd
+import pytest
+
+MODULE = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-ees'
+          / 'tap_uk_ees' / 'ks4_summary.py')
+
+CSV = """time_period,geographic_level,old_la_code,new_la_code,la_name,establishment_type_group,breakdown_topic,breakdown,attainment8_average,progress8_average,engmath_94_percent,engmath_95_percent,ebacc_entering_percent,ebacc_94_percent,ebacc_95_percent,ebacc_aps_average
+202425,National,,,,All state-funded,Total,Total,46.1,z,64.5,45.7,40.5,26.9,17.7,4.1
+202425,National,,,,All state-funded,Sex,Boys,44.1,z,62.0,43.0,38.0,24.0,16.0,3.9
+202425,Regional,,,,All state-funded,Total,Total,47.2,z,66.0,47.0,41.0,28.0,18.0,4.2
+202425,Local authority,207,E09000020,Kensington and Chelsea,All state-funded,Total,Total,54.5,z,77,61.4,45.6,32,26.6,4.89
+202425,Local authority,207,E09000020,Kensington and Chelsea,All state-funded,Sex,Girls,57.0,z,80,64.0,48.0,35,28.0,5.1
+202324,Local authority,207,E09000020,Kensington and Chelsea,All state-funded,Total,Total,54.5,0.29,76,60.0,44.0,31,25.0,4.8
+202425,Local authority,212,E09000032,Wandsworth,All state-funded,Total,Total,51.8,z,72,55.0,50.0,33,24.0,4.6
+202425,Local authority,212,E09000032,Wandsworth,Academies and free schools,Total,Total,52.0,z,73,56.0,51.0,34,25.0,4.7
+"""
+
+LA_KEYS = ('time_period', 'old_la_code', 'new_la_code', 'la_name')
+
+
+def _df():
+    return pd.read_csv(io.StringIO(CSV), dtype=str, keep_default_na=False)
+
+
+@pytest.fixture
+def summary():
+    spec = importlib.util.spec_from_file_location('ks4_summary', MODULE)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_la_rows_are_one_per_la_and_year(summary):
+    rows = summary.headline_rows(_df(), 'Local authority')
+    assert sorted(zip(rows['time_period'], rows['old_la_code'])) == [
+        ('202324', '207'), ('202425', '207'), ('202425', '212')]
+
+
+def test_an_la_record_carries_codes_name_and_the_headline_measures(summary):
+    rows = summary.headline_rows(_df(), 'Local authority')
+    row = rows[(rows['time_period'] == '202425') & (rows['old_la_code'] == '207')].iloc[0]
+    assert summary.headline_record(row, LA_KEYS) == {
+        'time_period': '202425', 'old_la_code': '207', 'new_la_code': 'E09000020',
+        'la_name': 'Kensington and Chelsea',
+        'attainment_8_score': '54.5', 'progress_8_score': 'z',
+        'english_maths_standard_pass_pct': '77', 'english_maths_strong_pass_pct': '61.4',
+        'ebacc_entry_pct': '45.6', 'ebacc_standard_pass_pct': '32',
+        'ebacc_strong_pass_pct': '26.6', 'ebacc_avg_score': '4.89',
+    }
+
+
+def test_the_england_rows_are_one_per_year(summary):
+    rows = summary.headline_rows(_df(), 'National')
+    assert list(rows['time_period']) == ['202425']
+    assert summary.headline_record(rows.iloc[0], ('time_period',))['attainment_8_score'] == '46.1'
+
+
+def test_column_names_and_labels_match_whatever_their_case(summary):
+    df = _df()
+    df.columns = [c.upper() for c in df.columns]
+    df['GEOGRAPHIC_LEVEL'] = df['GEOGRAPHIC_LEVEL'].str.upper()
+    assert len(summary.headline_rows(df, 'Local authority')) == 3
@@ -0,0 +1,100 @@
+"""DfE re-publishes earlier years inside later KS4 releases.
+
+The 2024/25 results file holds 2022/23, 2023/24 and 2024/25 under current
+column names. The 2023/24 release's own file, re-issued in March 2026 under
+older names, was read after it and overwrote every 2023/24 row with blanks
+(audit C2). For the KS4 results stream the newest release owns every year it
+contains.
+"""
+import importlib.util
+import re
+from pathlib import Path
+
+import pandas as pd
+import pytest
+
+TAP_DIR = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-ees'
+           / 'tap_uk_ees')
+
+
+@pytest.fixture
+def precedence():
+    spec = importlib.util.spec_from_file_location(
+        'release_precedence', TAP_DIR / 'release_precedence.py')
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def _release(period):
+    return {'id': f'release-{period}', 'time_period': period}
+
+
+def test_releases_are_taken_newest_first_whatever_order_the_api_gives(precedence):
+    releases = [_release('202223'), _release('202425'), _release(None), _release('202324')]
+    ordered = precedence.newest_first(releases)
+    assert [r['time_period'] for r in ordered] == ['202425', '202324', '202223', None]
+
+
+def test_a_year_a_newer_release_supplied_is_dropped_from_an_older_one(precedence):
+    newer = pd.DataFrame({'time_period': ['202425', '202324', '202223'], 'school_urn': ['1'] * 3})
+    older = pd.DataFrame({'time_period': ['202324', '202324', '201920'], 'school_urn': ['1', '2', '1']})
+
+    kept, skipped = precedence.drop_owned_periods(older, precedence.periods_in(newer))
+
+    assert list(kept['time_period']) == ['201920']
+    assert skipped == {'202324': 2}
+
+
+def test_periods_match_despite_surrounding_spaces(precedence):
+    owned = precedence.periods_in(pd.DataFrame({'time_period': [' 202324 ']}))
+    kept, skipped = precedence.drop_owned_periods(pd.DataFrame({'time_period': ['202324']}), owned)
+    assert owned == {'202324'}
+    assert kept.empty
+    assert skipped == {'202324': 1}
+
+
+def test_nothing_is_dropped_before_any_year_is_owned(precedence):
+    df = pd.DataFrame({'time_period': ['202324'], 'school_urn': ['1']})
+    kept, skipped = precedence.drop_owned_periods(df, set())
+    assert kept.equals(df)
+    assert skipped == {}
+
+
+def test_a_file_without_time_period_is_left_alone(precedence):
+    df = pd.DataFrame({'school_urn': ['1']})
+    kept, skipped = precedence.drop_owned_periods(df, {'202324'})
+    assert kept.equals(df)
+    assert skipped == {}
+    assert precedence.periods_in(df) == set()
+
+
+def test_only_the_ks4_results_stream_opts_in():
+    # A general rule would wipe KS2: the 2024/25 KS2 file holds 98,448 of the
+    # 955,956 rows the 2023/24 release has for 2023/24.
+    source = (TAP_DIR / 'tap.py').read_text()
+    opted_in = [chunk.split('(')[0] for chunk in source.split('\nclass ')[1:]
+                if re.search(r'_newest_release_owns_period\s*=\s*True', chunk)]
+    assert opted_in == ['EESKS4PerformanceStream']
+
+
+@pytest.mark.parametrize('slug, period', [
+    ('2024-25', '202425'),
+    ('2024-25-revised', '202425'),
+    ('2025-26-provisional', '202526'),
+    ('latest', None),
+    ('', None),
+])
+def test_a_release_slug_gives_its_year_whatever_its_suffix(precedence, slug, period):
+    # KS2 already publishes "2024-25-revised" and "2025-26-provisional". An
+    # unread suffix sent the release last, behind the provisional one for the
+    # same year, which then owned the year and dropped every revised row.
+    assert precedence.slug_to_time_period(slug) == period
+
+
+def test_within_a_year_the_api_order_is_kept(precedence):
+    # The API lists a year's revised release before its first release.
+    releases = [{'id': 'revised', 'time_period': '202526'},
+                {'id': 'first', 'time_period': '202526'},
+                {'id': 'older', 'time_period': '202425'}]
+    assert [r['id'] for r in precedence.newest_first(releases)] == ['revised', 'first', 'older']
@@ -0,0 +1,61 @@
+"""Every field the GIAS establishments stream declares must be a column of the
+extract GIAS publishes, or it silently loads as null. H4 adds UKPRN (matches
+Ofsted's further-education providers) and InspectorateName (name) ("ISI" for
+about 1,340 independent schools). HEADER is GIAS's own, from
+edubasealldata20261013.csv.
+"""
+import re
+from pathlib import Path
+
+TAP = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-gias'
+       / 'tap_uk_gias' / 'tap.py')
+
+HEADER = (
+    'URN', 'LA (code)', 'LA (name)', 'EstablishmentNumber', 'EstablishmentName',
+    'TypeOfEstablishment (code)', 'TypeOfEstablishment (name)', 'EstablishmentTypeGroup (code)',
+    'EstablishmentTypeGroup (name)', 'EstablishmentStatus (code)', 'EstablishmentStatus (name)',
+    'ReasonEstablishmentOpened (code)', 'ReasonEstablishmentOpened (name)', 'OpenDate',
+    'ReasonEstablishmentClosed (code)', 'ReasonEstablishmentClosed (name)', 'CloseDate',
+    'PhaseOfEducation (code)', 'PhaseOfEducation (name)', 'StatutoryLowAge', 'StatutoryHighAge',
+    'Boarders (code)', 'Boarders (name)', 'NurseryProvision (name)', 'OfficialSixthForm (code)',
+    'OfficialSixthForm (name)', 'Gender (code)', 'Gender (name)', 'ReligiousCharacter (code)',
+    'ReligiousCharacter (name)', 'ReligiousEthos (name)', 'Diocese (code)', 'Diocese (name)',
+    'AdmissionsPolicy (code)', 'AdmissionsPolicy (name)', 'SchoolCapacity', 'SpecialClasses (code)',
+    'SpecialClasses (name)', 'CensusDate', 'NumberOfPupils', 'NumberOfBoys', 'NumberOfGirls',
+    'PercentageFSM', 'TrustSchoolFlag (code)', 'TrustSchoolFlag (name)', 'Trusts (code)',
+    'Trusts (name)', 'SchoolSponsorFlag (name)', 'SchoolSponsors (name)', 'FederationFlag (name)',
+    'Federations (code)', 'Federations (name)', 'UKPRN', 'FEHEIdentifier',
+    'FurtherEducationType (name)', 'LastChangedDate', 'Street', 'Locality', 'Address3', 'Town',
+    'County (name)', 'Postcode', 'SchoolWebsite', 'TelephoneNum', 'HeadTitle (name)',
+    'HeadFirstName', 'HeadLastName', 'HeadPreferredJobTitle', 'BSOInspectorateName (name)',
+    'InspectorateReport', 'DateOfLastInspectionVisit', 'NextInspectionVisit', 'TeenMoth (name)',
+    'TeenMothPlaces', 'CCF (name)', 'SENPRU (name)', 'EBD (name)', 'PlacesPRU', 'FTProv (name)',
+    'EdByOther (name)', 'Section41Approved (name)', 'SEN1 (name)', 'SEN2 (name)', 'SEN3 (name)',
+    'SEN4 (name)', 'SEN5 (name)', 'SEN6 (name)', 'SEN7 (name)', 'SEN8 (name)', 'SEN9 (name)',
+    'SEN10 (name)', 'SEN11 (name)', 'SEN12 (name)', 'SEN13 (name)',
+    'TypeOfResourcedProvision (name)', 'ResourcedProvisionOnRoll', 'ResourcedProvisionCapacity',
+    'SenUnitOnRoll', 'SenUnitCapacity', 'GOR (code)', 'GOR (name)',
+    'DistrictAdministrative (code)', 'DistrictAdministrative (name)', 'AdministrativeWard (code)',
+    'AdministrativeWard (name)', 'ParliamentaryConstituency (code)',
+    'ParliamentaryConstituency (name)', 'UrbanRural (code)', 'UrbanRural (name)',
+    'GSSLACode (name)', 'Easting', 'Northing', 'MSOA (name)', 'LSOA (name)',
+    'InspectorateName (name)', 'SENStat', 'SENNoStat', 'BoardingEstablishment (name)', 'PropsName',
+    'PreviousLA (code)', 'PreviousLA (name)', 'PreviousEstablishmentNumber', 'Country (name)',
+    'UPRN', 'SiteName', 'QABName (code)', 'QABName (name)', 'EstablishmentAccredited (code)',
+    'EstablishmentAccredited (name)', 'QABReport', 'CHNumber', 'MSOA (code)', 'LSOA (code)', 'FSM',
+    'AccreditationExpiryDate',
+)
+
+
+def declared():
+    source = TAP.read_text()
+    block = source[source.index('class GIASEstablishmentsStream'):source.index('class GIASLinksStream')]
+    return re.findall(r'th\.Property\("([^"]+)"', block)
+
+
+def test_every_declared_field_is_a_gias_column():
+    assert set(declared()) - set(HEADER) == set()
+
+
+def test_the_inspection_fields_are_declared():
+    assert {'UKPRN', 'InspectorateName (name)'} <= set(declared())
@@ -0,0 +1,165 @@
+"""Ofsted publishes further-education inspection outcomes on a gov.uk page per
+academic year, separate from the school MI the tap already reads (audit H4).
+The links below are the real ones from the 2025/26 page; the headers are
+Ofsted's own.
+"""
+import importlib.util
+from pathlib import Path
+
+import pandas as pd
+import pytest
+
+MODULE = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-ofsted'
+          / 'tap_uk_ofsted' / 'fe_mi.py')
+
+A = 'https://assets.publishing.service.gov.uk/media/'
+PAGE_2025_26 = f'''
+<a href="{A}6989b16dc106cc5501b6e4c3/Management_information_-_further_education_and_skills_-_as_at_31_January_2026.csv">x</a>
+<a href="{A}6989b18cd3f57710b50a9b92/Management_information_-_further_education_and_skills_-_10_November_2025_to_31_January_2026.csv">x</a>
+<a href="{A}69aec8d6d620c14fa183eeb6/Management_information_-_further_education_and_skills_-__10_November_2025_to_28_February_2026.csv">x</a>
+<a href="{A}69aec8fcc78869bf8eb8a54a/Management_information_-_further_education_and_skills_-_as_at_28_February_2026.csv">x</a>
+<a href="{A}69aef091bde9c3f213c89a1e/Management_information_-_further_education_and_skills_-_as_at_31_January_2026.ods">x</a>
+<a href="{A}6a9fec1e92e72b8ac437eef3/Management_information_-_further_education_and_skills_-_as_at_31_August_2026.csv">x</a>
+<a href="{A}6a9fed6b5f6e942efe37eed4/Management_information_-_new_provider_monitoring_inspections_-_10_November_2025_to_31_August_2026.csv">x</a>
+<a href="{A}6a9fed815f6e942efe37eed5/Management_information_-_monitoring_inspections_-_10_November_2025_to_31_August_2026.csv">x</a>
+<a href="{A}6a9feda65f6e942efe37eed6/Management_information_-_further_education_and_skills_-_10_November_2025_to_31_August_2026.csv">x</a>
+'''
+
+SEARCH_RESULTS = [
+    {'title': 'Further education and skills inspections and outcomes: management information from September 2024 to August 2025',
+     'link': '/government/statistical-data-sets/fe-2024-25'},
+    {'title': 'Further education and skills inspections and outcomes: management information from November 2025 to August 2026',
+     'link': '/government/statistical-data-sets/fe-2025-26'},
+    {'title': 'Further education and skills inspections and outcomes: management information from September 2023 to August 2024',
+     'link': '/government/statistical-data-sets/fe-2023-24'},
+    {'title': 'Monthly management information: Ofsted\'s school inspections outcomes',
+     'link': '/government/statistical-data-sets/schools'},
+]
+
+PROVIDER_HEADER = (
+    'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group',
+    'Local authority', 'Region', 'Ofsted region',
+    'Has the provider received a full inspection under the renewed EIF?',
+    'Has the provider received a new provider monitoring inspection?',
+    'Date of latest short inspection before renewed EIF (if relevant)',
+    'Inspection number of latest full inspection before renewed EIF',
+    'First day of inspection of latest inspection before renewed EIF',
+    'Latest overall effectiveness before renewed EIF',
+    'Latest quality of education before renewed EIF',
+    'Latest behaviour and attitudes before renewed EIF',
+    'Latest personal development before renewed EIF',
+    'Latest leadership and management before renewed EIF',
+    'Latest contribution to meeting skills needs before renewed EIF',
+    'Latest safeguarding before renewed EIF', '',
+)
+
+INSPECTION_HEADER = (
+    'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group',
+    'Local authority', 'Region', 'Ofsted region', 'Inspection number', 'First day of inspection',
+    'Date published', 'Safeguarding standards', 'Inclusion', 'Leadership and governance',
+    'Contribution to meeting skills needs',
+    'Education programmes for young people - curriculum, teaching and training',
+    'Education programmes for young people - achievement',
+    'Education programmes for young people - participation and development',
+    'Adult learning programmes - curriculum, teaching and training',
+    'Adult learning programmes - achievement',
+    'Adult learning programmes - participation and development',
+    'Apprenticeships - curriculum, teaching and training', 'Apprenticeships - achievement',
+    'Apprenticeships - participation and development',
+    'Provision for learners and apprentices with high needs - curriculum, teaching and training',
+    'Provision for learners and apprentices with high needs - achievement',
+    'Provision for learners and apprentices with high needs - participation and development', '',
+)
+
+
+@pytest.fixture
+def fe():
+    spec = importlib.util.spec_from_file_location('fe_mi', MODULE)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_the_newest_as_at_file_is_chosen(fe):
+    assert fe.latest_as_at_csv(PAGE_2025_26).endswith('_as_at_31_August_2026.csv')
+
+
+def test_the_latest_cumulative_inspections_file_is_chosen_not_a_monitoring_one(fe):
+    assert fe.latest_period_csv(PAGE_2025_26) == (
+        A + '6a9feda65f6e942efe37eed6/Management_information_-_further_education_and_skills_'
+            '-_10_November_2025_to_31_August_2026.csv')
+
+
+def test_a_doubled_underscore_file_name_still_parses(fe):
+    page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'August' not in line)
+    assert fe.latest_period_csv(page).endswith('__10_November_2025_to_28_February_2026.csv')
+
+
+def test_monitoring_files_alone_give_nothing(fe):
+    page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'monitoring' in line)
+    assert fe.latest_period_csv(page) is None
+    assert fe.latest_as_at_csv(page) is None
+
+
+def test_pages_are_the_fe_series_newest_first(fe):
+    assert fe.fe_pages(SEARCH_RESULTS) == [
+        '/government/statistical-data-sets/fe-2025-26',
+        '/government/statistical-data-sets/fe-2024-25',
+        '/government/statistical-data-sets/fe-2023-24',
+    ]
+
+
+def test_report_cards_are_read_only_from_renewed_framework_pages(fe):
+    # Earlier pages hold previous-framework inspection lists, not report cards.
+    assert fe.renewed_framework_pages(SEARCH_RESULTS) == ['/government/statistical-data-sets/fe-2025-26']
+
+
+def test_providers_fall_back_to_the_previous_page(fe):
+    pages = {'/new': '<p>No files yet</p>', '/old': PAGE_2025_26}
+    assert fe.providers_csv_url(['/new', '/old'], pages.__getitem__).endswith('_as_at_31_August_2026.csv')
+
+
+@pytest.mark.parametrize('columns, header', [('PROVIDER_COLUMNS', PROVIDER_HEADER),
+                                             ('INSPECTION_COLUMNS', INSPECTION_HEADER)])
+def test_every_mapped_column_is_in_ofsteds_file(fe, columns, header):
+    assert set(getattr(fe, columns)) - set(header) == set()
+
+
+def test_csv_reading_and_records(fe):
+    content = ('' + ','.join(f'"{c}"' for c in PROVIDER_HEADER[:14]) + '\n'
+               + '145175,10005429,Queen Mary\'s College,Sixth form college,Sixth form colleges,Hampshire,'
+                 'South East,South East,No,No,-,10227183,06/12/2022,2\n'
+               + ',,Blank provider,,,,,,,,,,,\n').encode('utf-8')
+    rows = list(fe.records(fe.read_mi_csv(content), fe.PROVIDER_COLUMNS, 'provider_urn'))
+    assert rows == [{
+        'provider_urn': '145175', 'provider_ukprn': '10005429', 'provider_name': "Queen Mary's College",
+        'provider_type': 'Sixth form college', 'renewed_eif_full_inspection': 'No',
+        'graded_inspection_date': '06/12/2022', 'overall_effectiveness': '2', 'short_inspection_date': '-',
+    }]
+
+
+def test_a_gov_uk_outage_yields_no_providers_rather_than_failing(fe):
+    # The FE streams share the monthly Ofsted extract with the school MI; an
+    # error here must not fail that extract (review of H4).
+    def down():
+        raise ConnectionError('gov.uk unreachable')
+    logged = []
+    assert list(fe.provider_records(down, lambda page: '', lambda url: b'', logged.append)) == []
+    assert logged
+
+
+def test_one_unreachable_page_does_not_stop_the_other_report_cards(fe):
+    results = [{'title': fe.PAGE_TITLE_PREFIX + 'September 2026 to August 2027', 'link': '/fe-2026-27'},
+               {'title': fe.PAGE_TITLE_PREFIX + 'November 2025 to August 2026', 'link': '/fe-2025-26'}]
+    csv = (','.join(INSPECTION_HEADER[:12]) + '\n'
+           + '143526,10028480,The Autism Project,,,,,,10427540,12/05/2026,22/07/2026,Met\n').encode()
+
+    def page(path):
+        if path == '/fe-2026-27':
+            raise ConnectionError('timeout')
+        return PAGE_2025_26
+
+    logged = []
+    rows = list(fe.inspection_records(lambda: results, page, lambda url: csv, logged.append))
+    assert [r['inspection_number'] for r in rows] == ['10427540']
+    assert logged
@@ -3,7 +3,9 @@
  Casts a string column to numeric, treating any non-numeric value as NULL.
  Handles all EES suppression codes (z, c, x, q, u, etc.) without needing
  an explicit list — any string that doesn't look like a number becomes NULL.
+  A trailing percent sign is accepted: DfE's 2023/24 KS2 information file
+  writes percentages as "34%" (audit C2).
 #}
 {% macro safe_numeric(col) -%}
-    CASE WHEN {{ col }} ~ '^-?[0-9]+(\.[0-9]+)?$' THEN {{ col }}::numeric ELSE NULL END
+    CASE WHEN {{ col }} ~ '^-?[0-9]+(\.[0-9]+)?%?$' THEN rtrim({{ col }}, '%')::numeric ELSE NULL END
 {%- endmacro %}
@@ -0,0 +1,98 @@
+-- Intermediate model: Ofsted further education and skills status per URN
+-- (audit H4), in int_ofsted_latest's columns so it can be unioned there.
+-- A provider matches a school by Provider URN, else by UKPRN (stg_gias_ukprn).
+-- Rule, in the C1 status rule's terms
+-- (docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md):
+--   a report card on or after the previous-framework inspection wins and
+--   leaves no grade in force; otherwise a previous-framework grade 1-4 is the
+--   current grade, dated by that inspection; the latest visit is the later of
+--   it and any short inspection (whose outcome the MI does not give).
+
+with gias as (
+    select urn, ukprn from {{ ref('stg_gias_ukprn') }}
+),
+
+providers as (
+    select distinct on (g.urn)
+        g.urn,
+        p.graded_inspection_date,
+        p.overall_effectiveness,
+        p.short_inspection_date
+    from gias g
+    join {{ ref('stg_ofsted_fe_providers') }} p
+      on p.provider_urn = g.urn
+      or (g.ukprn is not null and p.provider_ukprn = g.ukprn)
+    order by g.urn, (p.provider_urn = g.urn) desc, p.graded_inspection_date desc nulls last
+),
+
+report_cards as (
+    select distinct on (g.urn)
+        g.urn,
+        c.rc_inspection_date,
+        c.rc_safeguarding_met
+    from gias g
+    join {{ ref('stg_ofsted_fe_inspections') }} c
+      on c.provider_urn = g.urn
+      or (g.ukprn is not null and c.provider_ukprn = g.ukprn)
+    order by g.urn, (c.provider_urn = g.urn) desc, c.rc_inspection_date desc, c.inspection_number desc
+),
+
+combined as (
+    select
+        coalesce(p.urn, c.urn)                                          as urn,
+        p.graded_inspection_date,
+        p.overall_effectiveness,
+        p.short_inspection_date,
+        c.rc_inspection_date                                            as card_date,
+        c.rc_safeguarding_met                                           as card_safeguarding_met
+    from providers p
+    full outer join report_cards c on c.urn = p.urn
+),
+
+decided as (
+    select
+        *,
+        card_date is not null
+            and card_date >= coalesce(greatest(graded_inspection_date, short_inspection_date), card_date)
+            as report_card_latest
+    from combined
+)
+
+select
+    urn,
+    case when report_card_latest then card_date
+         else greatest(graded_inspection_date, short_inspection_date) end              as latest_visit_date,
+    case when report_card_latest then 'report_card'
+         when short_inspection_date > graded_inspection_date then 'short'
+         else 'graded' end                                                              as latest_visit_kind,
+    null::text                                                                          as latest_visit_outcome,
+    case when not report_card_latest then overall_effectiveness end                     as current_grade,
+    case when not report_card_latest then graded_inspection_date end                    as current_grade_date,
+    case when not report_card_latest then 'graded' end                                  as current_grade_basis,
+    graded_inspection_date,
+    null::date                                                                          as ungraded_inspection_date,
+    coalesce(graded_inspection_date, card_date)                                         as inspection_date,
+    null::text                                                                          as inspection_type,
+    'Further education and skills'::text                                                as framework,
+    overall_effectiveness,
+    null::integer                                                                       as quality_of_education,
+    null::integer                                                                       as behaviour_attitudes,
+    null::integer                                                                       as personal_development,
+    null::integer                                                                       as leadership_management,
+    null::integer                                                                       as early_years_provision,
+    null::integer                                                                       as sixth_form_provision,
+    null::text                                                                          as ungraded_outcome,
+    null::integer                                                                       as ungraded_grade,
+    case when report_card_latest then card_safeguarding_met end                         as rc_safeguarding_met,
+    null::integer                                                                       as rc_inclusion,
+    null::integer                                                                       as rc_curriculum_teaching,
+    null::integer                                                                       as rc_achievement,
+    null::integer                                                                       as rc_attendance_behaviour,
+    null::integer                                                                       as rc_personal_development,
+    null::integer                                                                       as rc_leadership_governance,
+    null::integer                                                                       as rc_early_years,
+    null::integer                                                                       as rc_sixth_form,
+    case when report_card_latest then card_date end                                     as rc_inspection_date,
+    null::text                                                                          as report_url
+from decided
+where report_card_latest or overall_effectiveness between 1 and 4
@@ -0,0 +1,128 @@
+version: 2
+
+unit_tests:
+  - name: int_ofsted_fe_grade_only
+    description: Queen Mary's College (145175), Good on 6 December 2022, no report card.
+    model: int_ofsted_fe_latest
+    given:
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 145175, ukprn: '10005429'}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 145175, provider_ukprn: '10005429', graded_inspection_date: '2022-12-06', overall_effectiveness: 2}
+      - input: ref('stg_ofsted_fe_inspections')
+        rows: []
+    expect:
+      rows:
+        - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, rc_inspection_date: null, framework: 'Further education and skills'}
+
+  - name: int_ofsted_fe_short_inspection_is_a_visit_not_a_grade
+    model: int_ofsted_fe_latest
+    given:
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 1, ukprn: null}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 1, graded_inspection_date: '2019-03-05', overall_effectiveness: 2, short_inspection_date: '2023-02-01'}
+      - input: ref('stg_ofsted_fe_inspections')
+        rows: []
+    expect:
+      rows:
+        - {urn: 1, latest_visit_date: '2023-02-01', latest_visit_kind: short, current_grade: 2, current_grade_date: '2019-03-05', current_grade_basis: graded}
+
+  - name: int_ofsted_fe_newer_report_card_wins
+    description: The Autism Project - CareTrade (143526). Good 2021; report card 12 May 2026, safeguarding met.
+    model: int_ofsted_fe_latest
+    given:
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 143526, ukprn: '10000002'}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 143526, provider_ukprn: '10000002', graded_inspection_date: '2021-11-16', overall_effectiveness: 2}
+      - input: ref('stg_ofsted_fe_inspections')
+        rows:
+          - {inspection_number: '10380001', provider_urn: 143526, provider_ukprn: '10000002', rc_inspection_date: '2026-05-12', rc_safeguarding_met: true}
+    expect:
+      rows:
+        - {urn: 143526, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, current_grade: null, current_grade_date: null, current_grade_basis: null, rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, overall_effectiveness: 2}
+
+  - name: int_ofsted_fe_report_card_only
+    model: int_ofsted_fe_latest
+    given:
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 150042, ukprn: null}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 150042, graded_inspection_date: null, overall_effectiveness: null}
+      - input: ref('stg_ofsted_fe_inspections')
+        rows:
+          - {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false}
+    expect:
+      rows:
+        - {urn: 150042, latest_visit_date: '2026-01-20', latest_visit_kind: report_card, current_grade: null, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false}
+
+  - name: int_ofsted_fe_listed_never_inspected_has_no_row
+    model: int_ofsted_fe_latest
+    given:
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 130645, ukprn: null}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 130645, graded_inspection_date: null, overall_effectiveness: null}
+      - input: ref('stg_ofsted_fe_inspections')
+        rows: []
+    expect:
+      rows: []
+
+  - name: int_ofsted_fe_matches_by_ukprn_when_the_urn_differs
+    model: int_ofsted_fe_latest
+    given:
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 5, ukprn: '10099999'}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 9999999, provider_ukprn: '10099999', graded_inspection_date: '2024-01-30', overall_effectiveness: 3}
+      - input: ref('stg_ofsted_fe_inspections')
+        rows: []
+    expect:
+      rows:
+        - {urn: 5, current_grade: 3, current_grade_date: '2024-01-30'}
+
+  - name: int_ofsted_fe_prefers_the_urn_match
+    model: int_ofsted_fe_latest
+    given:
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 6, ukprn: '10088888'}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 6, provider_ukprn: '10077777', graded_inspection_date: '2023-05-01', overall_effectiveness: 1}
+          - {provider_urn: 8888888, provider_ukprn: '10088888', graded_inspection_date: '2024-05-01', overall_effectiveness: 4}
+      - input: ref('stg_ofsted_fe_inspections')
+        rows: []
+    expect:
+      rows:
+        - {urn: 6, current_grade: 1, current_grade_date: '2023-05-01'}
+
+  - name: int_ofsted_fe_report_card_older_than_grade_is_ignored
+    description: Bad data guard. A report card dated before the graded inspection cannot be the latest, and a row must never carry both a grade and a report card.
+    model: int_ofsted_fe_latest
+    given:
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 7, ukprn: null}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 7, graded_inspection_date: '2026-03-01', overall_effectiveness: 2}
+      - input: ref('stg_ofsted_fe_inspections')
+        rows:
+          - {inspection_number: '1', provider_urn: 7, rc_inspection_date: '2026-01-10', rc_safeguarding_met: true}
+    expect:
+      rows:
+        - {urn: 7, latest_visit_kind: graded, current_grade: 2, rc_inspection_date: null, rc_safeguarding_met: null}
@@ -3,6 +3,8 @@
 -- inspection) and the overall grade still in force, if any. A grade is dated
 -- by the inspection that awarded or confirmed it, never by a later visit.
 -- Rule and examples: docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md
+-- Schools with no school-framework record take their further education and
+-- skills status from int_ofsted_fe_latest (same columns, same order; audit H4).

 with inspections as (
    select
@@ -62,51 +64,64 @@ graded as (
                then 'graded_earlier'
        end as grade_case
    from latest
+),
+
+school_status as (
+    select
+        urn,
+        latest_visit_date,
+        latest_visit_kind,
+        case when latest_visit_kind = 'ungraded' then ungraded_outcome end as latest_visit_outcome,
+        case grade_case
+            when 'confirmed' then ungraded_grade
+            when 'graded_latest' then overall_effectiveness
+            when 'graded_earlier' then overall_effectiveness
+        end as current_grade,
+        case grade_case
+            when 'confirmed' then ungraded_inspection_date
+            when 'graded_latest' then graded_inspection_date
+            when 'graded_earlier' then graded_inspection_date
+        end as current_grade_date,
+        case grade_case
+            when 'confirmed' then 'confirmed'
+            when 'graded_latest' then 'graded'
+            when 'graded_earlier' then 'graded'
+        end as current_grade_basis,
+        graded_inspection_date,
+        ungraded_inspection_date,
+        inspection_date,
+        inspection_type,
+        framework,
+        overall_effectiveness,
+        quality_of_education,
+        behaviour_attitudes,
+        personal_development,
+        leadership_management,
+        early_years_provision,
+        sixth_form_provision,
+        ungraded_outcome,
+        ungraded_grade,
+        rc_safeguarding_met,
+        rc_inclusion,
+        rc_curriculum_teaching,
+        rc_achievement,
+        rc_attendance_behaviour,
+        rc_personal_development,
+        rc_leadership_governance,
+        rc_early_years,
+        rc_sixth_form,
+        rc_inspection_date,
+        report_url
+    from graded
+),
+
+fe_status as (
+    -- Same columns in the same order as school_status.
+    select f.*
+    from {{ ref('int_ofsted_fe_latest') }} f
+    where not exists (select 1 from school_status s where s.urn = f.urn)
 )

-select
-    urn,
-    latest_visit_date,
-    latest_visit_kind,
-    case when latest_visit_kind = 'ungraded' then ungraded_outcome end as latest_visit_outcome,
-    case grade_case
-        when 'confirmed' then ungraded_grade
-        when 'graded_latest' then overall_effectiveness
-        when 'graded_earlier' then overall_effectiveness
-    end as current_grade,
-    case grade_case
-        when 'confirmed' then ungraded_inspection_date
-        when 'graded_latest' then graded_inspection_date
-        when 'graded_earlier' then graded_inspection_date
-    end as current_grade_date,
-    case grade_case
-        when 'confirmed' then 'confirmed'
-        when 'graded_latest' then 'graded'
-        when 'graded_earlier' then 'graded'
-    end as current_grade_basis,
-    graded_inspection_date,
-    ungraded_inspection_date,
-    inspection_date,
-    inspection_type,
-    framework,
-    overall_effectiveness,
-    quality_of_education,
-    behaviour_attitudes,
-    personal_development,
-    leadership_management,
-    early_years_provision,
-    sixth_form_provision,
-    ungraded_outcome,
-    ungraded_grade,
-    rc_safeguarding_met,
-    rc_inclusion,
-    rc_curriculum_teaching,
-    rc_achievement,
-    rc_attendance_behaviour,
-    rc_personal_development,
-    rc_leadership_governance,
-    rc_early_years,
-    rc_sixth_form,
-    rc_inspection_date,
-    report_url
-from graded
+select * from school_status
+union all
+select * from fe_status
@@ -8,6 +8,8 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 102408, graded_inspection_date: '2025-06-17', ungraded_inspection_date: '2020-02-06', overall_effectiveness: null, ungraded_grade: 2, ungraded_outcome: 'School remains Good'}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 102408, latest_visit_date: '2025-06-17', latest_visit_kind: graded, latest_visit_outcome: null, current_grade: null, current_grade_date: null, current_grade_basis: null}
@@ -18,6 +20,8 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 1, graded_inspection_date: '2019-06-01', overall_effectiveness: 2}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 1, latest_visit_date: '2019-06-01', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2019-06-01', current_grade_basis: graded}
@@ -29,6 +33,8 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 104762, graded_inspection_date: '2020-01-07', ungraded_inspection_date: '2024-07-18', overall_effectiveness: 2, ungraded_grade: 2, ungraded_outcome: 'School remains Good'}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 104762, latest_visit_date: '2024-07-18', latest_visit_kind: ungraded, latest_visit_outcome: 'School remains Good', current_grade: 2, current_grade_date: '2024-07-18', current_grade_basis: confirmed}
@@ -40,6 +46,8 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 139888, graded_inspection_date: '2020-03-03', ungraded_inspection_date: '2025-05-21', overall_effectiveness: 2, ungraded_grade: null, ungraded_outcome: 'Standards maintained'}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 139888, latest_visit_date: '2025-05-21', latest_visit_kind: ungraded, latest_visit_outcome: 'Standards maintained', current_grade: 2, current_grade_date: '2020-03-03', current_grade_basis: graded}
@@ -51,6 +59,8 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 136454, ungraded_inspection_date: '2024-11-13', ungraded_grade: null, ungraded_outcome: 'Standards maintained'}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 136454, latest_visit_date: '2024-11-13', latest_visit_kind: ungraded, latest_visit_outcome: 'Standards maintained', current_grade: null, current_grade_date: null, current_grade_basis: null}
@@ -62,6 +72,8 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 110048, ungraded_inspection_date: '2023-10-05', ungraded_grade: 2, ungraded_outcome: 'School remains Good', rc_inspection_date: '2026-05-06', rc_inclusion: 3}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 110048, latest_visit_date: '2026-05-06', latest_visit_kind: report_card, latest_visit_outcome: null, current_grade: null, current_grade_date: null, current_grade_basis: null}
@@ -74,6 +86,8 @@ unit_tests:
        rows:
          - {urn: 138186, graded_inspection_date: '2023-06-13', overall_effectiveness: 3}
          - {urn: 138186, graded_inspection_date: '2023-06-13', overall_effectiveness: 3, rc_inspection_date: '2026-06-02', rc_inclusion: 3}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 138186, latest_visit_date: '2026-06-02', latest_visit_kind: report_card, current_grade: null}
@@ -84,6 +98,8 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 2, graded_inspection_date: '2024-03-01', ungraded_inspection_date: '2024-03-01', overall_effectiveness: 1, ungraded_grade: 2, ungraded_outcome: 'School remains Good'}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 2, latest_visit_kind: graded, current_grade: 1, current_grade_basis: graded}
@@ -94,6 +110,8 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 3, graded_inspection_date: '2018-05-01', overall_effectiveness: 9}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 3, latest_visit_kind: graded, current_grade: null}
@@ -108,6 +126,34 @@ unit_tests:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 4, ungraded_inspection_date: '2026-03-02', ungraded_grade: 2, ungraded_outcome: 'School remains Good', rc_inspection_date: '2025-12-01', rc_inclusion: 3}
+      - input: ref('int_ofsted_fe_latest')
+        rows: []
    expect:
      rows:
        - {urn: 4, latest_visit_date: '2026-03-02', latest_visit_kind: ungraded, latest_visit_outcome: 'School remains Good', current_grade: null, current_grade_date: null, current_grade_basis: null}
+
+  - name: fe_status_fills_a_school_with_no_school_record
+    description: Queen Mary's College (145175) has no school-framework record; its FE status passes through (audit H4).
+    model: int_ofsted_latest
+    given:
+      - input: ref('stg_ofsted_inspections')
+        rows: []
+      - input: ref('int_ofsted_fe_latest')
+        rows:
+          - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'}
+    expect:
+      rows:
+        - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'}
+
+  - name: a_school_record_wins_over_an_fe_record
+    model: int_ofsted_latest
+    given:
+      - input: ref('stg_ofsted_inspections')
+        rows:
+          - {urn: 1, graded_inspection_date: '2019-06-01', overall_effectiveness: 2, framework: 'Schools - S5'}
+      - input: ref('int_ofsted_fe_latest')
+        rows:
+          - {urn: 1, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, rc_inspection_date: '2026-05-12', framework: 'Further education and skills'}
+    expect:
+      rows:
+        - {urn: 1, latest_visit_date: '2019-06-01', latest_visit_kind: graded, current_grade: 2, framework: 'Schools - S5'}
@@ -0,0 +1,21 @@
+-- Intermediate model: every URN Ofsted's own data lists, inspected or not
+-- (audit H4). Only these may be called "Not yet inspected". Reads the raw
+-- school MI because stg_ofsted_inspections drops rows with no inspection date.
+
+with school_mi as (
+    select distinct cast(urn as integer) as urn
+    from {{ source('raw', 'ofsted_inspections') }}
+    where urn is not null
+),
+
+fe as (
+    select distinct g.urn
+    from {{ ref('stg_gias_ukprn') }} g
+    join {{ ref('stg_ofsted_fe_providers') }} p
+      on p.provider_urn = g.urn
+      or (g.ukprn is not null and p.provider_ukprn = g.ukprn)
+)
+
+select urn from school_mi
+union
+select urn from fe
@@ -0,0 +1,29 @@
+version: 2
+
+unit_tests:
+  - name: int_ofsted_listed_includes_uninspected_schools_and_fe_providers
+    description: >
+      Abbas and Templecombe (150031) is in Ofsted's school MI with no inspection;
+      Exeter and North Devon Colleges Group (130645) is in the FE MI with none;
+      a UKPRN-only FE match counts too. A school in neither is absent (audit H4).
+    model: int_ofsted_listed
+    given:
+      - input: source('raw', 'ofsted_inspections')
+        rows:
+          - {urn: '150031', inspection_date: 'NULL'}
+          - {urn: '102408', inspection_date: '17/06/2025'}
+      - input: ref('stg_gias_ukprn')
+        rows:
+          - {urn: 130645, ukprn: null}
+          - {urn: 5, ukprn: '10099999'}
+          - {urn: 133058, ukprn: null}
+      - input: ref('stg_ofsted_fe_providers')
+        rows:
+          - {provider_urn: 130645, provider_ukprn: null}
+          - {provider_urn: 9999999, provider_ukprn: '10099999'}
+    expect:
+      rows:
+        - {urn: 150031}
+        - {urn: 102408}
+        - {urn: 130645}
+        - {urn: 5}
@@ -138,7 +138,7 @@ models:
        tests:
          - not_null
          - accepted_values:
-              values: ['report_card', 'graded', 'ungraded']
+              values: ['report_card', 'graded', 'ungraded', 'short']
      - name: current_grade
        tests:
          - accepted_values:
@@ -279,11 +279,24 @@ models:
        tests: [not_null, unique]

  - name: fact_ks4_national_averages
-    description: Computed national KS4 averages (means across state schools in our dataset — not official DfE figures) — one row per academic year
+    description: Official DfE KS4 national headline averages (England, all state-funded schools, all pupils) — one row per academic year
    columns:
      - name: year
        tests: [not_null, unique]

+  - name: fact_ks4_la_averages
+    description: Official DfE KS4 local-authority averages (all state-funded schools, all pupils) — one row per academic year and LA; la_code is the GIAS LA code
+    columns:
+      - name: year
+        tests: [not_null]
+      - name: la_code
+        tests: [not_null]
+      - name: la_name
+        tests: [not_null]
+    tests:
+      - unique:
+          column_name: "year || '-' || la_code"
+
  - name: fact_deprivation
    description: IDACI deprivation index — one row per URN
    columns:
@@ -1,9 +1,10 @@
 -- Mart: Canonical school dimension — one row per active URN

-- int_ofsted_latest is only ref()'d inside a conditional block below (it may not
-- exist before the Ofsted pipeline first runs), so declare the dependency
-- explicitly for dbt's DAG.
+-- int_ofsted_latest and int_ofsted_listed are only ref()'d inside conditional
+-- blocks below (they may not exist before the Ofsted pipeline first runs), so
+-- declare the dependencies explicitly for dbt's DAG.
 -- depends_on: {{ ref('int_ofsted_latest') }}
+-- depends_on: {{ ref('int_ofsted_listed') }}

 with schools as (
    select * from {{ ref('stg_gias_establishments') }}
@@ -15,6 +16,12 @@ with schools as (
    identifier='int_ofsted_latest'
 ) %}

+{% set listed_relation = adapter.get_relation(
+    database=target.database,
+    schema='intermediate',
+    identifier='int_ofsted_listed'
+) %}
+
 select
    s.urn,
    s.local_authority_code * 1000 + s.establishment_number as laestab,
@@ -70,6 +77,14 @@ select
    s.status_code,
    s.nursery_provision,
    s.admissions_policy_code,
+    -- The inspectorate GIAS names ("ISI" for most independent schools) and
+    -- whether Ofsted's own data lists the school, inspected or not (audit H4).
+    s.inspectorate_name as inspectorate,
+    {% if listed_relation is not none %}
+    l.urn is not null as ofsted_listed,
+    {% else %}
+    null::boolean as ofsted_listed,
+    {% endif %}

    -- Latest Ofsted (populated after monthly Ofsted pipeline runs). The grade
    -- still in force and the latest visit, from int_ofsted_latest — never a
@@ -88,6 +103,9 @@ from schools s
 {% if ofsted_relation is not none %}
 left join {{ ref('int_ofsted_latest') }} o on s.urn = o.urn
 {% endif %}
+{% if listed_relation is not none %}
+left join {{ ref('int_ofsted_listed') }} l on s.urn = l.urn
+{% endif %}
 -- 1 = Open; 3 = Open, but proposed to close (still operating; drops out when
 -- GIAS flips to Closed — marts fully rebuild each run).
 where s.status_code in (1, 3)
@@ -14,6 +14,8 @@ unit_tests:
      - input: ref('int_ofsted_latest')
        rows:
          - {urn: 102408, inspection_date: '2025-06-17', overall_effectiveness: null, ungraded_grade: 2, framework: 'Schools - S5', latest_visit_date: '2025-06-17', current_grade: null}
+      - input: ref('int_ofsted_listed')
+        rows: []
    expect:
      rows:
        - {urn: 102408, ofsted_grade: null, ofsted_date: '2025-06-17', ofsted_framework: 'Schools - S5'}
@@ -30,6 +32,30 @@ unit_tests:
      - input: ref('int_ofsted_latest')
        rows:
          - {urn: 139888, inspection_date: '2020-03-03', overall_effectiveness: 2, ungraded_grade: null, framework: 'Schools - S5', latest_visit_date: '2025-05-21', current_grade: 2}
+      - input: ref('int_ofsted_listed')
+        rows: []
    expect:
      rows:
        - {urn: 139888, ofsted_grade: 2, ofsted_date: '2025-05-21'}
+
+  - name: dim_school_carries_inspectorate_and_ofsted_listing
+    description: >
+      Audit H4. GIAS names ISI for Abbey College Cambridge; Ofsted lists Abbas
+      and Templecombe without an inspection; a third school is in neither.
+    model: dim_school
+    given:
+      - input: ref('stg_gias_establishments')
+        rows:
+          - {urn: 136083, school_name: 'Abbey College Cambridge', status_code: 1, school_type_code: 11, local_authority_code: 873, phase_code: 4, inspectorate_name: 'ISI'}
+          - {urn: 150031, school_name: 'Abbas and Templecombe Church School', status_code: 1, school_type_code: 28, local_authority_code: 933, phase_code: 2, inspectorate_name: 'Not applicable'}
+          - {urn: 1, school_name: 'No data school', status_code: 1, school_type_code: 1, local_authority_code: 312, phase_code: 2, inspectorate_name: null}
+      - input: ref('int_ofsted_latest')
+        rows: []
+      - input: ref('int_ofsted_listed')
+        rows:
+          - {urn: 150031}
+    expect:
+      rows:
+        - {urn: 136083, inspectorate: 'ISI', ofsted_listed: false}
+        - {urn: 150031, inspectorate: 'Not applicable', ofsted_listed: true}
+        - {urn: 1, inspectorate: null, ofsted_listed: false}
@@ -0,0 +1,22 @@
+{{ config(materialized='table') }}
+
+-- Mart: OFFICIAL DfE KS4 local-authority averages — one row per academic year
+-- and LA (all state-funded schools, all pupils), from the same EES data set
+-- as fact_ks4_national_averages. Feeds "vs LA avg" on search rows and map
+-- cards, replacing a mean the API took over every school in the LA,
+-- independent and special included (audit H2). la_code is the GIAS LA code.
+
+select
+    year,
+    la_code,
+    la_name,
+    attainment_8_score,
+    progress_8_score,
+    english_maths_standard_pass_pct,
+    english_maths_strong_pass_pct,
+    ebacc_entry_pct,
+    ebacc_standard_pass_pct,
+    ebacc_strong_pass_pct,
+    ebacc_avg_score
+from {{ ref('stg_ees_ks4_la') }}
+order by year, la_code
@@ -24,6 +24,12 @@ sources:
      - name: ofsted_inspections
        description: Ofsted Management Information inspection records

+      - name: ofsted_fe_providers
+        description: Ofsted further education and skills MI, newest "as at" file — one row per provider with its latest inspection under the previous framework
+
+      - name: ofsted_fe_inspections
+        description: Ofsted further education and skills MI — renewed-framework inspections (report cards) since 10 November 2025, one row per inspection
+
      - name: ees_ks2_attainment
        description: KS2 school attainment (long format — one row per school × subject × breakdown)

@@ -79,6 +85,9 @@ sources:
      - name: ees_ks4_national
        description: Official KS4 national headline averages from DfE EES data catalogue — one row per academic year

+      - name: ees_ks4_la
+        description: Official KS4 local-authority averages (all state-funded schools) from the same DfE EES data set as ees_ks4_national — one row per academic year and LA
+
      # Phonics: no school-level data on EES (only national/LA level)

      - name: fbit_finance
@@ -0,0 +1,22 @@
+version: 2
+
+unit_tests:
+  - name: stg_ees_ks2_reads_percentages_written_with_a_sign
+    description: >
+      DfE's 2023/24 KS2 information file writes percentages as "34%", and every
+      2023/24 percentage loaded as null (audit C2). Plain numbers still load;
+      suppression codes stay null. School 147411's 2023/24 figures are DfE's.
+    model: stg_ees_ks2
+    given:
+      - input: source('raw', 'ees_ks2_attainment')
+        rows:
+          - {school_urn: '147411', time_period: '202324', subject: 'Reading, writing and maths', breakdown_topic: 'All pupils', breakdown: 'Total', expected_standard_pupil_percent: '70'}
+          - {school_urn: '100000', time_period: '202425', subject: 'Reading, writing and maths', breakdown_topic: 'All pupils', breakdown: 'Total', expected_standard_pupil_percent: '79'}
+      - input: source('raw', 'ees_ks2_info')
+        rows:
+          - {school_urn: '147411', time_period: '202324', totpups: '818', telig: '112', ptfsm6cla1a: '34%', ptealgrp2: '56%', psenelk: '20%', psenele: 'c', ptmobn: '87.5%'}
+          - {school_urn: '100000', time_period: '202425', totpups: '792', telig: '105', ptfsm6cla1a: '32', ptealgrp2: 'x', psenelk: '14', psenele: '2', ptmobn: '87'}
+    expect:
+      rows:
+        - {urn: 147411, year: 202324, total_pupils: 818, rwm_expected_pct: 70, disadvantaged_pct: 34, eal_pct: 56, sen_support_pct: 20, sen_ehcp_pct: null, stability_pct: 87.5}
+        - {urn: 100000, year: 202425, total_pupils: 792, rwm_expected_pct: 79, disadvantaged_pct: 32, eal_pct: null, sen_support_pct: 14, sen_ehcp_pct: 2, stability_pct: 87}
@@ -0,0 +1,21 @@
+-- Staging model: official DfE KS4 local-authority averages — one row per
+-- academic year and LA (all state-funded schools, all pupils). Same EES data
+-- set as stg_ees_ks4_national. la_code is the GIAS LA code
+-- (dim_location.local_authority_code). Suppressed values ('z', 'x') are
+-- coerced to NULL by safe_numeric.
+
+select
+    cast(trim(time_period) as integer)                        as year,
+    cast(trim(old_la_code) as integer)                        as la_code,
+    trim(la_name)                                             as la_name,
+    {{ safe_numeric('attainment_8_score') }}                  as attainment_8_score,
+    {{ safe_numeric('progress_8_score') }}                    as progress_8_score,
+    {{ safe_numeric('english_maths_standard_pass_pct') }}     as english_maths_standard_pass_pct,
+    {{ safe_numeric('english_maths_strong_pass_pct') }}       as english_maths_strong_pass_pct,
+    {{ safe_numeric('ebacc_entry_pct') }}                     as ebacc_entry_pct,
+    {{ safe_numeric('ebacc_standard_pass_pct') }}             as ebacc_standard_pass_pct,
+    {{ safe_numeric('ebacc_strong_pass_pct') }}               as ebacc_strong_pass_pct,
+    {{ safe_numeric('ebacc_avg_score') }}                     as ebacc_avg_score
+from {{ source('raw', 'ees_ks4_la') }}
+where time_period ~ '^[0-9]+$'
+  and old_la_code ~ '^[0-9]+$'
@@ -0,0 +1,15 @@
+version: 2
+
+unit_tests:
+  - name: stg_ees_ks4_la_casts_codes_years_and_measures
+    description: DfE's LA rows, keyed by the GIAS LA code. Suppressed values ('z') become null.
+    model: stg_ees_ks4_la
+    given:
+      - input: source('raw', 'ees_ks4_la')
+        rows:
+          - {time_period: '202425', old_la_code: '207', new_la_code: 'E09000020', la_name: 'Kensington and Chelsea', attainment_8_score: '54.5', progress_8_score: 'z', english_maths_standard_pass_pct: '77'}
+          - {time_period: '202324', old_la_code: '207', new_la_code: 'E09000020', la_name: 'Kensington and Chelsea', attainment_8_score: '54.5', progress_8_score: '0.29', english_maths_standard_pass_pct: '76'}
+    expect:
+      rows:
+        - {year: 202425, la_code: 207, la_name: 'Kensington and Chelsea', attainment_8_score: 54.5, progress_8_score: null, english_maths_standard_pass_pct: 77}
+        - {year: 202324, la_code: 207, la_name: 'Kensington and Chelsea', attainment_8_score: 54.5, progress_8_score: 0.29, english_maths_standard_pass_pct: 76}
@@ -38,6 +38,7 @@ renamed as (
        "UrbanRural (name)"                             as urban_rural,
        "ParliamentaryConstituency (name)"              as parliamentary_constituency,
        "NurseryProvision (name)"                       as nursery_provision,
+        nullif(trim("InspectorateName (name)"), '')     as inspectorate_name,
        cast(nullif("Easting", '') as integer)          as easting,
        cast(nullif("Northing", '') as integer)         as northing,
        cast(nullif("StatutoryLowAge", '') as integer)  as statutory_low_age,
@@ -0,0 +1,15 @@
+-- URN → UKPRN from the raw GIAS extract, for matching Ofsted's further
+-- education providers (int_ofsted_fe_latest, int_ofsted_listed; audit H4).
+-- A model on the raw source rather than a column of stg_gias_establishments:
+-- a ref to that model would put the Ofsted models into the daily build's
+-- stg_gias_establishments+ selection (pipeline/tests/test_dag_selectors.py).
+-- Built by the monthly Ofsted DAG with the views on it, never by the daily
+-- one: rebuilding a view drops the views on it (CASCADE), and dim_school, which
+-- the daily build makes, reads int_ofsted_latest and int_ofsted_listed.
+
+select
+    cast("URN" as integer)          as urn,
+    nullif(trim("UKPRN"), '')       as ukprn
+from {{ source('raw', 'gias_establishments') }}
+-- target-postgres types URN as bigint (the tap declares an integer).
+where "URN" is not null
@@ -0,0 +1,29 @@
+version: 2
+
+unit_tests:
+  - name: stg_ofsted_fe_providers_parses_grades_and_dates
+    description: Ofsted writes "-" for no grade or date; grades are 1-4.
+    model: stg_ofsted_fe_providers
+    given:
+      - input: source('raw', 'ofsted_fe_providers')
+        rows:
+          - {provider_urn: '145175', provider_ukprn: '10005429', provider_name: "Queen Mary's College", renewed_eif_full_inspection: 'No', graded_inspection_date: '06/12/2022', overall_effectiveness: '2', short_inspection_date: '-'}
+          - {provider_urn: '130645', provider_ukprn: '-', provider_name: 'Exeter and North Devon Colleges Group', renewed_eif_full_inspection: 'No', graded_inspection_date: '-', overall_effectiveness: '-', short_inspection_date: '-'}
+          - {provider_urn: '', provider_ukprn: '10000001', provider_name: 'No URN'}
+    expect:
+      rows:
+        - {provider_urn: 145175, provider_ukprn: '10005429', renewed_eif_full_inspection: false, graded_inspection_date: '2022-12-06', overall_effectiveness: 2, short_inspection_date: null}
+        - {provider_urn: 130645, provider_ukprn: null, renewed_eif_full_inspection: false, graded_inspection_date: null, overall_effectiveness: null, short_inspection_date: null}
+
+  - name: stg_ofsted_fe_inspections_parses_dates_and_safeguarding
+    model: stg_ofsted_fe_inspections
+    given:
+      - input: source('raw', 'ofsted_fe_inspections')
+        rows:
+          - {inspection_number: '10380001', provider_urn: '143526', provider_ukprn: '10000002', first_day_of_inspection: '12/05/2026', date_published: '16/06/2026', safeguarding_standards: 'Met'}
+          - {inspection_number: '10380002', provider_urn: '150042', provider_ukprn: '10000003', first_day_of_inspection: '20/01/2026', date_published: '24/02/2026', safeguarding_standards: 'Not met'}
+          - {inspection_number: '10380003', provider_urn: '1', provider_ukprn: '', first_day_of_inspection: '', date_published: '', safeguarding_standards: ''}
+    expect:
+      rows:
+        - {inspection_number: '10380001', provider_urn: 143526, rc_inspection_date: '2026-05-12', published_date: '2026-06-16', rc_safeguarding_met: true}
+        - {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', published_date: '2026-02-24', rc_safeguarding_met: false}
@@ -0,0 +1,18 @@
+-- Staging model: Ofsted's renewed-framework FE inspections (report cards),
+-- one row per inspection (audit H4). Only the headline is kept: the date and
+-- the safeguarding result.
+
+select
+    trim(inspection_number)                                             as inspection_number,
+    cast(trim(provider_urn) as integer)                                 as provider_urn,
+    nullif(nullif(trim(provider_ukprn), ''), '-')                       as provider_ukprn,
+    to_date(trim(first_day_of_inspection), 'DD/MM/YYYY')                as rc_inspection_date,
+    case when trim(date_published) ~ '^\d{2}/\d{2}/\d{4}$'
+        then to_date(trim(date_published), 'DD/MM/YYYY') end            as published_date,
+    case lower(trim(safeguarding_standards))
+        when 'met' then true
+        when 'not met' then false
+    end                                                                 as rc_safeguarding_met
+from {{ source('raw', 'ofsted_fe_inspections') }}
+where trim(provider_urn) ~ '^[0-9]+$'
+  and trim(first_day_of_inspection) ~ '^\d{2}/\d{2}/\d{4}$'
@@ -0,0 +1,18 @@
+-- Staging model: Ofsted further education and skills providers — the "as at"
+-- MI file, one row per provider with its latest inspection under the previous
+-- framework (audit H4). Ofsted writes '-' for no grade or date.
+
+select
+    cast(trim(provider_urn) as integer)                                          as provider_urn,
+    nullif(nullif(trim(provider_ukprn), ''), '-')                                as provider_ukprn,
+    provider_name,
+    provider_type,
+    lower(trim(renewed_eif_full_inspection)) = 'yes'                             as renewed_eif_full_inspection,
+    case when trim(graded_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$'
+        then to_date(trim(graded_inspection_date), 'DD/MM/YYYY') end             as graded_inspection_date,
+    case when trim(overall_effectiveness) in ('1', '2', '3', '4')
+        then trim(overall_effectiveness)::integer end                            as overall_effectiveness,
+    case when trim(short_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$'
+        then to_date(trim(short_inspection_date), 'DD/MM/YYYY') end              as short_inspection_date
+from {{ source('raw', 'ofsted_fe_providers') }}
+where trim(provider_urn) ~ '^[0-9]+$'
@@ -0,0 +1,16 @@
+-- Where a year's KS2 school information loaded (pupil counts present), its
+-- percentages loaded too. DfE's files give a disadvantaged % for 96–97% of
+-- those schools; 2023/24 loaded none because the file writes "34%" (audit
+-- C2). A year without an information file (2022/23) has no pupil counts and
+-- is skipped.
+
+select
+    year,
+    count(total_pupils) as with_pupils,
+    count(*) filter (where total_pupils is not null and disadvantaged_pct is not null)
+        as with_disadvantaged
+from {{ ref('stg_ees_ks2') }}
+group by year
+having count(total_pupils) >= 1000
+   and count(*) filter (where total_pupils is not null and disadvantaged_pct is not null)
+       < 0.9 * count(total_pupils)
@@ -0,0 +1,13 @@
+-- DfE publishes an average for about 152 LAs a year. Fewer than 145 in the
+-- latest year (or none at all) means the LA filter in tap_uk_ees
+-- (ks4_summary.headline_rows) stopped matching, e.g. after DfE renamed a label.
+
+with latest as (
+    select max(year) as year from {{ ref('fact_ks4_la_averages') }}
+)
+
+select l.year, count(f.la_code) as las
+from latest l
+left join {{ ref('fact_ks4_la_averages') }} f on f.year = l.year
+group by l.year
+having count(f.la_code) < 145
@@ -0,0 +1,20 @@
+{{ config(severity='warn') }}
+
+-- DfE's LA averages cover the latest year with school results. They come from
+-- a fixed EES data set (ks4_summary.KS4_SUMMARY_CSV_URL); when DfE publishes a
+-- new year under a new data set, the school results move on and this mart does
+-- not, so /api/la-averages serves an empty map and every "vs LA avg" goes.
+-- A warning, not an error: it must not hold back a new year's school results.
+
+with schools as (
+    select max(year) as year from {{ ref('stg_ees_ks4') }} where attainment_8_score is not null
+),
+
+las as (
+    select max(year) as year from {{ ref('fact_ks4_la_averages') }}
+)
+
+select s.year as latest_school_year, l.year as latest_la_year
+from schools s
+cross join las l
+where l.year is null or l.year < s.year
@@ -0,0 +1,12 @@
+-- Every year loaded from EES has an Attainment 8 for at least half its
+-- schools. DfE's files reach 82% each year; 2023/24 loaded at 0% after DfE
+-- re-issued its file under older column names (audit C2). Pre-2019 years come
+-- from stg_legacy_ks4 and are not checked here.
+
+select
+    year,
+    count(*) as schools,
+    count(attainment_8_score) as with_attainment_8
+from {{ ref('stg_ees_ks4') }}
+group by year
+having count(attainment_8_score) < 0.5 * count(*)
@@ -0,0 +1,7 @@
+-- A school-framework record always wins over a further education one for the
+-- same URN (audit H4): no FE row may stand where Ofsted's school MI has one.
+
+select f.urn
+from {{ ref('fact_ofsted_latest') }} f
+join {{ ref('stg_ofsted_inspections') }} s on s.urn = f.urn
+where f.framework = 'Further education and skills'
Author	SHA1	Message	Date
TudorandClaude Opus 5.5	4f342c50fd	docs: H4 rollout order and where stg_gias_ukprn is built (review) PR Checks / Frontend Typecheck + Tests (pull_request) Waiting to run Details PR Checks / Backend Smoke (pull_request) Waiting to run Details PR Checks / Build Backend (no push) (pull_request) Waiting to run Details PR Checks / Build Frontend (no push) (pull_request) Waiting to run Details PR Checks / Build Pipeline (no push) (pull_request) Waiting to run Details PR Checks / AI Code Review (Claude) (pull_request) Waiting to run Details Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:55:49 +01:00
TudorandClaude Opus 5.5	32715af568	fix(ofsted): a gov.uk outage skips the FE streams, not the Ofsted extract (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:55:22 +01:00
TudorandClaude Opus 5.5	c16dd0044f	fix(dags): build stg_gias_ukprn monthly, never daily (review) Rebuilding the view in the daily build dropped the Ofsted views on it (CASCADE), which dim_school reads: every daily run would have failed. A selector test now rejects any build that drops views it then reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:54:51 +01:00
TudorandClaude Opus 5.5	cfc9367bb3	fix(dbt): stg_gias_ukprn reads URN as the bigint target-postgres loads (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:28:51 +01:00
TudorandClaude Opus 5.5	8e82e2e4cf	feat(dbt): dim_school.inspectorate and ofsted_listed (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:26:42 +01:00
TudorandClaude Opus 5.5	6cc135ec58	feat(dbt): further education status where a school has no school record (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:26:05 +01:00
TudorandClaude Opus 5.5	222886465a	feat(dbt): further education status per URN under the C1 rule (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:25:26 +01:00
TudorandClaude Opus 5.5	6abee62c2a	feat(dbt): stage Ofsted's further education and skills MI (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:25:04 +01:00
TudorandClaude Opus 5.5	ab3e12e1ac	feat(ofsted): extract Ofsted's further education and skills MI (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:24:37 +01:00
TudorandClaude Opus 5.5	1949e85c58	feat(gias): load UKPRN and the inspectorate GIAS names (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:23:31 +01:00
TudorandClaude Opus 5.5	000626c5d6	docs: plan for inspection status beyond Ofsted's school data (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 21:00:54 +01:00
TudorandClaude Opus 5.5	5f7528b399	docs: design for inspection status beyond Ofsted's school data (H4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 23:00:38 +01:00
tudor	2b5e681482	Merge pull request 'fix(site): DfE LA averages for "vs LA avg"; no gap for independents (H2, part 2 of 2)' (#186 ) from fix/la-averages-site into main Stage (build -> staging -> E2E gate) / prepare (push) Successful in 1s Details Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 21s Details Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m38s Details Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 13s Details Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 26s Details Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Successful in 3m30s Details Reviewed-on: #186	2026-10-06 15:07:36 +00:00
tudor	3bd1dbd276	Merge pull request 'fix(pipeline): load 2023/24 KS4/KS2 data and DfE LA averages (C2, H2, part 1 of 2)' (#185 ) from fix/ks4-2023-24-pipeline into main Stage (build -> staging -> E2E gate) / prepare (push) Successful in 0s Details Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 45s Details Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m33s Details Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 2m6s Details Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 6s Details Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Successful in 3m27s Details Reviewed-on: #185	2026-10-06 12:42:19 +00:00
TudorandClaude Opus 5.5	5f93a7ecd2	docs: name the benchmarks computed from our dataset (review) PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m19s Details PR Checks / Backend Smoke (pull_request) Successful in 11s Details PR Checks / Build Backend (no push) (pull_request) Successful in 35s Details PR Checks / Build Frontend (no push) (pull_request) Successful in 1m36s Details PR Checks / Build Pipeline (no push) (pull_request) Successful in 1m17s Details PR Checks / AI Code Review (Claude) (pull_request) Successful in 35s Details Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 12:50:40 +01:00
TudorandClaude Opus 5.5	951c666f8a	test(dbt): warn when DfE's LA averages fall behind the school results (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 12:50:27 +01:00
TudorandClaude Opus 5.5	e09d7a202b	fix(ees): read the year from suffixed release slugs (review) A '2025-26-revised' release read as an unknown year went last, behind the '2025-26' release, which then owned the year and dropped every revised row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 12:50:10 +01:00
TudorandClaude Opus 5.5	4ae3853277	fix(dbt): read percentages DfE writes as "34%" (C2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 12:33:36 +01:00
TudorandClaude Opus 5.5	242c603aec	test(dbt): fail the build when a published year loads empty (C2, H2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 12:33:03 +01:00
TudorandClaude Opus 5.5	59dd20ab00	feat(dbt): fact_ks4_la_averages from DfE's LA rows (H2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 12:32:22 +01:00
TudorandClaude Opus 5.5	dbb60acff3	feat(ees): keep DfE's KS4 LA averages from the summary data set (H2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 10:46:53 +01:00
TudorandClaude Opus 5.5	967b1f0eed	fix(ees): read 2023/24 KS4 school information under DfE's older names (C2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 10:45:57 +01:00
TudorandClaude Opus 5.5	4db1131d0f	fix(ees): newest KS4 release owns every year it contains (C2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 10:45:14 +01:00
TudorandClaude Opus 5.5	0e987ef06e	docs: plan for 2023/24 KS4/KS2 data and DfE LA averages (C2, H2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 10:34:00 +01:00
TudorandClaude Opus 5.5	ac5b7ccd7f	docs: design for 2023/24 KS4/KS2 data and DfE LA averages (C2, H2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 10:08:25 +01:00