diff --git a/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md b/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md new file mode 100644 index 0000000..fdbe319 --- /dev/null +++ b/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md @@ -0,0 +1,229 @@ +# Inspection Status Beyond Ofsted's School Data — Design + +**Date:** 2026-10-06 +**Status:** approved design, not yet implemented +**Scope:** `tap_uk_gias` (two fields), `tap_uk_ofsted` (two FE streams), Ofsted +and GIAS staging, `int_ofsted_latest` / `fact_ofsted_latest`, new +`int_ofsted_fe_latest` and `int_ofsted_listed`, `dim_school`, monthly Ofsted DAG +selector, backend list and detail payloads, search badge, map card, school page +Ofsted section, compare Ofsted rows +**Fixes:** audit finding H4, from the 3 Oct 2026 accuracy audit + +## Goal + +Never label a school "Not yet inspected" unless Ofsted's own data lists it +without an inspection, and show every inspection result we can source, with its +date. + +## The problem + +The search badge and the map card (`buildOfstedListBadge`, +`nextjs-app/lib/utils.ts`) print "Not yet inspected" whenever we hold no Ofsted +record. The school page hides its Ofsted section for these schools, and compare +says "No inspection in our dataset". On production on 6 October 2026 (build +`2b5e681`), 2,587 listed schools have no Ofsted record. Joined with GIAS +(6 October 2026) and Ofsted's management information (31 August 2026): + +| Group | Schools | Evidence | +|---|---|---| +| Independent schools inspected by ISI | 1,341 | GIAS `InspectorateName` = "ISI". GIAS holds no date, outcome or report link for any of them; ISI publishes no bulk data. | +| Further-education and post-16 providers inspected by Ofsted | 519 | In Ofsted's "Further education and skills" MI: 509 by Provider URN, 10 only by UKPRN through GIAS. 207 FE colleges, 129 special post-16 institutions, 97 higher-education institutions (their FE and apprenticeship provision), 78 16–19 academies and free schools, 8 others. Queen Mary's College (145175): Good, 6 December 2022. | +| Listed by Ofsted, never inspected | 263+ | New academies and free schools present in Ofsted's school MI with every inspection field `NULL`, e.g. Abbas and Templecombe Church School (150031, opened January 2024). "Not yet inspected" is Ofsted's own position for these. | +| No usable inspection data | about 460 | Secure units (inspected as children's homes), higher-education institutions outside the FE MI, newer independents, schools opened after the MI date. | + +Of the 519 FE providers: 443 have only a grade from the previous framework +(overall effectiveness 1–4 and the inspection's first day); 45 also have a newer +report card under the renewed framework (from 10 November 2025); 2 have only a +report card; 29 are listed with no inspection. 116 FE providers nationally have +a short inspection after their latest full inspection; the MI gives its date +but not its outcome. + +## Non-goals + +- Area grades from the previous FE framework, and the report card's + programme-by-theme grid. The FE display is the headline and a link. +- ISI dates or outcomes (no data source). +- A per-school ISI link: ISI's pages use ISI's own identifiers, not URNs. +- Inspections of secure units and children's homes (social care framework). +- Changing how school-framework records are chosen (the C1 rule stands). + +## Data + +### Extractors + +**GIAS (`tap_uk_gias`).** Declare `UKPRN` and `InspectorateName (name)` on the +establishments stream. They are plain values, not codes, so the generated GIAS +dictionaries (backend and pipeline copies, parity-tested) are untouched. + +**Ofsted (`tap_uk_ofsted`).** Two streams from Ofsted's "Further education and +skills inspections and outcomes: management information from …" pages on +gov.uk. Pages are found through the gov.uk search API by that title prefix; a +new page appears each academic year. + +- `ofsted_fe_providers`: the newest page's newest + `…further_education_and_skills_-_as_at_.csv`. One row per provider: + `Provider URN`, `Provider UKPRN`, `Provider name`, `Provider type`, + `Has the provider received a full inspection under the renewed EIF?`, + `First day of inspection of latest inspection before renewed EIF`, + `Latest overall effectiveness before renewed EIF`, + `Date of latest short inspection before renewed EIF (if relevant)`. + Key: `provider_urn`. +- `ofsted_fe_inspections`: renewed-framework inspections. Each page carries + cumulative files `…further_education_and_skills_-__to_.csv` + (`10_November_2025_to_31_January_2026`, `…_to_28_February_2026`, …, sometimes + with a doubled underscore); the stream reads the file with the latest end date + from every page since November 2025, so a report card survives the next + year's page. Files named `new_provider_monitoring` or `monitoring_inspections` + are excluded. Fields: `Provider URN`, `Provider UKPRN`, `Inspection number`, + `First day of inspection`, `Date published`, `Safeguarding standards`. + Key: `inspection_number`. + +Page and file selection, and the CSV-to-field mapping, live in a module that +imports only pandas and the standard library, so CI's pytest (no +`singer_sdk`) can test them against saved real page HTML and real headers. + +The existing school-MI stream is unchanged. It already loads every row Ofsted +lists, inspected or not; `stg_ofsted_inspections` drops the uninspected rows. + +### dbt + +- `stg_gias_establishments` gains `ukprn` and `inspectorate_name`. +- `stg_ofsted_fe_providers` and `stg_ofsted_fe_inspections`: dates parsed as + `DD/MM/YYYY`; grades `1`–`4` cast to integers and `-` to null; safeguarding + `Met` / `Not met` to a boolean. +- `int_ofsted_fe_latest`, one row per URN we hold. A provider matches a school + by `Provider URN` = URN; failing that, by `Provider UKPRN` = the school's GIAS + `ukprn`. The rule, in the terms of the C1 current-status rule: + + | FE record | `current_grade` | `latest_visit` | + |---|---|---| + | Report card newer than the previous-framework inspection (or no such inspection) | null; `rc_inspection_date`, `rc_safeguarding_met` set | the report card (`report_card`) | + | Previous-framework grade 1–4, no newer report card | that grade, dated by the inspection's first day, basis `graded` | the later of that inspection (`graded`) and any short inspection (`short`) | + | Grade `-` and no report card | — | no row | + + A short inspection's outcome is not in the data, so it is a visit, never a + grade change. +- `int_ofsted_latest` / `fact_ofsted_latest` take FE rows only for URNs with no + school-framework record; a school record always wins. FE rows carry + `framework = 'Further education and skills'`; school area and report-card + area columns stay null. `latest_visit_kind` gains the value `short`. +- `int_ofsted_listed`: every URN Ofsted's data includes, inspected or not — + all rows of `raw.ofsted_inspections` (before staging's date filter) plus every + matched FE provider. +- `dim_school` gains `inspectorate` (GIAS `inspectorate_name`: "ISI", "Ofsted", + "Not applicable" or null) and `ofsted_listed` (boolean), the latter joined by + the same optional-relation pattern used for `int_ofsted_latest`. + `pipeline/tests/test_dag_selectors.py`'s `OPTIONAL_PARENTS` adds + `int_ofsted_listed`. +- The monthly Ofsted DAG build selects + `stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ int_ofsted_listed+` as + well. +- Tests: the existing `fact_ofsted_latest` tests (unique URN, + `assert_ofsted_current_grade_consistent`) cover FE rows. A new test asserts no + URN draws on both a school and an FE record. + +## What each page shows + +**Search badge and map card.** `buildOfstedListBadge` may return no badge. In +order: + +1. report card → "Report Card · year"; +2. grade in force → grade word · year; +3. inspected, no grade in force → "Inspected · year"; +4. GIAS inspectorate "ISI" → "Inspected by ISI" (grey); +5. listed by Ofsted → "Not yet inspected" (grey); +6. otherwise no badge. + +FE schools reach 1–3 with the same labels: Queen Mary's College "Good · 2022"; +The Autism Project – CareTrade (143526) "Report Card · 2026". + +**School page.** + +- FE: the existing Ofsted section with "Further education and skills" under the + heading. A grade reads as today ("Good", "Graded inspection, 6 December + 2022"); a report card shows its date and the safeguarding result; a later + short inspection gets "Latest visit: Short inspection, [date]". No area or + programme grid. The existing "Ofsted reports ↗" link resolves for FE + providers (Leyton Sixth Form College 130457 → `/provider/40/130457`). +- ISI: an inspection section reading "Inspected by the Independent Schools + Inspectorate (ISI)" with "ISI reports ↗" linking to + `https://www.isi.net/reports/`. +- Listed, never inspected: an Ofsted section reading "Not yet inspected", with + the Ofsted reports link (for a new academy, Ofsted's page links the + predecessor school's reports). +- No data: no section, as today. + +**Compare.** FE and listed schools follow the school page's states; ISI schools +read "Inspected by ISI"; schools with no data keep "No inspection in our +dataset". + +**Backend.** The list payload and the detail `school_info` gain `inspectorate` +and `ofsted_listed`. FE detail needs no new block: `_ofsted_block` already +carries `current_grade`, `latest_visit`, the report-card date and `framework`. +`inspectorate` and `ofsted_listed` join `_EXTRA_OPTIONAL_COLUMNS` in +`backend/data_loader.py`, so a `dim_school` built before PR 1 makes the main +query retry without them (no badge for groups 4–5) instead of failing. + +Copy follows the site's factual rule: inspectorate names and Ofsted's own terms +("Short inspection", "Not yet inspected"), no derived sentences. + +## Delivery + +Two PRs, as for C1 and C2. + +**PR 1 — pipeline:** the GIAS fields, the two FE streams and their selection +module, the staging and intermediate models, `dim_school`'s two columns, the +DAG selector, `OPTIONAL_PARENTS`, unit and data tests. + +**PR 2 — site (after the DAGs have run on PR 1):** the backend fields and the +optional-column fallback, the badge rule, the school-page states, compare, and +E2E. + +## Testing + +- **Extractor (pytest, pandas-only module):** from saved page HTML, the newest + "as at" file and the latest cumulative file per page are chosen and + monitoring files are ignored; the doubled-underscore file name parses; real + headers map to the declared fields. +- **dbt unit tests:** previous-framework grade only; grade with a later short + inspection (grade dated by the full inspection, latest visit `short`); grade + with a newer report card (report card, no grade); report card only; + UKPRN-only match; a school record wins over an FE record; a listed, + uninspected school is in `int_ofsted_listed` and not in `int_ofsted_latest`. +- **Backend pytest:** both new fields in the list and detail payloads; the main + query falls back when either column is missing. +- **Jest:** one case per badge rule including "no badge"; every badge caller + renders nothing when there is no badge; the school page's FE, ISI and + listed-uninspected states; compare cells. +- **E2E:** Queen Mary's College (145175) shows an Ofsted grade, not "Not yet + inspected"; Abbey College Cambridge (136083) shows "Inspected by ISI"; Abbas + and Templecombe Church School (150031) shows "Not yet inspected". + +## Rollout and verification + +1. Locally, load real GIAS, school MI and FE MI through the changed extractors, + build the models, and check: about 490 FE rows (443 + 45 + 2); 145175 graded + 2 on 2022-12-06; 143526 report card 2026-05-12, safeguarding met; 1,341 + schools with inspectorate "ISI"; 150031 listed, not in `fact_ofsted_latest`. +2. PR 1 merges; Tudor runs the daily DAG (GIAS fields) and the monthly Ofsted + DAG on staging; the staging API shows the same examples. +3. PR 2 merges, so its E2E journeys run against loaded data. +4. Production, Tudor's decision: promote PR 1, run both DAGs, promote PR 2. If + PR 2 arrives first, the fallback query runs and groups 4–5 show no badge + until the DAGs have run. +5. Validate on production: every "Not yet inspected" belongs to a school + Ofsted's own data lists without an inspection; every "Inspected by ISI" + matches GIAS; every FE grade, date and report card matches Ofsted's FE files; + schools with no data show no badge. Then mark H4 resolved in the audit + report. + +## Risks + +- **FE page or file naming changes.** Selection is by title prefix and file-name + pattern; if nothing matches, the FE streams emit nothing and FE schools fall + back to "no badge", never a false claim. The extractor tests pin the patterns + against real HTML. +- **A provider's UKPRN shared by several URNs.** Each matched URN takes the + provider's result; the URN match is always tried first. +- **"Not yet inspected" disappears briefly** if PR 2 is promoted before the + DAGs run (fallback query). It never appears wrongly.