school_compare/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md

Inspection Status Beyond Ofsted's School Data — Design

Date: 2026-10-06 Status: approved design, not yet implemented Scope: tap_uk_gias (two fields), tap_uk_ofsted (two FE streams), Ofsted and GIAS staging, int_ofsted_latest / fact_ofsted_latest, new int_ofsted_fe_latest and int_ofsted_listed, dim_school, monthly Ofsted DAG selector, backend list and detail payloads, search badge, map card, school page Ofsted section, compare Ofsted rows Fixes: audit finding H4, from the 3 Oct 2026 accuracy audit

Goal

Never label a school "Not yet inspected" unless Ofsted's own data lists it without an inspection, and show every inspection result we can source, with its date.

The problem

The search badge and the map card (buildOfstedListBadge, nextjs-app/lib/utils.ts) print "Not yet inspected" whenever we hold no Ofsted record. The school page hides its Ofsted section for these schools, and compare says "No inspection in our dataset". On production on 6 October 2026 (build 2b5e681), 2,587 listed schools have no Ofsted record. Joined with GIAS (6 October 2026) and Ofsted's management information (31 August 2026):

Group	Schools	Evidence
Independent schools inspected by ISI	1,341	GIAS InspectorateName = "ISI". GIAS holds no date, outcome or report link for any of them; ISI publishes no bulk data.
Further-education and post-16 providers inspected by Ofsted	519	In Ofsted's "Further education and skills" MI: 509 by Provider URN, 10 only by UKPRN through GIAS. 207 FE colleges, 129 special post-16 institutions, 97 higher-education institutions (their FE and apprenticeship provision), 78 16–19 academies and free schools, 8 others. Queen Mary's College (145175): Good, 6 December 2022.
Listed by Ofsted, never inspected	263+	New academies and free schools present in Ofsted's school MI with every inspection field NULL, e.g. Abbas and Templecombe Church School (150031, opened January 2024). "Not yet inspected" is Ofsted's own position for these.
No usable inspection data	about 460	Secure units (inspected as children's homes), higher-education institutions outside the FE MI, newer independents, schools opened after the MI date.

Of the 519 FE providers: 443 have only a grade from the previous framework (overall effectiveness 1–4 and the inspection's first day); 45 also have a newer report card under the renewed framework (from 10 November 2025); 2 have only a report card; 29 are listed with no inspection. 116 FE providers nationally have a short inspection after their latest full inspection; the MI gives its date but not its outcome.

Non-goals

• Area grades from the previous FE framework, and the report card's programme-by-theme grid. The FE display is the headline and a link.
• ISI dates or outcomes (no data source).
• A per-school ISI link: ISI's pages use ISI's own identifiers, not URNs.
• Inspections of secure units and children's homes (social care framework).
• Changing how school-framework records are chosen (the C1 rule stands).

Data

Extractors

GIAS (tap_uk_gias). Declare UKPRN and InspectorateName (name) on the establishments stream. They are plain values, not codes, so the generated GIAS dictionaries (backend and pipeline copies, parity-tested) are untouched.

Ofsted (tap_uk_ofsted). Two streams from Ofsted's "Further education and skills inspections and outcomes: management information from …" pages on gov.uk. Pages are found through the gov.uk search API by that title prefix; a new page appears each academic year.

• ofsted_fe_providers: the newest page's newest …further_education_and_skills_-_as_at_<date>.csv. One row per provider: Provider URN, Provider UKPRN, Provider name, Provider type, Has the provider received a full inspection under the renewed EIF?, First day of inspection of latest inspection before renewed EIF, Latest overall effectiveness before renewed EIF, Date of latest short inspection before renewed EIF (if relevant). Key: provider_urn.
• ofsted_fe_inspections: renewed-framework inspections. Each page carries cumulative files …further_education_and_skills_-_<start>_to_<end>.csv (10_November_2025_to_31_January_2026, …_to_28_February_2026, …, sometimes with a doubled underscore); the stream reads the file with the latest end date from every page since November 2025, so a report card survives the next year's page. Files named new_provider_monitoring or monitoring_inspections are excluded. Fields: Provider URN, Provider UKPRN, Inspection number, First day of inspection, Date published, Safeguarding standards. Key: inspection_number.

Page and file selection, and the CSV-to-field mapping, live in a module that imports only pandas and the standard library, so CI's pytest (no singer_sdk) can test them against saved real page HTML and real headers.

The existing school-MI stream is unchanged. It already loads every row Ofsted lists, inspected or not; stg_ofsted_inspections drops the uninspected rows.

dbt

• stg_gias_establishments gains ukprn and inspectorate_name.

• stg_ofsted_fe_providers and stg_ofsted_fe_inspections: dates parsed as DD/MM/YYYY; grades 1–4 cast to integers and - to null; safeguarding Met / Not met to a boolean.

• int_ofsted_fe_latest, one row per URN we hold. A provider matches a school by Provider URN = URN; failing that, by Provider UKPRN = the school's GIAS ukprn. The rule, in the terms of the C1 current-status rule:

  FE record	current_grade	latest_visit
  Report card newer than the previous-framework inspection (or no such inspection)	null; rc_inspection_date, rc_safeguarding_met set	the report card (report_card)
  Previous-framework grade 1–4, no newer report card	that grade, dated by the inspection's first day, basis graded	the later of that inspection (graded) and any short inspection (short)
  Grade - and no report card	—	no row

  A short inspection's outcome is not in the data, so it is a visit, never a grade change.

• int_ofsted_latest / fact_ofsted_latest take FE rows only for URNs with no school-framework record; a school record always wins. FE rows carry framework = 'Further education and skills'; school area and report-card area columns stay null. latest_visit_kind gains the value short.

• int_ofsted_listed: every URN Ofsted's data includes, inspected or not — all rows of raw.ofsted_inspections (before staging's date filter) plus every matched FE provider.

• dim_school gains inspectorate (GIAS inspectorate_name: "ISI", "Ofsted", "Not applicable" or null) and ofsted_listed (boolean), the latter joined by the same optional-relation pattern used for int_ofsted_latest. pipeline/tests/test_dag_selectors.py's OPTIONAL_PARENTS adds int_ofsted_listed.

• The monthly Ofsted DAG build selects stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ int_ofsted_listed+ as well.

• Tests: the existing fact_ofsted_latest tests (unique URN, assert_ofsted_current_grade_consistent) cover FE rows. A new test asserts no URN draws on both a school and an FE record.

What each page shows

Search badge and map card. buildOfstedListBadge may return no badge. In order:

1. report card → "Report Card · year";
2. grade in force → grade word · year;
3. inspected, no grade in force → "Inspected · year";
4. GIAS inspectorate "ISI" → "Inspected by ISI" (grey);
5. listed by Ofsted → "Not yet inspected" (grey);
6. otherwise no badge.

FE schools reach 1–3 with the same labels: Queen Mary's College "Good · 2022"; The Autism Project – CareTrade (143526) "Report Card · 2026".

School page.

• FE: the existing Ofsted section with "Further education and skills" under the heading. A grade reads as today ("Good", "Graded inspection, 6 December 2022"); a report card shows its date and the safeguarding result; a later short inspection gets "Latest visit: Short inspection, [date]". No area or programme grid. The existing "Ofsted reports ↗" link resolves for FE providers (Leyton Sixth Form College 130457 → /provider/40/130457).
• ISI: an inspection section reading "Inspected by the Independent Schools Inspectorate (ISI)" with "ISI reports ↗" linking to https://www.isi.net/reports/.
• Listed, never inspected: an Ofsted section reading "Not yet inspected", with the Ofsted reports link (for a new academy, Ofsted's page links the predecessor school's reports).
• No data: no section, as today.

Compare. FE and listed schools follow the school page's states; ISI schools read "Inspected by ISI"; schools with no data keep "No inspection in our dataset".

Backend. The list payload and the detail school_info gain inspectorate and ofsted_listed. FE detail needs no new block: _ofsted_block already carries current_grade, latest_visit, the report-card date and framework. inspectorate and ofsted_listed join _EXTRA_OPTIONAL_COLUMNS in backend/data_loader.py, so a dim_school built before PR 1 makes the main query retry without them (no badge for groups 4–5) instead of failing.

Copy follows the site's factual rule: inspectorate names and Ofsted's own terms ("Short inspection", "Not yet inspected"), no derived sentences.

Delivery

Two PRs, as for C1 and C2.

PR 1 — pipeline: the GIAS fields, the two FE streams and their selection module, the staging and intermediate models, dim_school's two columns, the DAG selector, OPTIONAL_PARENTS, unit and data tests.

PR 2 — site (after the DAGs have run on PR 1): the backend fields and the optional-column fallback, the badge rule, the school-page states, compare, and E2E.

Testing

• Extractor (pytest, pandas-only module): from saved page HTML, the newest "as at" file and the latest cumulative file per page are chosen and monitoring files are ignored; the doubled-underscore file name parses; real headers map to the declared fields.
• dbt unit tests: previous-framework grade only; grade with a later short inspection (grade dated by the full inspection, latest visit short); grade with a newer report card (report card, no grade); report card only; UKPRN-only match; a school record wins over an FE record; a listed, uninspected school is in int_ofsted_listed and not in int_ofsted_latest.
• Backend pytest: both new fields in the list and detail payloads; the main query falls back when either column is missing.
• Jest: one case per badge rule including "no badge"; every badge caller renders nothing when there is no badge; the school page's FE, ISI and listed-uninspected states; compare cells.
• E2E: Queen Mary's College (145175) shows an Ofsted grade, not "Not yet inspected"; Abbey College Cambridge (136083) shows "Inspected by ISI"; Abbas and Templecombe Church School (150031) shows "Not yet inspected".

Rollout and verification

1. Locally, load real GIAS, school MI and FE MI through the changed extractors, build the models, and check: about 490 FE rows (443 + 45 + 2); 145175 graded 2 on 2022-12-06; 143526 report card 2026-05-12, safeguarding met; 1,341 schools with inspectorate "ISI"; 150031 listed, not in fact_ofsted_latest.
2. PR 1 merges; Tudor runs the daily DAG (GIAS fields) and the monthly Ofsted DAG on staging; the staging API shows the same examples.
3. PR 2 merges, so its E2E journeys run against loaded data.
4. Production, Tudor's decision: promote PR 1, run both DAGs, promote PR 2. If PR 2 arrives first, the fallback query runs and groups 4–5 show no badge until the DAGs have run.
5. Validate on production: every "Not yet inspected" belongs to a school Ofsted's own data lists without an inspection; every "Inspected by ISI" matches GIAS; every FE grade, date and report card matches Ofsted's FE files; schools with no data show no badge. Then mark H4 resolved in the audit report.

Risks

• FE page or file naming changes. Selection is by title prefix and file-name pattern; if nothing matches, the FE streams emit nothing and FE schools fall back to "no badge", never a false claim. The extractor tests pin the patterns against real HTML.
• A provider's UKPRN shared by several URNs. Each matched URN takes the provider's result; the URN match is always tried first.
• "Not yet inspected" disappears briefly if PR 2 is promoted before the DAGs run (fallback query). It never appears wrongly.