school_compare/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md

# Inspection Status Beyond Ofsted's School Data — Design

**Date:** 2026-10-06
**Status:** approved design, not yet implemented
**Scope:** `tap_uk_gias` (two fields), `tap_uk_ofsted` (two FE streams), Ofsted
and GIAS staging, `int_ofsted_latest` / `fact_ofsted_latest`, new
`int_ofsted_fe_latest` and `int_ofsted_listed`, `dim_school`, monthly Ofsted DAG
selector, backend list and detail payloads, search badge, map card, school page
Ofsted section, compare Ofsted rows
**Fixes:** audit finding H4, from the 3 Oct 2026 accuracy audit

## Goal

Never label a school "Not yet inspected" unless Ofsted's own data lists it
without an inspection, and show every inspection result we can source, with its
date.

## The problem

The search badge and the map card (`buildOfstedListBadge`,
`nextjs-app/lib/utils.ts`) print "Not yet inspected" whenever we hold no Ofsted
record. The school page hides its Ofsted section for these schools, and compare
says "No inspection in our dataset". On production on 6 October 2026 (build
`2b5e681`), 2,587 listed schools have no Ofsted record. Joined with GIAS
(6 October 2026) and Ofsted's management information (31 August 2026):

| Group | Schools | Evidence |
|---|---|---|
| Independent schools inspected by ISI | 1,341 | GIAS `InspectorateName` = "ISI". GIAS holds no date, outcome or report link for any of them; ISI publishes no bulk data. |
| Further-education and post-16 providers inspected by Ofsted | 519 | In Ofsted's "Further education and skills" MI: 509 by Provider URN, 10 only by UKPRN through GIAS. 207 FE colleges, 129 special post-16 institutions, 97 higher-education institutions (their FE and apprenticeship provision), 78 16–19 academies and free schools, 8 others. Queen Mary's College (145175): Good, 6 December 2022. |
| Listed by Ofsted, never inspected | 263+ | New academies and free schools present in Ofsted's school MI with every inspection field `NULL`, e.g. Abbas and Templecombe Church School (150031, opened January 2024). "Not yet inspected" is Ofsted's own position for these. |
| No usable inspection data | about 460 | Secure units (inspected as children's homes), higher-education institutions outside the FE MI, newer independents, schools opened after the MI date. |

Of the 519 FE providers: 443 have only a grade from the previous framework
(overall effectiveness 1–4 and the inspection's first day); 45 also have a newer
report card under the renewed framework (from 10 November 2025); 2 have only a
report card; 29 are listed with no inspection. 116 FE providers nationally have
a short inspection after their latest full inspection; the MI gives its date
but not its outcome.

## Non-goals

- Area grades from the previous FE framework, and the report card's
  programme-by-theme grid. The FE display is the headline and a link.
- ISI dates or outcomes (no data source).
- A per-school ISI link: ISI's pages use ISI's own identifiers, not URNs.
- Inspections of secure units and children's homes (social care framework).
- Changing how school-framework records are chosen (the C1 rule stands).

## Data

### Extractors

**GIAS (`tap_uk_gias`).** Declare `UKPRN` and `InspectorateName (name)` on the
establishments stream. They are plain values, not codes, so the generated GIAS
dictionaries (backend and pipeline copies, parity-tested) are untouched.

**Ofsted (`tap_uk_ofsted`).** Two streams from Ofsted's "Further education and
skills inspections and outcomes: management information from …" pages on
gov.uk. Pages are found through the gov.uk search API by that title prefix; a
new page appears each academic year.

- `ofsted_fe_providers`: the newest page's newest
  `…further_education_and_skills_-_as_at_<date>.csv`. One row per provider:
  `Provider URN`, `Provider UKPRN`, `Provider name`, `Provider type`,
  `Has the provider received a full inspection under the renewed EIF?`,
  `First day of inspection of latest inspection before renewed EIF`,
  `Latest overall effectiveness before renewed EIF`,
  `Date of latest short inspection before renewed EIF (if relevant)`.
  Key: `provider_urn`.
- `ofsted_fe_inspections`: renewed-framework inspections. Each page carries
  cumulative files `…further_education_and_skills_-_<start>_to_<end>.csv`
  (`10_November_2025_to_31_January_2026`, `…_to_28_February_2026`, …, sometimes
  with a doubled underscore); the stream reads the file with the latest end date
  from every page since November 2025, so a report card survives the next
  year's page. Files named `new_provider_monitoring` or `monitoring_inspections`
  are excluded. Fields: `Provider URN`, `Provider UKPRN`, `Inspection number`,
  `First day of inspection`, `Date published`, `Safeguarding standards`.
  Key: `inspection_number`.

Page and file selection, and the CSV-to-field mapping, live in a module that
imports only pandas and the standard library, so CI's pytest (no
`singer_sdk`) can test them against saved real page HTML and real headers.

The existing school-MI stream is unchanged. It already loads every row Ofsted
lists, inspected or not; `stg_ofsted_inspections` drops the uninspected rows.

### dbt

- `stg_gias_establishments` gains `ukprn` and `inspectorate_name`.
- `stg_ofsted_fe_providers` and `stg_ofsted_fe_inspections`: dates parsed as
  `DD/MM/YYYY`; grades `1`–`4` cast to integers and `-` to null; safeguarding
  `Met` / `Not met` to a boolean.
- `int_ofsted_fe_latest`, one row per URN we hold. A provider matches a school
  by `Provider URN` = URN; failing that, by `Provider UKPRN` = the school's GIAS
  `ukprn`. The rule, in the terms of the C1 current-status rule:

  | FE record | `current_grade` | `latest_visit` |
  |---|---|---|
  | Report card newer than the previous-framework inspection (or no such inspection) | null; `rc_inspection_date`, `rc_safeguarding_met` set | the report card (`report_card`) |
  | Previous-framework grade 1–4, no newer report card | that grade, dated by the inspection's first day, basis `graded` | the later of that inspection (`graded`) and any short inspection (`short`) |
  | Grade `-` and no report card | — | no row |

  A short inspection's outcome is not in the data, so it is a visit, never a
  grade change.
- `int_ofsted_latest` / `fact_ofsted_latest` take FE rows only for URNs with no
  school-framework record; a school record always wins. FE rows carry
  `framework = 'Further education and skills'`; school area and report-card
  area columns stay null. `latest_visit_kind` gains the value `short`.
- `int_ofsted_listed`: every URN Ofsted's data includes, inspected or not —
  all rows of `raw.ofsted_inspections` (before staging's date filter) plus every
  matched FE provider.
- `dim_school` gains `inspectorate` (GIAS `inspectorate_name`: "ISI", "Ofsted",
  "Not applicable" or null) and `ofsted_listed` (boolean), the latter joined by
  the same optional-relation pattern used for `int_ofsted_latest`.
  `pipeline/tests/test_dag_selectors.py`'s `OPTIONAL_PARENTS` adds
  `int_ofsted_listed`.
- The monthly Ofsted DAG build selects
  `stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ int_ofsted_listed+` as
  well.
- Tests: the existing `fact_ofsted_latest` tests (unique URN,
  `assert_ofsted_current_grade_consistent`) cover FE rows. A new test asserts no
  URN draws on both a school and an FE record.

## What each page shows

**Search badge and map card.** `buildOfstedListBadge` may return no badge. In
order:

1. report card → "Report Card · year";
2. grade in force → grade word · year;
3. inspected, no grade in force → "Inspected · year";
4. GIAS inspectorate "ISI" → "Inspected by ISI" (grey);
5. listed by Ofsted → "Not yet inspected" (grey);
6. otherwise no badge.

FE schools reach 1–3 with the same labels: Queen Mary's College "Good · 2022";
The Autism Project – CareTrade (143526) "Report Card · 2026".

**School page.**

- FE: the existing Ofsted section with "Further education and skills" under the
  heading. A grade reads as today ("Good", "Graded inspection, 6 December
  2022"); a report card shows its date and the safeguarding result; a later
  short inspection gets "Latest visit: Short inspection, [date]". No area or
  programme grid. The existing "Ofsted reports ↗" link resolves for FE
  providers (Leyton Sixth Form College 130457 → `/provider/40/130457`).
- ISI: an inspection section reading "Inspected by the Independent Schools
  Inspectorate (ISI)" with "ISI reports ↗" linking to
  `https://www.isi.net/reports/`.
- Listed, never inspected: an Ofsted section reading "Not yet inspected", with
  the Ofsted reports link (for a new academy, Ofsted's page links the
  predecessor school's reports).
- No data: no section, as today.

**Compare.** FE and listed schools follow the school page's states; ISI schools
read "Inspected by ISI"; schools with no data keep "No inspection in our
dataset".

**Backend.** The list payload and the detail `school_info` gain `inspectorate`
and `ofsted_listed`. FE detail needs no new block: `_ofsted_block` already
carries `current_grade`, `latest_visit`, the report-card date and `framework`.
`inspectorate` and `ofsted_listed` join `_EXTRA_OPTIONAL_COLUMNS` in
`backend/data_loader.py`, so a `dim_school` built before PR 1 makes the main
query retry without them (no badge for groups 4–5) instead of failing.

Copy follows the site's factual rule: inspectorate names and Ofsted's own terms
("Short inspection", "Not yet inspected"), no derived sentences.

## Delivery

Two PRs, as for C1 and C2.

**PR 1 — pipeline:** the GIAS fields, the two FE streams and their selection
module, the staging and intermediate models, `dim_school`'s two columns, the
DAG selector, `OPTIONAL_PARENTS`, unit and data tests.

**PR 2 — site (after the DAGs have run on PR 1):** the backend fields and the
optional-column fallback, the badge rule, the school-page states, compare, and
E2E.

## Testing

- **Extractor (pytest, pandas-only module):** from saved page HTML, the newest
  "as at" file and the latest cumulative file per page are chosen and
  monitoring files are ignored; the doubled-underscore file name parses; real
  headers map to the declared fields.
- **dbt unit tests:** previous-framework grade only; grade with a later short
  inspection (grade dated by the full inspection, latest visit `short`); grade
  with a newer report card (report card, no grade); report card only;
  UKPRN-only match; a school record wins over an FE record; a listed,
  uninspected school is in `int_ofsted_listed` and not in `int_ofsted_latest`.
- **Backend pytest:** both new fields in the list and detail payloads; the main
  query falls back when either column is missing.
- **Jest:** one case per badge rule including "no badge"; every badge caller
  renders nothing when there is no badge; the school page's FE, ISI and
  listed-uninspected states; compare cells.
- **E2E:** Queen Mary's College (145175) shows an Ofsted grade, not "Not yet
  inspected"; Abbey College Cambridge (136083) shows "Inspected by ISI"; Abbas
  and Templecombe Church School (150031) shows "Not yet inspected".

## Rollout and verification

1. Locally, load real GIAS, school MI and FE MI through the changed extractors,
   build the models, and check: about 490 FE rows (443 + 45 + 2); 145175 graded
   2 on 2022-12-06; 143526 report card 2026-05-12, safeguarding met; 1,341
   schools with inspectorate "ISI"; 150031 listed, not in `fact_ofsted_latest`.
2. PR 1 merges; Tudor runs the daily DAG (GIAS fields) and the monthly Ofsted
   DAG on staging; the staging API shows the same examples.
3. PR 2 merges, so its E2E journeys run against loaded data.
4. Production, Tudor's decision: promote PR 1, run both DAGs, promote PR 2. If
   PR 2 arrives first, the fallback query runs and groups 4–5 show no badge
   until the DAGs have run.
5. Validate on production: every "Not yet inspected" belongs to a school
   Ofsted's own data lists without an inspection; every "Inspected by ISI"
   matches GIAS; every FE grade, date and report card matches Ofsted's FE files;
   schools with no data show no badge. Then mark H4 resolved in the audit
   report.

## Risks

- **FE page or file naming changes.** Selection is by title prefix and file-name
  pattern; if nothing matches, the FE streams emit nothing and FE schools fall
  back to "no badge", never a false claim. The extractor tests pin the patterns
  against real HTML.
- **A provider's UKPRN shared by several URNs.** Each matched URN takes the
  provider's result; the URN match is always tried first.
- **"Not yet inspected" disappears briefly** if PR 2 is promoted before the
  DAGs run (fallback query). It never appears wrongly.

## Amendments (implementation plan, 2026-10-13)

- **UKPRN comes from a new view, `stg_gias_ukprn`,** on the raw GIAS source,
  not from a `ukprn` column on `stg_gias_establishments`. A `ref` to
  `stg_gias_establishments` from the Ofsted models would put them in the daily
  build's `stg_gias_establishments+` selection, where their Ofsted parents are
  not built (`pipeline/tests/test_dag_selectors.py`). The daily build selects
  `stg_gias_ukprn` by name; `stg_gias_establishments` gains only
  `inspectorate_name`.
- **A report card is recognised by `rc_inspection_date` as well as by its area
  labels** (`hasReportCard`), on the school page and in compare. An FE report
  card carries a safeguarding result and no school areas, so the label test
  alone read it as "No overall grade".
- **The compare page's Ofsted link is fixed.** `ofsted_page_url` built
  `https://reports.ofsted.gov.uk/provider/21/{urn}`; category 21 is primary
  schools, so the link 404s for every secondary, special and FE school
  (checked: 137086, 101099, 130457, 145175). It now uses
  `…/inspection-reports/find-inspection-report/provider/ELS/{urn}`, which
  redirects to the right category and is what the school page already uses.