From 000626c5d6088c5988e2d43408075dbf2493c7a7 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 13 Oct 2026 21:00:54 +0100 Subject: [PATCH] docs: plan for inspection status beyond Ofsted's school data (H4) Co-Authored-By: Claude Opus 5.5 --- ...inspection-status-beyond-ofsted-schools.md | 1962 +++++++++++++++++ ...ion-status-beyond-ofsted-schools-design.md | 20 + 2 files changed, 1982 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-13-inspection-status-beyond-ofsted-schools.md diff --git a/docs/superpowers/plans/2026-10-13-inspection-status-beyond-ofsted-schools.md b/docs/superpowers/plans/2026-10-13-inspection-status-beyond-ofsted-schools.md new file mode 100644 index 0000000..a4b3079 --- /dev/null +++ b/docs/superpowers/plans/2026-10-13-inspection-status-beyond-ofsted-schools.md @@ -0,0 +1,1962 @@ +# Inspection Status Beyond Ofsted's School Data Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop labelling schools "Not yet inspected" unless Ofsted's own data lists them uninspected, and show ISI and Ofsted further-education results where the data exists (audit H4). + +**Architecture:** PR 1 adds two GIAS fields, two Ofsted further-education (FE) streams, an FE status model folded into `int_ofsted_latest` / `fact_ofsted_latest` (framework "Further education and skills"), an `int_ofsted_listed` model, and `dim_school.inspectorate` / `dim_school.ofsted_listed`. PR 2, after the daily and monthly Ofsted DAGs have run on PR 1, publishes the two fields, changes the badge rule, adds the ISI / "Not yet inspected" school-page states and the FE label, updates compare, and fixes the Ofsted page link. + +**Tech Stack:** Singer SDK taps (Python, pandas, requests), meltanolabs target-postgres, dbt-postgres ~1.10 (YAML unit tests, singular tests), FastAPI + pandas + SQLAlchemy, Next.js / React / TypeScript, Jest + Testing Library, Playwright. + +**Spec:** `docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md` (read its "Amendments" section: UKPRN comes from a new `stg_gias_ukprn` view; report cards are detected by `rc_inspection_date`; `ofsted_page_url` is fixed). + +## Global Constraints + +- Badge order (search rows and map card): report card → "Report Card · year"; grade in force → grade word · year; inspected, no grade → "Inspected · year"; GIAS inspectorate `ISI` → "Inspected by ISI" (grey, css class `ofstedPending`); `ofsted_listed` → "Not yet inspected" (grey, `ofstedPending`); otherwise no badge. +- FE framework value, exactly: `Further education and skills`. +- FE rule: a report card on or after the previous-framework inspection (or with none) wins and leaves no grade in force; otherwise a previous-framework grade 1–4 is the current grade, dated by that inspection's first day, basis `graded`; latest visit is the later of that inspection (`graded`) and any short inspection (`short`); grade `-` with no report card gives no FE row. +- A school-framework record always wins over an FE record for the same URN. +- FE providers match a URN by `Provider URN` first, then by `Provider UKPRN` = GIAS UKPRN. +- The Ofsted models must never `ref` `stg_gias_establishments` (the daily build's `stg_gias_establishments+` would select them); they read UKPRN from `stg_gias_ukprn`, a view on the raw GIAS source. +- Copy: Ofsted's and GIAS's own terms ("Short inspection", "Not yet inspected", "Inspected by the Independent Schools Inspectorate (ISI)", "Further education and skills"); no derived sentences. +- ISI link: `https://www.isi.net/reports/`. Ofsted page link: `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}` (redirects to the right category; `provider/21/{urn}` 404s for every non-primary school). +- PR 2 must not merge before the daily DAG (GIAS fields) and `school_data_monthly_ofsted` have run on staging with PR 1, nor be promoted before they have run on production. Promotion is Tudor's; never trigger it. +- Never push to `main`. Commits end with the Co-Authored-By trailer the executing session's harness specifies; PR bodies end with the Claude Code line. Do not start a local app server. + +## Review Focus + +1. A school whose GIAS inspectorate is ISI **and** that Ofsted also lists (dual registration) must show "Inspected by ISI" only when there is no Ofsted inspection record; an Ofsted record always wins. Pinned in Task 11 (`an Ofsted record wins over ISI`). +2. An FE provider whose report card predates its previous-framework inspection (bad data) must not produce a row with both a grade and a report card (that breaks `assert_ofsted_current_grade_consistent`). Pinned in Task 5 (`int_ofsted_fe_report_card_older_than_grade_is_ignored`). +3. A UKPRN shared by an FE provider and a URN that Ofsted also lists under its own URN must take the URN match. Pinned in Task 5 (`int_ofsted_fe_prefers_the_urn_match`). +4. The FE "as at" file missing from the newest academic-year page (a new page with no file yet) must fall back to the previous page, not emit nothing. Pinned in Task 3 (`test_providers_fall_back_to_the_previous_page`). +5. A school-page or compare render for a school with `ofsted_listed` null (marts built before PR 1) must behave as "not listed": no "Not yet inspected". Pinned in Task 11 (null case) and Task 12. + +## File Map + +**PR 1 — branch `fix/ofsted-inspectorates-pipeline`** (exists; spec `5f7528b`) + +- Modify `pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py` (two fields). +- Create `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py` (page/file selection, CSV reading, column maps — pandas-only). +- Modify `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (two streams). +- Create `pipeline/tests/test_gias_declared_fields.py`, `pipeline/tests/test_ofsted_fe_mi.py`. +- Modify `pipeline/transform/models/staging/stg_gias_establishments.sql`, `_stg_sources.yml`. +- Create `pipeline/transform/models/staging/stg_gias_ukprn.sql`, `stg_ofsted_fe_providers.sql`, `stg_ofsted_fe_inspections.sql`, `stg_ofsted_fe.yml`. +- Create `pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`, `int_ofsted_fe_latest.yml`, `int_ofsted_listed.sql`, `int_ofsted_listed.yml`. +- Modify `pipeline/transform/models/intermediate/int_ofsted_latest.sql`, `int_ofsted_latest.yml`. +- Modify `pipeline/transform/models/marts/dim_school.sql`, `dim_school.yml`, `_marts_schema.yml`. +- Create `pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql`. +- Modify `pipeline/dags/school_data_pipeline.py` (daily and monthly selectors), `pipeline/tests/test_dag_selectors.py` (`OPTIONAL_PARENTS`). + +**PR 2 — branch `fix/ofsted-inspectorates-site`** (from `origin/main` after PR 1 merges; touches no PR 1 file) + +- Modify `backend/schemas.py` (`SCHOOL_COLUMNS`), `backend/data_loader.py` (main query, optional columns), `backend/app.py` (detail and compare `school_info`), `backend/ofsted_codes.py` (`ofsted_page_url`). +- Create `backend/tests/test_inspection_status_payload.py`; modify `backend/tests/test_ofsted_codes.py`, `backend/tests/test_supplementary_enrichment.py`. +- Modify `nextjs-app/lib/types.ts`, `nextjs-app/lib/ofstedStatus.ts`, `nextjs-app/lib/utils.ts`, `nextjs-app/lib/compareLogic.ts`. +- Modify `nextjs-app/components/SchoolRow.tsx`, `SecondarySchoolRow.tsx`, `LeafletMapInner.tsx`, `school/OfstedSection.tsx`, `school/PrimarySchoolSections.tsx`, `school/SecondarySchoolSections.tsx`, `compare/CompareOfsted.tsx`, `compare/CompareAtAGlance.tsx`. +- Create `nextjs-app/components/school/InspectionNotice.tsx`. +- Modify tests: `nextjs-app/__tests__/lib/utils.test.ts`, `__tests__/lib/ofstedStatus.test.ts`, `__tests__/lib/compareLogic.test.ts`, `__tests__/components/OfstedSection.test.tsx`, `__tests__/components/CompareOfsted.test.tsx`, `__tests__/components/SchoolRow.test.tsx`; create `__tests__/components/InspectionNotice.test.tsx`. +- Modify `e2e/tests/journeys.spec.ts`. + +**Commands used throughout.** `$SCRATCH` is the session scratchpad directory. Python tests run in CI's environment: + +```bash +uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest -q +``` + +Local tap commands use `--with-editable` (a plain `--with ./path` reuses a cached build of the unchanged version and ignores edits). dbt runs through `$SCRATCH/dbt_local/dbt.sh` (Task 1). + +--- + +## PR 1 — pipeline + +### Task 1: Local dbt harness + +`$SCRATCH/dbt_local/pgdata` survives from earlier work; its helper scripts do not. + +**Files:** none in the repo. + +- [ ] **Step 1: Recreate the helpers** + +```bash +mkdir -p "$SCRATCH/dbt_local" && cd "$SCRATCH/dbt_local" +cat > start_pg.py <<'EOF' +import pgserver, sys +srv = pgserver.get_server(sys.argv[1], cleanup_mode=None) +print(srv.get_uri()) +if "school_compare" not in srv.psql("select datname from pg_database;"): + srv.psql("create database school_compare;") +EOF +uv run --no-project --with pgserver python start_pg.py "$SCRATCH/dbt_local/pgdata" +``` + +Expected: a URI ending `?host=`. Then, with that socket dir: + +```bash +cat > "$SCRATCH/dbt_local/dbt.sh" < +export PG_HOST= PG_PORT=5432 PG_USER=postgres PG_PASSWORD= PG_DATABASE=school_compare +D=$SCRATCH/dbt_local +cd /Users/tudor/projects/school_compare/pipeline/transform +uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main "\$@" --profiles-dir . --target dev --target-path \$D/target --log-path \$D/logs > \$D/last.log 2>&1 +status=\$? +sed 's/\x1b\[[0-9;]*m//g' \$D/last.log | grep -E "PASS|FAIL|ERROR|WARN |OK created|Done\.|Error|Compilation|Invalid" +exit \$status +EOF +cat > "$SCRATCH/dbt_local/sql.sh" <<'EOF' +#!/bin/bash +# Run SQL from stdin against the local school_compare database; print the output. +D=$(cd "$(dirname "$0")" && pwd) +SQL=$(cat) D="$D" uv run --no-project --with pgserver python -c ' +import os, pgserver +srv = pgserver.get_server(os.environ["D"] + "/pgdata", cleanup_mode=None) +print(srv.psql("\\c school_compare\n" + os.environ["SQL"]))' +EOF +cat > "$SCRATCH/dbt_local/raw_tables.py" <<'EOF' +"""Create raw. with a text column per declared field (what target-postgres loads). +Usage: raw_tables.py : [ ...]""" +import importlib, os, sys, pgserver +module, cls = sys.argv[1].split(":") +tap = getattr(importlib.import_module(module), cls)(config={}) +wanted = set(sys.argv[2:]) +stmts = ["create schema if not exists raw;"] +for stream in tap.streams.values(): + if stream.name in wanted: + cols = ", ".join(f'"{c}" text' for c in stream.schema["properties"]) + stmts.append(f'drop table if exists raw.{stream.name} cascade; create table raw.{stream.name} ({cols});') + print("raw." + stream.name) +srv = pgserver.get_server(os.environ["SCRATCH"] + "/dbt_local/pgdata", cleanup_mode=None) +srv.psql("\\c school_compare\n" + "\n".join(stmts)) +EOF +chmod +x "$SCRATCH/dbt_local/dbt.sh" "$SCRATCH/dbt_local/sql.sh" +``` + +- [ ] **Step 2: Raw tables for GIAS and the school MI** + +```bash +cd /Users/tudor/projects/school_compare +SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-gias --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_gias.tap:TapUKGIAS gias_establishments gias_links +SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_ofsted.tap:TapUKOfsted ofsted_inspections +``` + +Expected: `raw.gias_establishments`, `raw.gias_links`, `raw.ofsted_inspections`. + +- [ ] **Step 3: Baseline build** + +Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_ofsted_inspections int_ofsted_latest fact_ofsted_latest dim_school` +Expected: exit 0; the 10 `int_ofsted_latest` and 2 `dim_school` unit tests PASS on empty tables. If `pgserver` cannot start, stop and report: dbt tests are then proven only by the staging DAG runs, and the PR description must say so. + +### Task 2: GIAS inspectorate and UKPRN + +**Files:** +- Modify: `pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py` (after `th.Property("StatutoryHighAge", th.StringType),`) +- Modify: `pipeline/transform/models/staging/stg_gias_establishments.sql` +- Create: `pipeline/transform/models/staging/stg_gias_ukprn.sql` +- Modify: `pipeline/dags/school_data_pipeline.py` (daily `dbt_build` selection) +- Test: `pipeline/tests/test_gias_declared_fields.py` + +**Interfaces:** +- Produces: raw columns `"UKPRN"`, `"InspectorateName (name)"`; `stg_gias_establishments.inspectorate_name text`; `stg_gias_ukprn(urn integer, ukprn text)`. + +- [ ] **Step 1: Write the failing test** + +`pipeline/tests/test_gias_declared_fields.py`: + +```python +"""Every field the GIAS establishments stream declares must be a column of the +extract GIAS publishes, or it silently loads as null. H4 adds UKPRN (matches +Ofsted's further-education providers) and InspectorateName (name) ("ISI" for +about 1,340 independent schools). HEADER is GIAS's own, from +edubasealldata20261013.csv. +""" +import re +from pathlib import Path + +TAP = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-gias' + / 'tap_uk_gias' / 'tap.py') + +HEADER = ( + 'URN', 'LA (code)', 'LA (name)', 'EstablishmentNumber', 'EstablishmentName', + 'TypeOfEstablishment (code)', 'TypeOfEstablishment (name)', 'EstablishmentTypeGroup (code)', + 'EstablishmentTypeGroup (name)', 'EstablishmentStatus (code)', 'EstablishmentStatus (name)', + 'ReasonEstablishmentOpened (code)', 'ReasonEstablishmentOpened (name)', 'OpenDate', + 'ReasonEstablishmentClosed (code)', 'ReasonEstablishmentClosed (name)', 'CloseDate', + 'PhaseOfEducation (code)', 'PhaseOfEducation (name)', 'StatutoryLowAge', 'StatutoryHighAge', + 'Boarders (code)', 'Boarders (name)', 'NurseryProvision (name)', 'OfficialSixthForm (code)', + 'OfficialSixthForm (name)', 'Gender (code)', 'Gender (name)', 'ReligiousCharacter (code)', + 'ReligiousCharacter (name)', 'ReligiousEthos (name)', 'Diocese (code)', 'Diocese (name)', + 'AdmissionsPolicy (code)', 'AdmissionsPolicy (name)', 'SchoolCapacity', 'SpecialClasses (code)', + 'SpecialClasses (name)', 'CensusDate', 'NumberOfPupils', 'NumberOfBoys', 'NumberOfGirls', + 'PercentageFSM', 'TrustSchoolFlag (code)', 'TrustSchoolFlag (name)', 'Trusts (code)', + 'Trusts (name)', 'SchoolSponsorFlag (name)', 'SchoolSponsors (name)', 'FederationFlag (name)', + 'Federations (code)', 'Federations (name)', 'UKPRN', 'FEHEIdentifier', + 'FurtherEducationType (name)', 'LastChangedDate', 'Street', 'Locality', 'Address3', 'Town', + 'County (name)', 'Postcode', 'SchoolWebsite', 'TelephoneNum', 'HeadTitle (name)', + 'HeadFirstName', 'HeadLastName', 'HeadPreferredJobTitle', 'BSOInspectorateName (name)', + 'InspectorateReport', 'DateOfLastInspectionVisit', 'NextInspectionVisit', 'TeenMoth (name)', + 'TeenMothPlaces', 'CCF (name)', 'SENPRU (name)', 'EBD (name)', 'PlacesPRU', 'FTProv (name)', + 'EdByOther (name)', 'Section41Approved (name)', 'SEN1 (name)', 'SEN2 (name)', 'SEN3 (name)', + 'SEN4 (name)', 'SEN5 (name)', 'SEN6 (name)', 'SEN7 (name)', 'SEN8 (name)', 'SEN9 (name)', + 'SEN10 (name)', 'SEN11 (name)', 'SEN12 (name)', 'SEN13 (name)', + 'TypeOfResourcedProvision (name)', 'ResourcedProvisionOnRoll', 'ResourcedProvisionCapacity', + 'SenUnitOnRoll', 'SenUnitCapacity', 'GOR (code)', 'GOR (name)', + 'DistrictAdministrative (code)', 'DistrictAdministrative (name)', 'AdministrativeWard (code)', + 'AdministrativeWard (name)', 'ParliamentaryConstituency (code)', + 'ParliamentaryConstituency (name)', 'UrbanRural (code)', 'UrbanRural (name)', + 'GSSLACode (name)', 'Easting', 'Northing', 'MSOA (name)', 'LSOA (name)', + 'InspectorateName (name)', 'SENStat', 'SENNoStat', 'BoardingEstablishment (name)', 'PropsName', + 'PreviousLA (code)', 'PreviousLA (name)', 'PreviousEstablishmentNumber', 'Country (name)', + 'UPRN', 'SiteName', 'QABName (code)', 'QABName (name)', 'EstablishmentAccredited (code)', + 'EstablishmentAccredited (name)', 'QABReport', 'CHNumber', 'MSOA (code)', 'LSOA (code)', 'FSM', + 'AccreditationExpiryDate', +) + + +def declared(): + source = TAP.read_text() + block = source[source.index('class GIASEstablishmentsStream'):source.index('class GIASLinksStream')] + return re.findall(r'th\.Property\("([^"]+)"', block) + + +def test_every_declared_field_is_a_gias_column(): + assert set(declared()) - set(HEADER) == set() + + +def test_the_inspection_fields_are_declared(): + assert {'UKPRN', 'InspectorateName (name)'} <= set(declared()) +``` + +- [ ] **Step 2: Run it to see it fail** + +Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_gias_declared_fields.py -q` +Expected: `test_the_inspection_fields_are_declared` FAILS; `test_every_declared_field_is_a_gias_column` passes (a guard for existing fields). + +- [ ] **Step 3: Declare the fields, stage them, add the UKPRN view** + +In `tap_uk_gias/tap.py`, after `th.Property("StatutoryHighAge", th.StringType),`: + +```python + # Matches Ofsted's further-education providers, which some FE + # records name by UKPRN only (audit H4). + th.Property("UKPRN", th.StringType), + # "ISI" for independent schools the Independent Schools Inspectorate + # inspects; "Ofsted" or "Not applicable" otherwise (audit H4). + th.Property("InspectorateName (name)", th.StringType), +``` + +In `stg_gias_establishments.sql`, after the line `"NurseryProvision (name)" as nursery_provision,`: + +```sql + nullif(trim("InspectorateName (name)"), '') as inspectorate_name, +``` + +`pipeline/transform/models/staging/stg_gias_ukprn.sql`: + +```sql +-- URN → UKPRN from the raw GIAS extract, for matching Ofsted's further +-- education providers (int_ofsted_fe_latest, int_ofsted_listed; audit H4). +-- A model on the raw source rather than a column of stg_gias_establishments: +-- a ref to that model would put the Ofsted models into the daily build's +-- stg_gias_establishments+ selection (pipeline/tests/test_dag_selectors.py). + +select + cast("URN" as integer) as urn, + nullif(trim("UKPRN"), '') as ukprn +from {{ source('raw', 'gias_establishments') }} +where "URN" ~ '^[0-9]+$' +``` + +In `pipeline/dags/school_data_pipeline.py`, in the daily build, change `--select stg_gias_establishments+ stg_gias_links+ gias_code_names+` to `--select stg_gias_establishments+ stg_gias_links+ gias_code_names+ stg_gias_ukprn`. + +- [ ] **Step 4: Run the tests and build** + +Run: the Step 2 pytest command → `2 passed`. +Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_dag_selectors.py -q` → all pass. +Re-create the GIAS raw tables (Task 1 Step 2, first command) → the table now has `"UKPRN"` and `"InspectorateName (name)"`. +Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_gias_ukprn dim_school` → exit 0. + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/tests/test_gias_declared_fields.py pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py pipeline/transform/models/staging/stg_gias_establishments.sql pipeline/transform/models/staging/stg_gias_ukprn.sql pipeline/dags/school_data_pipeline.py +git commit -m "feat(gias): load UKPRN and the inspectorate GIAS names (H4)" +``` + +### Task 3: Ofsted further-education extractor + +**Files:** +- Create: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py` +- Modify: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (imports, two stream classes, `discover_streams`) +- Test: `pipeline/tests/test_ofsted_fe_mi.py` + +**Interfaces:** +- Produces: `fe_pages(results: list[dict]) -> list[str]`; `renewed_framework_pages(results: list[dict]) -> list[str]`; `latest_as_at_csv(html: str) -> str | None`; `latest_period_csv(html: str) -> str | None`; `read_mi_csv(content: bytes) -> pd.DataFrame`; `records(df, columns: dict[str, str], key: str) -> Iterator[dict]`; `providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None`; `PROVIDER_COLUMNS`, `INSPECTION_COLUMNS`. Streams `ofsted_fe_providers` (key `provider_urn`; fields `provider_urn, provider_ukprn, provider_name, provider_type, renewed_eif_full_inspection, graded_inspection_date, overall_effectiveness, short_inspection_date`) and `ofsted_fe_inspections` (key `inspection_number`; fields `inspection_number, provider_urn, provider_ukprn, first_day_of_inspection, date_published, safeguarding_standards`), all strings. + +- [ ] **Step 1: Write the failing tests** + +`pipeline/tests/test_ofsted_fe_mi.py`: + +```python +"""Ofsted publishes further-education inspection outcomes on a gov.uk page per +academic year, separate from the school MI the tap already reads (audit H4). +The links below are the real ones from the 2025/26 page; the headers are +Ofsted's own. +""" +import importlib.util +from pathlib import Path + +import pandas as pd +import pytest + +MODULE = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-ofsted' + / 'tap_uk_ofsted' / 'fe_mi.py') + +A = 'https://assets.publishing.service.gov.uk/media/' +PAGE_2025_26 = f''' +x +x +x +x +x +x +x +x +x +''' + +SEARCH_RESULTS = [ + {'title': 'Further education and skills inspections and outcomes: management information from September 2024 to August 2025', + 'link': '/government/statistical-data-sets/fe-2024-25'}, + {'title': 'Further education and skills inspections and outcomes: management information from November 2025 to August 2026', + 'link': '/government/statistical-data-sets/fe-2025-26'}, + {'title': 'Further education and skills inspections and outcomes: management information from September 2023 to August 2024', + 'link': '/government/statistical-data-sets/fe-2023-24'}, + {'title': 'Monthly management information: Ofsted\'s school inspections outcomes', + 'link': '/government/statistical-data-sets/schools'}, +] + +PROVIDER_HEADER = ( + 'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group', + 'Local authority', 'Region', 'Ofsted region', + 'Has the provider received a full inspection under the renewed EIF?', + 'Has the provider received a new provider monitoring inspection?', + 'Date of latest short inspection before renewed EIF (if relevant)', + 'Inspection number of latest full inspection before renewed EIF', + 'First day of inspection of latest inspection before renewed EIF', + 'Latest overall effectiveness before renewed EIF', + 'Latest quality of education before renewed EIF', + 'Latest behaviour and attitudes before renewed EIF', + 'Latest personal development before renewed EIF', + 'Latest leadership and management before renewed EIF', + 'Latest contribution to meeting skills needs before renewed EIF', + 'Latest safeguarding before renewed EIF', '', +) + +INSPECTION_HEADER = ( + 'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group', + 'Local authority', 'Region', 'Ofsted region', 'Inspection number', 'First day of inspection', + 'Date published', 'Safeguarding standards', 'Inclusion', 'Leadership and governance', + 'Contribution to meeting skills needs', + 'Education programmes for young people - curriculum, teaching and training', + 'Education programmes for young people - achievement', + 'Education programmes for young people - participation and development', + 'Adult learning programmes - curriculum, teaching and training', + 'Adult learning programmes - achievement', + 'Adult learning programmes - participation and development', + 'Apprenticeships - curriculum, teaching and training', 'Apprenticeships - achievement', + 'Apprenticeships - participation and development', + 'Provision for learners and apprentices with high needs - curriculum, teaching and training', + 'Provision for learners and apprentices with high needs - achievement', + 'Provision for learners and apprentices with high needs - participation and development', '', +) + + +@pytest.fixture +def fe(): + spec = importlib.util.spec_from_file_location('fe_mi', MODULE) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def test_the_newest_as_at_file_is_chosen(fe): + assert fe.latest_as_at_csv(PAGE_2025_26).endswith('_as_at_31_August_2026.csv') + + +def test_the_latest_cumulative_inspections_file_is_chosen_not_a_monitoring_one(fe): + assert fe.latest_period_csv(PAGE_2025_26) == ( + A + '6a9feda65f6e942efe37eed6/Management_information_-_further_education_and_skills_' + '-_10_November_2025_to_31_August_2026.csv') + + +def test_a_doubled_underscore_file_name_still_parses(fe): + page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'August' not in line) + assert fe.latest_period_csv(page).endswith('__10_November_2025_to_28_February_2026.csv') + + +def test_monitoring_files_alone_give_nothing(fe): + page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'monitoring' in line) + assert fe.latest_period_csv(page) is None + assert fe.latest_as_at_csv(page) is None + + +def test_pages_are_the_fe_series_newest_first(fe): + assert fe.fe_pages(SEARCH_RESULTS) == [ + '/government/statistical-data-sets/fe-2025-26', + '/government/statistical-data-sets/fe-2024-25', + '/government/statistical-data-sets/fe-2023-24', + ] + + +def test_report_cards_are_read_only_from_renewed_framework_pages(fe): + # Earlier pages hold previous-framework inspection lists, not report cards. + assert fe.renewed_framework_pages(SEARCH_RESULTS) == ['/government/statistical-data-sets/fe-2025-26'] + + +def test_providers_fall_back_to_the_previous_page(fe): + pages = {'/new': '

No files yet

', '/old': PAGE_2025_26} + assert fe.providers_csv_url(['/new', '/old'], pages.__getitem__).endswith('_as_at_31_August_2026.csv') + + +@pytest.mark.parametrize('columns, header', [('PROVIDER_COLUMNS', PROVIDER_HEADER), + ('INSPECTION_COLUMNS', INSPECTION_HEADER)]) +def test_every_mapped_column_is_in_ofsteds_file(fe, columns, header): + assert set(getattr(fe, columns)) - set(header) == set() + + +def test_csv_reading_and_records(fe): + content = ('' + ','.join(f'"{c}"' for c in PROVIDER_HEADER[:14]) + '\n' + + '145175,10005429,Queen Mary\'s College,Sixth form college,Sixth form colleges,Hampshire,' + 'South East,South East,No,No,-,10227183,06/12/2022,2\n' + + ',,Blank provider,,,,,,,,,,,\n').encode('utf-8') + rows = list(fe.records(fe.read_mi_csv(content), fe.PROVIDER_COLUMNS, 'provider_urn')) + assert rows == [{ + 'provider_urn': '145175', 'provider_ukprn': '10005429', 'provider_name': "Queen Mary's College", + 'provider_type': 'Sixth form college', 'renewed_eif_full_inspection': 'No', + 'graded_inspection_date': '06/12/2022', 'overall_effectiveness': '2', 'short_inspection_date': '-', + }] +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_ofsted_fe_mi.py -q` +Expected: every test errors with `FileNotFoundError` for `fe_mi.py`. + +- [ ] **Step 3: Write the module** + +`pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py`: + +```python +"""Ofsted's further education and skills management information (audit H4). + +Ofsted publishes FE inspection outcomes on one gov.uk page per academic year +("Further education and skills inspections and outcomes: management +information from November 2025 to August 2026"). Each page carries "as at" +files (one row per provider, with its latest inspection under the previous +framework) and cumulative files of renewed-framework inspections, i.e. report +cards ("…_10_November_2025_to_31_January_2026.csv", "…_to_28_February_2026.csv", +…). Monitoring-visit files are a different series. + +Free of the Singer SDK so CI's pytest, which installs only the backend's +requirements, can load it. +""" +from __future__ import annotations + +import io +import re +from datetime import date +from typing import Callable, Iterator + +import pandas as pd + +SEARCH_URL = "https://www.gov.uk/api/search.json" +SEARCH_QUERY = "further education and skills inspections and outcomes management information" +PAGE_TITLE_PREFIX = "Further education and skills inspections and outcomes: management information from " +# Renewed-framework (report card) inspections began on 10 November 2025. +RENEWED_FRAMEWORK_START = date(2025, 11, 1) + +_MONTHS = {name: number for number, name in enumerate( + ["january", "february", "march", "april", "may", "june", "july", + "august", "september", "october", "november", "december"], start=1)} +_TITLE_START = re.compile(r"from ([A-Za-z]+) (\d{4})") +_CSV_LINK = re.compile(r'https://assets\.publishing\.service\.gov\.uk/[^"\s<>]+\.csv') +_AS_AT = re.compile(r"further_education_and_skills_-_+as_at_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$") +_PERIOD = re.compile( + r"further_education_and_skills_-_+(\d{1,2})_([A-Za-z]+)_(\d{4})_to_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$") + +# Ofsted column → Singer field. +PROVIDER_COLUMNS = { + "Provider URN": "provider_urn", + "Provider UKPRN": "provider_ukprn", + "Provider name": "provider_name", + "Provider type": "provider_type", + "Has the provider received a full inspection under the renewed EIF?": "renewed_eif_full_inspection", + "First day of inspection of latest inspection before renewed EIF": "graded_inspection_date", + "Latest overall effectiveness before renewed EIF": "overall_effectiveness", + "Date of latest short inspection before renewed EIF (if relevant)": "short_inspection_date", +} +INSPECTION_COLUMNS = { + "Provider URN": "provider_urn", + "Provider UKPRN": "provider_ukprn", + "Inspection number": "inspection_number", + "First day of inspection": "first_day_of_inspection", + "Date published": "date_published", + "Safeguarding standards": "safeguarding_standards", +} + + +def _date(day: str, month: str, year: str) -> date | None: + number = _MONTHS.get(month.lower()) + return date(int(year), number, int(day)) if number else None + + +def _page_start(title: str) -> date | None: + match = _TITLE_START.search(title) + number = _MONTHS.get(match.group(1).lower()) if match else None + return date(int(match.group(2)), number, 1) if number else None + + +def _dated_pages(results: list[dict]) -> list[tuple[date, str]]: + pages = [] + for result in results: + title = result.get("title", "") + start = _page_start(title) if title.startswith(PAGE_TITLE_PREFIX) else None + if start: + pages.append((start, result["link"])) + return sorted(pages, reverse=True) + + +def fe_pages(results: list[dict]) -> list[str]: + """gov.uk paths of the FE MI pages, newest academic year first.""" + return [link for _, link in _dated_pages(results)] + + +def renewed_framework_pages(results: list[dict]) -> list[str]: + """The pages that can hold report cards: from November 2025 onwards.""" + return [link for start, link in _dated_pages(results) if start >= RENEWED_FRAMEWORK_START] + + +def _latest(html: str, pattern: re.Pattern, date_groups: slice) -> str | None: + best = None + for link in set(_CSV_LINK.findall(html)): + match = pattern.search(link) + when = _date(*match.groups()[date_groups]) if match else None + if when and (best is None or when > best[0]): + best = (when, link) + return best[1] if best else None + + +def latest_as_at_csv(html: str) -> str | None: + """The newest "as at" CSV on a page.""" + return _latest(html, _AS_AT, slice(0, 3)) + + +def latest_period_csv(html: str) -> str | None: + """The renewed-framework inspections CSV with the latest end date on a page.""" + return _latest(html, _PERIOD, slice(3, 6)) + + +def providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None: + """The newest "as at" CSV, from the newest page that has one.""" + for page in pages: + url = latest_as_at_csv(fetch(page)) + if url: + return url + return None + + +def read_mi_csv(content: bytes) -> pd.DataFrame: + """Ofsted's MI CSVs: UTF-8 with a BOM, or Windows-1252. The header is the + line naming "Provider URN".""" + try: + text = content.decode("utf-8-sig") + except UnicodeDecodeError: + text = content.decode("cp1252") + lines = text.splitlines() + start = next((i for i, line in enumerate(lines[:20]) if "Provider URN" in line), 0) + return pd.read_csv(io.StringIO("\n".join(lines[start:])), dtype=str, keep_default_na=False) + + +def records(df: pd.DataFrame, columns: dict[str, str], key: str) -> Iterator[dict]: + """Singer records: the mapped columns, stripped. Rows without `key` are skipped.""" + present = {source: field for source, field in columns.items() if source in df.columns} + for _, row in df.iterrows(): + record = {field: str(row[source]).strip() for source, field in present.items()} + if record.get(key): + yield record +``` + +- [ ] **Step 4: Add the two streams** + +In `tap_uk_ofsted/tap.py`, after `from singer_sdk import typing as th`: + +```python + +from tap_uk_ofsted.fe_mi import ( + INSPECTION_COLUMNS, + PROVIDER_COLUMNS, + SEARCH_QUERY, + SEARCH_URL, + fe_pages, + latest_period_csv, + providers_csv_url, + read_mi_csv, + records, + renewed_framework_pages, +) +``` + +Before `class TapUKOfsted(Tap):`: + +```python +def _fe_search_results() -> list[dict]: + """gov.uk search results for Ofsted's FE MI pages (one per academic year).""" + resp = requests.get(SEARCH_URL, params={"q": SEARCH_QUERY, "count": 50}, timeout=60) + resp.raise_for_status() + return resp.json().get("results", []) + + +def _gov_uk_page(path: str) -> str: + url = path if path.startswith("http") else "https://www.gov.uk" + path + resp = requests.get(url, timeout=60) + resp.raise_for_status() + return resp.text + + +class OfstedFEProvidersStream(Stream): + """Ofsted's further education and skills providers: the newest "as at" + file, one row per provider with its latest inspection under the previous + framework (audit H4). Page and file selection: fe_mi.py.""" + + name = "ofsted_fe_providers" + primary_keys = ["provider_urn"] + replication_key = None + + schema = th.PropertiesList( + th.Property("provider_urn", th.StringType, required=True), + *[th.Property(field, th.StringType) for field in PROVIDER_COLUMNS.values() if field != "provider_urn"], + ).to_dict() + + def get_records(self, context): + url = providers_csv_url(fe_pages(_fe_search_results()), _gov_uk_page) + if not url: + self.logger.error("No Ofsted FE 'as at' file found") + return + self.logger.info("Downloading Ofsted FE providers MI: %s", url) + resp = requests.get(url, timeout=120) + resp.raise_for_status() + yield from records(read_mi_csv(resp.content), PROVIDER_COLUMNS, "provider_urn") + + +class OfstedFEInspectionsStream(Stream): + """Ofsted's renewed-framework FE inspections (report cards), from the + latest cumulative file on every page since November 2025, so a report + card survives the next academic year's page (audit H4).""" + + name = "ofsted_fe_inspections" + primary_keys = ["inspection_number"] + replication_key = None + + schema = th.PropertiesList( + th.Property("inspection_number", th.StringType, required=True), + *[th.Property(field, th.StringType) for field in INSPECTION_COLUMNS.values() if field != "inspection_number"], + ).to_dict() + + def get_records(self, context): + for page in renewed_framework_pages(_fe_search_results()): + url = latest_period_csv(_gov_uk_page(page)) + if not url: + self.logger.warning("No renewed-framework FE inspections file on %s", page) + continue + self.logger.info("Downloading Ofsted FE inspections MI: %s", url) + resp = requests.get(url, timeout=120) + resp.raise_for_status() + yield from records(read_mi_csv(resp.content), INSPECTION_COLUMNS, "inspection_number") +``` + +Change `discover_streams` to `return [OfstedInspectionsStream(self), OfstedFEProvidersStream(self), OfstedFEInspectionsStream(self)]`. + +- [ ] **Step 5: Run the tests, then both streams against gov.uk** + +Run: the Step 2 pytest command → `11 passed`. + +```bash +uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted python -c " +from tap_uk_ofsted.tap import TapUKOfsted +tap = TapUKOfsted(config={}) +p = list(tap.streams['ofsted_fe_providers'].get_records(None)) +i = list(tap.streams['ofsted_fe_inspections'].get_records(None)) +print(len(p), [r for r in p if r['provider_urn'] in ('145175', '143526', '130645')]) +print(len(i), [r for r in i if r['provider_urn'] == '143526'])" +``` + +Expected: about 1,979 providers (Ofsted's count as at 31 August 2026; a newer file may differ slightly) with Queen Mary's College graded `2` on `06/12/2022`, The Autism Project – CareTrade `2` on `16/11/2021`, Exeter and North Devon Colleges Group `-`; about 1,472 inspections with 143526 inspected `12/05/2026`, safeguarding `Met`. + +- [ ] **Step 6: Commit** + +```bash +git add pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py pipeline/tests/test_ofsted_fe_mi.py +git commit -m "feat(ofsted): extract Ofsted's further education and skills MI (H4)" +``` + +### Task 4: FE staging models + +**Files:** +- Modify: `pipeline/transform/models/staging/_stg_sources.yml` +- Create: `pipeline/transform/models/staging/stg_ofsted_fe_providers.sql`, `stg_ofsted_fe_inspections.sql`, `stg_ofsted_fe.yml` + +**Interfaces:** +- Consumes: raw `ofsted_fe_providers`, `ofsted_fe_inspections` (Task 3 fields). +- Produces: `stg_ofsted_fe_providers(provider_urn integer, provider_ukprn text, provider_name text, provider_type text, renewed_eif_full_inspection boolean, graded_inspection_date date, overall_effectiveness integer, short_inspection_date date)`; `stg_ofsted_fe_inspections(inspection_number text, provider_urn integer, provider_ukprn text, rc_inspection_date date, published_date date, rc_safeguarding_met boolean)`. + +- [ ] **Step 1: Declare the sources, create the raw tables, write the unit tests** + +In `_stg_sources.yml`, after the `ofsted_inspections` entry: + +```yaml + - name: ofsted_fe_providers + description: Ofsted further education and skills MI, newest "as at" file — one row per provider with its latest inspection under the previous framework + + - name: ofsted_fe_inspections + description: Ofsted further education and skills MI — renewed-framework inspections (report cards) since 10 November 2025, one row per inspection +``` + +Create the raw tables: `SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_ofsted.tap:TapUKOfsted ofsted_fe_providers ofsted_fe_inspections` (from the repo root) → two `raw.…` lines. + +`pipeline/transform/models/staging/stg_ofsted_fe.yml`: + +```yaml +version: 2 + +unit_tests: + - name: stg_ofsted_fe_providers_parses_grades_and_dates + description: Ofsted writes "-" for no grade or date; grades are 1-4. + model: stg_ofsted_fe_providers + given: + - input: source('raw', 'ofsted_fe_providers') + rows: + - {provider_urn: '145175', provider_ukprn: '10005429', provider_name: "Queen Mary's College", renewed_eif_full_inspection: 'No', graded_inspection_date: '06/12/2022', overall_effectiveness: '2', short_inspection_date: '-'} + - {provider_urn: '130645', provider_ukprn: '-', provider_name: 'Exeter and North Devon Colleges Group', renewed_eif_full_inspection: 'No', graded_inspection_date: '-', overall_effectiveness: '-', short_inspection_date: '-'} + - {provider_urn: '', provider_ukprn: '10000001', provider_name: 'No URN'} + expect: + rows: + - {provider_urn: 145175, provider_ukprn: '10005429', renewed_eif_full_inspection: false, graded_inspection_date: '2022-12-06', overall_effectiveness: 2, short_inspection_date: null} + - {provider_urn: 130645, provider_ukprn: null, renewed_eif_full_inspection: false, graded_inspection_date: null, overall_effectiveness: null, short_inspection_date: null} + + - name: stg_ofsted_fe_inspections_parses_dates_and_safeguarding + model: stg_ofsted_fe_inspections + given: + - input: source('raw', 'ofsted_fe_inspections') + rows: + - {inspection_number: '10380001', provider_urn: '143526', provider_ukprn: '10000002', first_day_of_inspection: '12/05/2026', date_published: '16/06/2026', safeguarding_standards: 'Met'} + - {inspection_number: '10380002', provider_urn: '150042', provider_ukprn: '10000003', first_day_of_inspection: '20/01/2026', date_published: '24/02/2026', safeguarding_standards: 'Not met'} + - {inspection_number: '10380003', provider_urn: '1', provider_ukprn: '', first_day_of_inspection: '', date_published: '', safeguarding_standards: ''} + expect: + rows: + - {inspection_number: '10380001', provider_urn: 143526, rc_inspection_date: '2026-05-12', published_date: '2026-06-16', rc_safeguarding_met: true} + - {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', published_date: '2026-02-24', rc_safeguarding_met: false} +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "stg_ofsted_fe_providers stg_ofsted_fe_inspections,test_type:unit"` +Expected: non-zero exit; parsing fails because the two models do not exist. + +- [ ] **Step 3: Write the models** + +`pipeline/transform/models/staging/stg_ofsted_fe_providers.sql`: + +```sql +-- Staging model: Ofsted further education and skills providers — the "as at" +-- MI file, one row per provider with its latest inspection under the previous +-- framework (audit H4). Ofsted writes '-' for no grade or date. + +select + cast(trim(provider_urn) as integer) as provider_urn, + nullif(nullif(trim(provider_ukprn), ''), '-') as provider_ukprn, + provider_name, + provider_type, + lower(trim(renewed_eif_full_inspection)) = 'yes' as renewed_eif_full_inspection, + case when trim(graded_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$' + then to_date(trim(graded_inspection_date), 'DD/MM/YYYY') end as graded_inspection_date, + case when trim(overall_effectiveness) in ('1', '2', '3', '4') + then trim(overall_effectiveness)::integer end as overall_effectiveness, + case when trim(short_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$' + then to_date(trim(short_inspection_date), 'DD/MM/YYYY') end as short_inspection_date +from {{ source('raw', 'ofsted_fe_providers') }} +where trim(provider_urn) ~ '^[0-9]+$' +``` + +`pipeline/transform/models/staging/stg_ofsted_fe_inspections.sql`: + +```sql +-- Staging model: Ofsted's renewed-framework FE inspections (report cards), +-- one row per inspection (audit H4). Only the headline is kept: the date and +-- the safeguarding result. + +select + trim(inspection_number) as inspection_number, + cast(trim(provider_urn) as integer) as provider_urn, + nullif(nullif(trim(provider_ukprn), ''), '-') as provider_ukprn, + to_date(trim(first_day_of_inspection), 'DD/MM/YYYY') as rc_inspection_date, + case when trim(date_published) ~ '^\d{2}/\d{2}/\d{4}$' + then to_date(trim(date_published), 'DD/MM/YYYY') end as published_date, + case lower(trim(safeguarding_standards)) + when 'met' then true + when 'not met' then false + end as rc_safeguarding_met +from {{ source('raw', 'ofsted_fe_inspections') }} +where trim(provider_urn) ~ '^[0-9]+$' + and trim(first_day_of_inspection) ~ '^\d{2}/\d{2}/\d{4}$' +``` + +- [ ] **Step 4: Run the unit tests and build** + +Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_ofsted_fe_providers stg_ofsted_fe_inspections` +Expected: exit 0; both unit tests PASS; both views created. + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/staging/_stg_sources.yml pipeline/transform/models/staging/stg_ofsted_fe_providers.sql pipeline/transform/models/staging/stg_ofsted_fe_inspections.sql pipeline/transform/models/staging/stg_ofsted_fe.yml +git commit -m "feat(dbt): stage Ofsted's further education and skills MI (H4)" +``` + +### Task 5: `int_ofsted_fe_latest` + +**Files:** +- Create: `pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`, `int_ofsted_fe_latest.yml` + +**Interfaces:** +- Consumes: `stg_gias_ukprn`, `stg_ofsted_fe_providers`, `stg_ofsted_fe_inspections`. +- Produces: `int_ofsted_fe_latest` with exactly `int_ofsted_latest`'s columns, in its order and types: `urn, latest_visit_date, latest_visit_kind, latest_visit_outcome, current_grade, current_grade_date, current_grade_basis, graded_inspection_date, ungraded_inspection_date, inspection_date, inspection_type, framework, overall_effectiveness, quality_of_education, behaviour_attitudes, personal_development, leadership_management, early_years_provision, sixth_form_provision, ungraded_outcome, ungraded_grade, rc_safeguarding_met, rc_inclusion, rc_curriculum_teaching, rc_achievement, rc_attendance_behaviour, rc_personal_development, rc_leadership_governance, rc_early_years, rc_sixth_form, rc_inspection_date, report_url`. `latest_visit_kind` ∈ `report_card`, `graded`, `short`. + +- [ ] **Step 1: Write the unit tests** + +`pipeline/transform/models/intermediate/int_ofsted_fe_latest.yml`: + +```yaml +version: 2 + +unit_tests: + - name: int_ofsted_fe_grade_only + description: Queen Mary's College (145175), Good on 6 December 2022, no report card. + model: int_ofsted_fe_latest + given: + - input: ref('stg_gias_ukprn') + rows: + - {urn: 145175, ukprn: '10005429'} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 145175, provider_ukprn: '10005429', graded_inspection_date: '2022-12-06', overall_effectiveness: 2} + - input: ref('stg_ofsted_fe_inspections') + rows: [] + expect: + rows: + - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, rc_inspection_date: null, framework: 'Further education and skills'} + + - name: int_ofsted_fe_short_inspection_is_a_visit_not_a_grade + model: int_ofsted_fe_latest + given: + - input: ref('stg_gias_ukprn') + rows: + - {urn: 1, ukprn: null} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 1, graded_inspection_date: '2019-03-05', overall_effectiveness: 2, short_inspection_date: '2023-02-01'} + - input: ref('stg_ofsted_fe_inspections') + rows: [] + expect: + rows: + - {urn: 1, latest_visit_date: '2023-02-01', latest_visit_kind: short, current_grade: 2, current_grade_date: '2019-03-05', current_grade_basis: graded} + + - name: int_ofsted_fe_newer_report_card_wins + description: The Autism Project - CareTrade (143526). Good 2021; report card 12 May 2026, safeguarding met. + model: int_ofsted_fe_latest + given: + - input: ref('stg_gias_ukprn') + rows: + - {urn: 143526, ukprn: '10000002'} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 143526, provider_ukprn: '10000002', graded_inspection_date: '2021-11-16', overall_effectiveness: 2} + - input: ref('stg_ofsted_fe_inspections') + rows: + - {inspection_number: '10380001', provider_urn: 143526, provider_ukprn: '10000002', rc_inspection_date: '2026-05-12', rc_safeguarding_met: true} + expect: + rows: + - {urn: 143526, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, current_grade: null, current_grade_date: null, current_grade_basis: null, rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, overall_effectiveness: 2} + + - name: int_ofsted_fe_report_card_only + model: int_ofsted_fe_latest + given: + - input: ref('stg_gias_ukprn') + rows: + - {urn: 150042, ukprn: null} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 150042, graded_inspection_date: null, overall_effectiveness: null} + - input: ref('stg_ofsted_fe_inspections') + rows: + - {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false} + expect: + rows: + - {urn: 150042, latest_visit_date: '2026-01-20', latest_visit_kind: report_card, current_grade: null, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false} + + - name: int_ofsted_fe_listed_never_inspected_has_no_row + model: int_ofsted_fe_latest + given: + - input: ref('stg_gias_ukprn') + rows: + - {urn: 130645, ukprn: null} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 130645, graded_inspection_date: null, overall_effectiveness: null} + - input: ref('stg_ofsted_fe_inspections') + rows: [] + expect: + rows: [] + + - name: int_ofsted_fe_matches_by_ukprn_when_the_urn_differs + model: int_ofsted_fe_latest + given: + - input: ref('stg_gias_ukprn') + rows: + - {urn: 5, ukprn: '10099999'} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 9999999, provider_ukprn: '10099999', graded_inspection_date: '2024-01-30', overall_effectiveness: 3} + - input: ref('stg_ofsted_fe_inspections') + rows: [] + expect: + rows: + - {urn: 5, current_grade: 3, current_grade_date: '2024-01-30'} + + - name: int_ofsted_fe_prefers_the_urn_match + model: int_ofsted_fe_latest + given: + - input: ref('stg_gias_ukprn') + rows: + - {urn: 6, ukprn: '10088888'} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 6, provider_ukprn: '10077777', graded_inspection_date: '2023-05-01', overall_effectiveness: 1} + - {provider_urn: 8888888, provider_ukprn: '10088888', graded_inspection_date: '2024-05-01', overall_effectiveness: 4} + - input: ref('stg_ofsted_fe_inspections') + rows: [] + expect: + rows: + - {urn: 6, current_grade: 1, current_grade_date: '2023-05-01'} + + - name: int_ofsted_fe_report_card_older_than_grade_is_ignored + description: Bad data guard. A report card dated before the graded inspection cannot be the latest, and a row must never carry both a grade and a report card. + model: int_ofsted_fe_latest + given: + - input: ref('stg_gias_ukprn') + rows: + - {urn: 7, ukprn: null} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 7, graded_inspection_date: '2026-03-01', overall_effectiveness: 2} + - input: ref('stg_ofsted_fe_inspections') + rows: + - {inspection_number: '1', provider_urn: 7, rc_inspection_date: '2026-01-10', rc_safeguarding_met: true} + expect: + rows: + - {urn: 7, latest_visit_kind: graded, current_grade: 2, rc_inspection_date: null, rc_safeguarding_met: null} +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_fe_latest,test_type:unit"` +Expected: non-zero exit; parsing fails (model missing). + +- [ ] **Step 3: Write the model** + +`pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`: + +```sql +-- Intermediate model: Ofsted further education and skills status per URN +-- (audit H4), in int_ofsted_latest's columns so it can be unioned there. +-- A provider matches a school by Provider URN, else by UKPRN (stg_gias_ukprn). +-- Rule, in the C1 status rule's terms +-- (docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md): +-- a report card on or after the previous-framework inspection wins and +-- leaves no grade in force; otherwise a previous-framework grade 1-4 is the +-- current grade, dated by that inspection; the latest visit is the later of +-- it and any short inspection (whose outcome the MI does not give). + +with gias as ( + select urn, ukprn from {{ ref('stg_gias_ukprn') }} +), + +providers as ( + select distinct on (g.urn) + g.urn, + p.graded_inspection_date, + p.overall_effectiveness, + p.short_inspection_date + from gias g + join {{ ref('stg_ofsted_fe_providers') }} p + on p.provider_urn = g.urn + or (g.ukprn is not null and p.provider_ukprn = g.ukprn) + order by g.urn, (p.provider_urn = g.urn) desc, p.graded_inspection_date desc nulls last +), + +report_cards as ( + select distinct on (g.urn) + g.urn, + c.rc_inspection_date, + c.rc_safeguarding_met + from gias g + join {{ ref('stg_ofsted_fe_inspections') }} c + on c.provider_urn = g.urn + or (g.ukprn is not null and c.provider_ukprn = g.ukprn) + order by g.urn, (c.provider_urn = g.urn) desc, c.rc_inspection_date desc, c.inspection_number desc +), + +combined as ( + select + coalesce(p.urn, c.urn) as urn, + p.graded_inspection_date, + p.overall_effectiveness, + p.short_inspection_date, + c.rc_inspection_date as card_date, + c.rc_safeguarding_met as card_safeguarding_met + from providers p + full outer join report_cards c on c.urn = p.urn +), + +decided as ( + select + *, + card_date is not null + and card_date >= coalesce(greatest(graded_inspection_date, short_inspection_date), card_date) + as report_card_latest + from combined +) + +select + urn, + case when report_card_latest then card_date + else greatest(graded_inspection_date, short_inspection_date) end as latest_visit_date, + case when report_card_latest then 'report_card' + when short_inspection_date > graded_inspection_date then 'short' + else 'graded' end as latest_visit_kind, + null::text as latest_visit_outcome, + case when not report_card_latest then overall_effectiveness end as current_grade, + case when not report_card_latest then graded_inspection_date end as current_grade_date, + case when not report_card_latest then 'graded' end as current_grade_basis, + graded_inspection_date, + null::date as ungraded_inspection_date, + coalesce(graded_inspection_date, card_date) as inspection_date, + null::text as inspection_type, + 'Further education and skills'::text as framework, + overall_effectiveness, + null::integer as quality_of_education, + null::integer as behaviour_attitudes, + null::integer as personal_development, + null::integer as leadership_management, + null::integer as early_years_provision, + null::integer as sixth_form_provision, + null::text as ungraded_outcome, + null::integer as ungraded_grade, + case when report_card_latest then card_safeguarding_met end as rc_safeguarding_met, + null::integer as rc_inclusion, + null::integer as rc_curriculum_teaching, + null::integer as rc_achievement, + null::integer as rc_attendance_behaviour, + null::integer as rc_personal_development, + null::integer as rc_leadership_governance, + null::integer as rc_early_years, + null::integer as rc_sixth_form, + case when report_card_latest then card_date end as rc_inspection_date, + null::text as report_url +from decided +where report_card_latest or overall_effectiveness between 1 and 4 +``` + +- [ ] **Step 4: Run the unit tests and build** + +Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_ukprn stg_ofsted_fe_providers stg_ofsted_fe_inspections int_ofsted_fe_latest` +Expected: exit 0; all 8 `int_ofsted_fe_latest` unit tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql pipeline/transform/models/intermediate/int_ofsted_fe_latest.yml +git commit -m "feat(dbt): further education status per URN under the C1 rule (H4)" +``` + +### Task 6: Fold FE into `int_ofsted_latest` / `fact_ofsted_latest` + +**Files:** +- Modify: `pipeline/transform/models/intermediate/int_ofsted_latest.sql`, `int_ofsted_latest.yml` +- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (`fact_ofsted_latest.latest_visit_kind` accepted values) +- Create: `pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql` + +**Interfaces:** +- Consumes: `int_ofsted_fe_latest` (Task 5; same columns and order as `int_ofsted_latest`). +- Produces: `int_ofsted_latest` / `fact_ofsted_latest` rows with `framework = 'Further education and skills'` for URNs that have no school-framework record; `latest_visit_kind` may be `short`. + +- [ ] **Step 1: Write the failing unit tests, and give the existing ones the new input** + +In `int_ofsted_latest.yml`, under `given:` of **each** of the 10 existing tests, after its `stg_ofsted_inspections` input, add: + +```yaml + - input: ref('int_ofsted_fe_latest') + rows: [] +``` + +Append: + +```yaml + - name: fe_status_fills_a_school_with_no_school_record + description: Queen Mary's College (145175) has no school-framework record; its FE status passes through (audit H4). + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: [] + - input: ref('int_ofsted_fe_latest') + rows: + - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'} + expect: + rows: + - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'} + + - name: a_school_record_wins_over_an_fe_record + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 1, graded_inspection_date: '2019-06-01', overall_effectiveness: 2, framework: 'Schools - S5'} + - input: ref('int_ofsted_fe_latest') + rows: + - {urn: 1, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, rc_inspection_date: '2026-05-12', framework: 'Further education and skills'} + expect: + rows: + - {urn: 1, latest_visit_date: '2019-06-01', latest_visit_kind: graded, current_grade: 2, framework: 'Schools - S5'} +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_latest,test_type:unit"` +Expected: non-zero exit — `int_ofsted_latest` does not reference `int_ofsted_fe_latest`, so the given input is rejected (or, if dbt accepts unused inputs, `fe_status_fills_a_school_with_no_school_record` FAILS with no rows). + +- [ ] **Step 3: Union the FE rows** + +In `int_ofsted_latest.sql`, add to the header comment: + +```sql +-- Schools with no school-framework record take their further education and +-- skills status from int_ofsted_fe_latest (same columns, same order; audit H4). +``` + +Replace the final `select … from graded` with the same select wrapped in a CTE, followed by the union: + +```sql +school_status as ( + select + urn, + -- (… the existing select list, unchanged, through report_url …) + report_url + from graded +), + +fe_status as ( + -- Same columns in the same order as school_status. + select f.* + from {{ ref('int_ofsted_fe_latest') }} f + where not exists (select 1 from school_status s where s.urn = f.urn) +) + +select * from school_status +union all +select * from fe_status +``` + +(The `school_status` select list is the existing final select list verbatim; only `select` becomes `school_status as (select` and a `)` closes it. Remember the comma after the `graded` CTE's closing parenthesis.) + +In `_marts_schema.yml`, `fact_ofsted_latest` → `latest_visit_kind` → `accepted_values` → `values: ['report_card', 'graded', 'ungraded', 'short']`. + +`pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql`: + +```sql +-- A school-framework record always wins over a further education one for the +-- same URN (audit H4): no FE row may stand where Ofsted's school MI has one. + +select f.urn +from {{ ref('fact_ofsted_latest') }} f +join {{ ref('stg_ofsted_inspections') }} s on s.urn = f.urn +where f.framework = 'Further education and skills' +``` + +- [ ] **Step 4: Run the unit tests and build** + +Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select int_ofsted_fe_latest int_ofsted_latest fact_ofsted_latest assert_ofsted_fe_never_overrides_school assert_ofsted_current_grade_consistent` +Expected: exit 0; 12 `int_ofsted_latest` unit tests PASS; schema and data tests PASS on empty tables. + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/intermediate/int_ofsted_latest.sql pipeline/transform/models/intermediate/int_ofsted_latest.yml pipeline/transform/models/marts/_marts_schema.yml pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql +git commit -m "feat(dbt): further education status where a school has no school record (H4)" +``` + +### Task 7: `int_ofsted_listed`, `dim_school` columns, DAG selectors + +**Files:** +- Create: `pipeline/transform/models/intermediate/int_ofsted_listed.sql`, `int_ofsted_listed.yml` +- Modify: `pipeline/transform/models/marts/dim_school.sql`, `dim_school.yml` +- Modify: `pipeline/dags/school_data_pipeline.py` (monthly Ofsted selection), `pipeline/tests/test_dag_selectors.py` + +**Interfaces:** +- Consumes: raw `ofsted_inspections`, `stg_gias_ukprn`, `stg_ofsted_fe_providers`, `stg_gias_establishments.inspectorate_name`. +- Produces: `int_ofsted_listed(urn integer)`; `dim_school.inspectorate text`, `dim_school.ofsted_listed boolean` (null when `int_ofsted_listed` does not exist yet). + +- [ ] **Step 1: Write the unit tests** + +`pipeline/transform/models/intermediate/int_ofsted_listed.yml`: + +```yaml +version: 2 + +unit_tests: + - name: int_ofsted_listed_includes_uninspected_schools_and_fe_providers + description: > + Abbas and Templecombe (150031) is in Ofsted's school MI with no inspection; + Exeter and North Devon Colleges Group (130645) is in the FE MI with none; + a UKPRN-only FE match counts too. A school in neither is absent (audit H4). + model: int_ofsted_listed + given: + - input: source('raw', 'ofsted_inspections') + rows: + - {urn: '150031', inspection_date: 'NULL'} + - {urn: '102408', inspection_date: '17/06/2025'} + - input: ref('stg_gias_ukprn') + rows: + - {urn: 130645, ukprn: null} + - {urn: 5, ukprn: '10099999'} + - {urn: 133058, ukprn: null} + - input: ref('stg_ofsted_fe_providers') + rows: + - {provider_urn: 130645, provider_ukprn: null} + - {provider_urn: 9999999, provider_ukprn: '10099999'} + expect: + rows: + - {urn: 150031} + - {urn: 102408} + - {urn: 130645} + - {urn: 5} +``` + +In `dim_school.yml`, under `given:` of both existing tests add: + +```yaml + - input: ref('int_ofsted_listed') + rows: [] +``` + +and append: + +```yaml + - name: dim_school_carries_inspectorate_and_ofsted_listing + description: > + Audit H4. GIAS names ISI for Abbey College Cambridge; Ofsted lists Abbas + and Templecombe without an inspection; a third school is in neither. + model: dim_school + given: + - input: ref('stg_gias_establishments') + rows: + - {urn: 136083, school_name: 'Abbey College Cambridge', status_code: 1, school_type_code: 11, local_authority_code: 873, phase_code: 4, inspectorate_name: 'ISI'} + - {urn: 150031, school_name: 'Abbas and Templecombe Church School', status_code: 1, school_type_code: 28, local_authority_code: 933, phase_code: 2, inspectorate_name: 'Not applicable'} + - {urn: 1, school_name: 'No data school', status_code: 1, school_type_code: 1, local_authority_code: 312, phase_code: 2, inspectorate_name: null} + - input: ref('int_ofsted_latest') + rows: [] + - input: ref('int_ofsted_listed') + rows: + - {urn: 150031} + expect: + rows: + - {urn: 136083, inspectorate: 'ISI', ofsted_listed: false} + - {urn: 150031, inspectorate: 'Not applicable', ofsted_listed: true} + - {urn: 1, inspectorate: null, ofsted_listed: false} +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_listed dim_school,test_type:unit"` +Expected: non-zero exit (model `int_ofsted_listed` missing; `dim_school` has no such columns). + +- [ ] **Step 3: Write `int_ofsted_listed` and extend `dim_school`** + +`pipeline/transform/models/intermediate/int_ofsted_listed.sql`: + +```sql +-- Intermediate model: every URN Ofsted's own data lists, inspected or not +-- (audit H4). Only these may be called "Not yet inspected". Reads the raw +-- school MI because stg_ofsted_inspections drops rows with no inspection date. + +with school_mi as ( + select distinct cast(urn as integer) as urn + from {{ source('raw', 'ofsted_inspections') }} + where urn is not null +), + +fe as ( + select distinct g.urn + from {{ ref('stg_gias_ukprn') }} g + join {{ ref('stg_ofsted_fe_providers') }} p + on p.provider_urn = g.urn + or (g.ukprn is not null and p.provider_ukprn = g.ukprn) +) + +select urn from school_mi +union +select urn from fe +``` + +In `dim_school.sql`: + +1. Below the existing `-- depends_on: {{ ref('int_ofsted_latest') }}` line add `-- depends_on: {{ ref('int_ofsted_listed') }}` and extend the comment above it: `int_ofsted_latest and int_ofsted_listed are only ref()'d inside conditional blocks below (they may not exist before the Ofsted pipeline first runs), so declare the dependencies explicitly for dbt's DAG.` +2. After the `{% set ofsted_relation = … %}` block add: + +```sql +{% set listed_relation = adapter.get_relation( + database=target.database, + schema='intermediate', + identifier='int_ofsted_listed' +) %} +``` + +3. After `s.admissions_policy_code,` add: + +```sql + -- The inspectorate GIAS names ("ISI" for most independent schools) and + -- whether Ofsted's own data lists the school, inspected or not (audit H4). + s.inspectorate_name as inspectorate, + {% if listed_relation is not none %} + l.urn is not null as ofsted_listed, + {% else %} + null::boolean as ofsted_listed, + {% endif %} +``` + +4. After the `{% if ofsted_relation is not none %} left join … {% endif %}` block add: + +```sql +{% if listed_relation is not none %} +left join {{ ref('int_ofsted_listed') }} l on s.urn = l.urn +{% endif %} +``` + +In `pipeline/dags/school_data_pipeline.py`, in the monthly Ofsted build change `--select stg_ofsted_inspections+ int_ofsted_latest+` to `--select stg_ofsted_inspections+ stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ int_ofsted_latest+`. + +In `pipeline/tests/test_dag_selectors.py`: + +```python +# dim_school reads int_ofsted_latest and int_ofsted_listed only when the +# relations exist (adapter.get_relation), so a missing table is not a failure. +OPTIONAL_PARENTS = {'int_ofsted_latest', 'int_ofsted_listed'} +``` + +- [ ] **Step 4: Run the tests and build** + +Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select int_ofsted_listed dim_school` +Expected: exit 0; the `int_ofsted_listed` unit test and 3 `dim_school` unit tests PASS. +Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests -q` → all pass. + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/intermediate/int_ofsted_listed.sql pipeline/transform/models/intermediate/int_ofsted_listed.yml pipeline/transform/models/marts/dim_school.sql pipeline/transform/models/marts/dim_school.yml pipeline/dags/school_data_pipeline.py pipeline/tests/test_dag_selectors.py +git commit -m "feat(dbt): dim_school.inspectorate and ofsted_listed (H4)" +``` + +### Task 8: Load the real data end to end and verify PR 1 + +**Files:** none in the repo. + +- [ ] **Step 1: Load GIAS and Ofsted through the changed taps** + +```bash +mkdir -p "$SCRATCH/e2e_h4" && cd "$SCRATCH/e2e_h4" +echo "drop table if exists raw.gias_establishments, raw.gias_links, raw.ofsted_inspections, raw.ofsted_fe_providers, raw.ofsted_fe_inspections cascade;" | "$SCRATCH/dbt_local/sql.sh" +SOCK=$(grep -o 'PG_HOST=[^ ]*' "$SCRATCH/dbt_local/dbt.sh" | cut -d= -f2) +echo "{\"sqlalchemy_url\": \"postgresql+psycopg://postgres@/school_compare?host=$SOCK\", \"default_target_schema\": \"raw\"}" > target.json +echo '{}' > tap.json +cat > run_load.sh <<'EOF' +#!/bin/bash +R=/Users/tudor/projects/school_compare/pipeline/plugins/extractors +for tap in gias ofsted; do + uv run --no-project --with-editable "$R/tap-uk-$tap" tap-uk-$tap --config tap.json 2> tap-$tap.log \ + | uv run --no-project --with meltanolabs-target-postgres target-postgres --config target.json > target-$tap.log 2>&1 + echo "$tap exit ${PIPESTATUS[*]}" +done +EOF +bash run_load.sh +``` + +Expected: `gias exit 0 0` and `ofsted exit 0 0`. + +- [ ] **Step 2: Build and test everything this PR touches** + +Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_gias_ukprn stg_ofsted_inspections stg_ofsted_fe_providers stg_ofsted_fe_inspections int_ofsted_fe_latest int_ofsted_latest int_ofsted_listed fact_ofsted_latest dim_school assert_ofsted_fe_never_overrides_school assert_ofsted_current_grade_consistent` +Expected: exit 0; all unit, schema and data tests PASS. + +- [ ] **Step 3: Check the values** + +Through `"$SCRATCH/dbt_local/sql.sh" <<'SQL' … SQL`: + +```sql +select urn, latest_visit_date, latest_visit_kind, current_grade, current_grade_date, rc_inspection_date, rc_safeguarding_met, framework +from marts.fact_ofsted_latest where urn in (145175, 143526, 130457) order by urn; +select count(*) from marts.fact_ofsted_latest where framework = 'Further education and skills'; +select count(*) filter (where inspectorate = 'ISI') as isi, count(*) filter (where ofsted_listed) as listed from marts.dim_school; +select d.urn, d.ofsted_listed, f.urn is not null as has_record from marts.dim_school d left join marts.fact_ofsted_latest f using (urn) where d.urn in (150031, 136083); +select count(*) as fe_rows_for_listed_schools from marts.fact_ofsted_latest f join marts.dim_school d using (urn) where f.framework = 'Further education and skills'; +``` + +Expected: 145175 graded 2 on 2022-12-06; 143526 report card 2026-05-12 with safeguarding true; 130457 graded 2 on 2024-01-30; about 490 FE rows for listed schools (443 + 45 + 2 as at 31 August 2026); about 1,341 ISI schools; 150031 listed with no record (unless Ofsted has inspected it since); 136083 not listed, no record. Record any drift from these numbers and its cause (newer MI files) in the PR description. + +- [ ] **Step 4: Run CI's Python suite** + +Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests pipeline/tests scripts/ci/tests -q` → all pass. + +### Task 9: Open PR 1 + +- [ ] **Step 1:** `git status --short` → only `.DS_Store`. +- [ ] **Step 2:** `git push -u origin fix/ofsted-inspectorates-pipeline`. +- [ ] **Step 3:** Open the PR through the Gitea API with basic auth from `git credential fill` (host `privaterepo.sitaru.org`; never print the secret): `POST /api/v1/repos/tudor/school_compare/pulls`, head `fix/ofsted-inspectorates-pipeline`, base `main`, title `fix(pipeline): inspectorate, Ofsted listing and further-education status (H4, part 1 of 2)`. Body: what changed; Task 8's numbers; **After merge:** run the daily DAG (GIAS fields) and then `school_data_monthly_ofsted` on staging — the monthly build reads `raw.gias_establishments."UKPRN"`, which exists only after a daily run with this PR; then merge PR 2; production the same order before PR 2 is promoted; ends with the Claude Code line. +- [ ] **Step 4:** Report the PR URL. Do not merge. + +--- + +## PR 2 — site + +After PR 1 merges: `git fetch origin && git switch -c fix/ofsted-inspectorates-site origin/main`. + +### Task 10: Backend payloads, optional columns, Ofsted page link + +**Files:** +- Modify: `backend/schemas.py` (`SCHOOL_COLUMNS`), `backend/data_loader.py` (`_MAIN_QUERY`, `_EXTRA_OPTIONAL_COLUMNS`, the fallback loop), `backend/app.py` (detail and compare `school_info`), `backend/ofsted_codes.py` (`ofsted_page_url`) +- Test: create `backend/tests/test_inspection_status_payload.py`; modify `backend/tests/test_ofsted_codes.py:45`, `backend/tests/test_supplementary_enrichment.py:36` + +**Interfaces:** +- Produces: list rows, detail `school_info` and compare `school_info` carry `inspectorate: str | null` and `ofsted_listed: bool | null`; `ofsted_page_url(urn)` returns `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}`. + +- [ ] **Step 1: Write the failing tests** + +`backend/tests/test_inspection_status_payload.py`: + +```python +"""Audit H4: every payload that feeds an Ofsted badge carries the inspectorate +GIAS names and whether Ofsted's own data lists the school, so the site never +says "Not yet inspected" on the strength of missing data.""" + +import pandas as pd +import pytest +from fastapi.testclient import TestClient + + +def _df(): + base = dict(year=202425, local_authority='Cambridgeshire', local_authority_code=873, + postcode='CB1 1AA', latitude=52.2, longitude=0.12, address='1 High St', + status='Open', town='Cambridge') + return pd.DataFrame([ + dict(base, urn=136083, school_name='Abbey College Cambridge', phase='Secondary', + school_type='Other independent school', inspectorate='ISI', ofsted_listed=False), + dict(base, urn=150031, school_name='Abbas and Templecombe Church School', phase='Primary', + school_type='Academy converter', inspectorate='Not applicable', ofsted_listed=True), + ]) + + +@pytest.fixture() +def client(monkeypatch): + from backend import app as app_module + from backend import database as database_module + + def _no_db(): + raise RuntimeError('no database in this test') + + monkeypatch.setattr(app_module, 'load_latest_school_data', _df) + monkeypatch.setattr(app_module, 'load_school_data', _df) + monkeypatch.setattr(app_module, 'get_supplementary_data', lambda db, urn: {}) + monkeypatch.setattr(app_module, '_place_registry', None) + monkeypatch.setattr(database_module, 'SessionLocal', _no_db) + return TestClient(app_module.app, raise_server_exceptions=False) + + +def test_the_list_carries_inspectorate_and_listing(client): + resp = client.get('/api/schools?page_size=50') + assert resp.status_code == 200, resp.text + by_urn = {s['urn']: s for s in resp.json()['schools']} + assert by_urn[136083]['inspectorate'] == 'ISI' + assert by_urn[150031]['ofsted_listed'] is True + + +def test_the_school_page_carries_them(client): + info = client.get('/api/schools/136083').json()['school_info'] + assert info['inspectorate'] == 'ISI' + assert info['ofsted_listed'] is False + + +def test_compare_carries_them(client): + body = client.get('/api/compare?urns=136083,150031').json() + assert body['comparison']['150031']['school_info']['ofsted_listed'] is True + assert body['comparison']['136083']['school_info']['inspectorate'] == 'ISI' + + +def test_a_mart_built_before_the_columns_still_loads(): + # Missing columns make the main query retry with NULLs (data_loader). + from backend import data_loader + assert {'inspectorate', 'ofsted_listed'} <= set(data_loader._EXTRA_OPTIONAL_COLUMNS) + fallback = str(data_loader._MAIN_QUERY_NO_EXTRA_COLS) + assert 'NULL AS inspectorate,' in fallback and 'NULL AS ofsted_listed,' in fallback +``` + +(`/api/compare` returns `{"comparison": {: {"school_info": …}}}`.) Change `backend/tests/test_ofsted_codes.py:45` and `backend/tests/test_supplementary_enrichment.py:36` to expect `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/` (keep each test's URN). + +- [ ] **Step 2: Run them to see them fail** + +Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests/test_inspection_status_payload.py backend/tests/test_ofsted_codes.py backend/tests/test_supplementary_enrichment.py -q` +Expected: the four new tests fail (`KeyError: 'inspectorate'`; the optional-columns assertion); the two link tests fail on the old `provider/21` URL. + +- [ ] **Step 3: Implement** + +- `backend/schemas.py`, `SCHOOL_COLUMNS`: after `"ofsted_rc_date",` add `"inspectorate",` and `"ofsted_listed",`. +- `backend/data_loader.py`, `_MAIN_QUERY`: after ` s.nursery_provision,` add ` s.inspectorate,` and ` s.ofsted_listed,`. Add `"inspectorate",` and `"ofsted_listed",` to `_EXTRA_OPTIONAL_COLUMNS`, and `("inspectorate", "s"),` and `("ofsted_listed", "s"),` to the tuple of `(column, alias)` pairs the fallback loop replaces. Extend the comment above `_EXTRA_OPTIONAL_COLUMNS`: `… parliamentary constituency, and the inspectorate / Ofsted-listing columns added for audit H4) …`. +- `backend/app.py`, detail `school_info` (after `"nursery_provision": latest.get("nursery_provision"),`): + +```python + # What the Ofsted badge may say when there is no inspection record + # (audit H4): the inspectorate GIAS names, and whether Ofsted's own + # data lists the school at all. + "inspectorate": latest.get("inspectorate"), + "ofsted_listed": latest.get("ofsted_listed"), +``` + +- `backend/app.py`, compare `school_info` (after `"has_sixth_form": convert_to_native(latest.get("has_sixth_form")),`): + +```python + # Audit H4: what compare may say when there is no inspection record. + "inspectorate": convert_to_native(latest.get("inspectorate")), + "ofsted_listed": convert_to_native(latest.get("ofsted_listed")), +``` + +- `backend/ofsted_codes.py`: + +```python +def ofsted_page_url(urn: int) -> str: + """The school's page on ofsted.gov.uk (all its reports live there — we + never deep-link an individual report). Ofsted's provider pages sit under a + category (21 primary, 23 secondary, 25 special, 40 FE colleges, …); this + address redirects to the right one, where provider/21 404s for every + non-primary school.""" + return f"https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}" +``` + +- [ ] **Step 4: Run the tests and the backend suite** + +Run: the Step 2 command → all pass. Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests -q` → all pass. + +- [ ] **Step 5: Commit** + +```bash +git add backend/schemas.py backend/data_loader.py backend/app.py backend/ofsted_codes.py backend/tests/test_inspection_status_payload.py backend/tests/test_ofsted_codes.py backend/tests/test_supplementary_enrichment.py +git commit -m "fix(api): inspectorate and Ofsted listing in payloads; Ofsted links that resolve (H4)" +``` + +### Task 11: The badge rule and its callers + +**Files:** +- Modify: `nextjs-app/lib/types.ts` (`School`, `OfstedLatestVisit`), `nextjs-app/lib/ofstedStatus.ts`, `nextjs-app/lib/utils.ts` (`buildOfstedListBadge`) +- Modify: `nextjs-app/components/SchoolRow.tsx`, `SecondarySchoolRow.tsx`, `LeafletMapInner.tsx` +- Test: `nextjs-app/__tests__/lib/utils.test.ts`, `__tests__/lib/ofstedStatus.test.ts`, `__tests__/components/SchoolRow.test.tsx` + +**Interfaces:** +- Produces: `School.inspectorate?: string | null`, `School.ofsted_listed?: boolean | null`; `OfstedLatestVisit.kind` adds `'short'`; `FE_FRAMEWORK = 'Further education and skills'`; `hasReportCard(ofsted: Pick): boolean`; `buildOfstedListBadge(school): OfstedListBadge | null`. + +- [ ] **Step 1: Write the failing tests** + +In `__tests__/lib/utils.test.ts`, inside `describe('buildOfstedListBadge', …)`: replace the test `returns pending badge when no grade and no inspection on record` with the block below, and in every other test in that describe use `buildOfstedListBadge(…)!` (the function may now return null): + +```ts + it('says "Not yet inspected" only when Ofsted lists the school uninspected (audit H4)', () => { + const badge = buildOfstedListBadge({ ofsted_grade: null, ofsted_date: null, ofsted_listed: true }); + expect(badge).toEqual({ label: 'Not yet inspected', cssClass: 'ofstedPending' }); + }); + + it('names ISI for a school GIAS says ISI inspects', () => { + expect(buildOfstedListBadge({ ofsted_date: null, inspectorate: 'ISI', ofsted_listed: false })) + .toEqual({ label: 'Inspected by ISI', cssClass: 'ofstedPending' }); + }); + + it('an Ofsted record wins over ISI', () => { + expect(buildOfstedListBadge({ ofsted_grade: 2, ofsted_grade_date: '2022-03-01', inspectorate: 'ISI' })!.label) + .toBe('Good · 2022'); + }); + + it('gives no badge without any inspection data', () => { + expect(buildOfstedListBadge({ ofsted_grade: null, ofsted_date: null, ofsted_framework: null })).toBeNull(); + expect(buildOfstedListBadge({ ofsted_listed: null, inspectorate: 'Not applicable' })).toBeNull(); + }); +``` + +In `__tests__/lib/ofstedStatus.test.ts` add: + +```ts +it('names a short inspection as Ofsted does', () => { + expect(latestVisitLine({ date: '2023-02-01', kind: 'short', outcome: null })).toBe('Short inspection, 1 February 2023'); +}); + +it('treats a report card with only a safeguarding result as a report card (FE)', () => { + expect(hasReportCard({ rc_inspection_date: '2026-05-12', report_card: {} })).toBe(true); + expect(hasReportCard({ rc_inspection_date: null, report_card: {} })).toBe(false); +}); +``` + +(change the file's first line to `import { gradeSourceLine, hasReportCard, latestVisitLine, showLatestVisitLine } from '@/lib/ofstedStatus';`). + +In `__tests__/components/SchoolRow.test.tsx` add (the file's school fixture is `base`): + +```tsx +it('renders no Ofsted badge for a school with no inspection data (audit H4)', () => { + const { container } = render(); + expect(container.textContent).not.toMatch(/Not yet inspected|Inspected/); +}); +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `cd nextjs-app && npx jest __tests__/lib/utils.test.ts __tests__/lib/ofstedStatus.test.ts __tests__/components/SchoolRow.test.tsx` +Expected: the new tests fail (no ISI rule; `hasReportCard` missing; "Not yet inspected" for no data). + +- [ ] **Step 3: Implement** + +`lib/types.ts`: in `School`, after `ofsted_framework?: string | null;` add + +```ts + /** The inspectorate GIAS names ("ISI", "Ofsted", "Not applicable"). Audit H4. */ + inspectorate?: string | null; + /** Whether Ofsted's own data lists the school, inspected or not. Audit H4. */ + ofsted_listed?: boolean | null; +``` + +and in `OfstedLatestVisit` change `kind` to `'report_card' | 'graded' | 'ungraded' | 'short';`. + +`lib/ofstedStatus.ts`: add `short: 'Short inspection',` to `VISIT_KIND`, and + +```ts +/** fact_ofsted_latest.framework for Ofsted's further education and skills records (audit H4). */ +export const FE_FRAMEWORK = 'Further education and skills'; + +/** + * A report card is in force. Checked by its date as well as its area labels: + * an FE report card carries a safeguarding result and no school areas. + */ +export function hasReportCard( + ofsted: Pick, +): boolean { + return !!ofsted.rc_inspection_date || Object.keys(ofsted.report_card ?? {}).length > 0; +} +``` + +`lib/utils.ts`, `buildOfstedListBadge`: add `inspectorate?: string | null; ofsted_listed?: boolean | null;` to the parameter type, change the return type to `OfstedListBadge | null`, replace the last bullet of the doc comment with + +```ts + * - No inspection on record (audit H4): "Inspected by ISI" when GIAS names + * ISI; "Not yet inspected" only when Ofsted's own data lists the school; + * otherwise no badge, never a claim the data cannot support. +``` + +and replace `return { label: 'Not yet inspected', cssClass: 'ofstedPending' };` with + +```ts + if (school.inspectorate === 'ISI') return { label: 'Inspected by ISI', cssClass: 'ofstedPending' }; + if (school.ofsted_listed) return { label: 'Not yet inspected', cssClass: 'ofstedPending' }; + return null; +``` + +`components/SchoolRow.tsx` and `components/SecondarySchoolRow.tsx`: wrap the badge span in `{ofstedBadge && ( … )}`. + +`components/LeafletMapInner.tsx`, `cardHtml`: replace the tags line with + +```ts +
${badge ? `${escapeHtml(badge.label)}` : ''}${type}
+``` + +- [ ] **Step 4: Run the tests, the suite and the type check** + +Run: `npx jest __tests__/lib/utils.test.ts __tests__/lib/ofstedStatus.test.ts __tests__/components/SchoolRow.test.tsx` → pass. Run: `npm run typecheck && npm test` → all pass (fix any other caller the type check names by handling `null`, never by asserting it away). + +- [ ] **Step 5: Commit** + +```bash +git add nextjs-app/lib/types.ts nextjs-app/lib/ofstedStatus.ts nextjs-app/lib/utils.ts nextjs-app/components/SchoolRow.tsx nextjs-app/components/SecondarySchoolRow.tsx nextjs-app/components/LeafletMapInner.tsx nextjs-app/__tests__/lib/utils.test.ts nextjs-app/__tests__/lib/ofstedStatus.test.ts nextjs-app/__tests__/components/SchoolRow.test.tsx +git commit -m "fix(search): no \"Not yet inspected\" without Ofsted's word; ISI badge (H4)" +``` + +### Task 12: School page and compare + +**Files:** +- Create: `nextjs-app/components/school/InspectionNotice.tsx`; test `nextjs-app/__tests__/components/InspectionNotice.test.tsx` +- Modify: `nextjs-app/components/school/OfstedSection.tsx`, `PrimarySchoolSections.tsx`, `SecondarySchoolSections.tsx`, `nextjs-app/lib/compareLogic.ts`, `nextjs-app/components/compare/CompareOfsted.tsx`, `CompareAtAGlance.tsx` +- Test: `nextjs-app/__tests__/components/OfstedSection.test.tsx`, `__tests__/components/CompareOfsted.test.tsx`, `__tests__/lib/compareLogic.test.ts` + +**Interfaces:** +- Consumes: `FE_FRAMEWORK`, `hasReportCard`, `School.inspectorate`, `School.ofsted_listed` (Task 11). +- Produces: `InspectionNotice({ school }: { school: Pick })` — renders the ISI or "Not yet inspected" section, or nothing; `noInspectionLabel(school): string` in `lib/compareLogic.ts`. + +- [ ] **Step 1: Write the failing tests** + +`__tests__/components/InspectionNotice.test.tsx`: + +```tsx +import { render, screen } from '@testing-library/react'; +import { InspectionNotice } from '@/components/school/InspectionNotice'; + +it('names ISI and links its reports for an ISI school (audit H4)', () => { + render(); + expect(screen.getByText('Inspected by the Independent Schools Inspectorate (ISI)')).toBeInTheDocument(); + expect(screen.getByRole('link', { name: /ISI reports/ })).toHaveAttribute('href', 'https://www.isi.net/reports/'); +}); + +it('says "Not yet inspected" for a school Ofsted lists without an inspection', () => { + render(); + expect(screen.getByText('Not yet inspected')).toBeInTheDocument(); + expect(screen.getByRole('link', { name: /Ofsted reports/ })).toHaveAttribute( + 'href', 'https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/150031'); +}); + +it('renders nothing without inspection data, including an unknown listing', () => { + const { container } = render(); + expect(container).toBeEmptyDOMElement(); +}); +``` + +In `__tests__/components/OfstedSection.test.tsx` add (uses the file's `renderSection`): + +```tsx +it('labels a further education record and prints its short inspection (audit H4)', () => { + renderSection({ + framework: 'Further education and skills', + current_grade: { grade: 2, date: '2019-03-05', basis: 'graded' }, + latest_visit: { date: '2023-02-01', kind: 'short', outcome: null }, + }); + expect(screen.getByText('Further education and skills')).toBeInTheDocument(); + expect(screen.getByText('Good')).toBeInTheDocument(); + expect(screen.getByText('Latest visit: Short inspection, 1 February 2023')).toBeInTheDocument(); +}); +``` + +In `__tests__/lib/compareLogic.test.ts` add: + +```ts +it('shows an FE report card with only a safeguarding result as a report card (audit H4)', () => { + const display = ofstedDisplay(ofsted({ rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, report_card: {} })); + expect(display.kind).toBe('report_card'); +}); + +it('words the no-record compare cell from the evidence', () => { + expect(noInspectionLabel({ inspectorate: 'ISI' })).toBe('Inspected by ISI'); + expect(noInspectionLabel({ ofsted_listed: true })).toBe('Not yet inspected'); + expect(noInspectionLabel({ ofsted_listed: null })).toBe('No inspection in our dataset'); +}); +``` + +(`ofsted(partial)` is the file's existing fixture helper; add `noInspectionLabel` to its import from `@/lib/compareLogic`.) + +In `__tests__/components/CompareOfsted.test.tsx`, inside `describe('CompareOfsted', …)`: + +```tsx + it('words a school without an Ofsted record from the evidence (audit H4)', () => { + const isi = { ...school(4, 'ISI School'), inspectorate: 'ISI' } as School; + const listed = { ...school(5, 'New Academy'), ofsted_listed: true } as School; + render(); + expect(screen.getAllByText('Inspected by ISI').length).toBeGreaterThan(0); + expect(screen.getAllByText('Not yet inspected').length).toBeGreaterThan(0); + expect(screen.queryByText(/No inspection outcome in our dataset/)).toBeNull(); + }); +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `npx jest __tests__/components/InspectionNotice.test.tsx __tests__/components/OfstedSection.test.tsx __tests__/lib/compareLogic.test.ts __tests__/components/CompareOfsted.test.tsx` +Expected: the new tests fail (component and helper missing; no FE label; FE report card read as `no_overall_grade`). + +- [ ] **Step 3: Implement** + +`components/school/InspectionNotice.tsx`: + +```tsx +/** + * InspectionNotice — the school page's inspection section when there is no + * Ofsted inspection record (audit H4): the inspectorate GIAS names (ISI), or + * Ofsted's own "not yet inspected" when its data lists the school. Nothing + * otherwise: no claim the data cannot support. Server component. + */ + +import type { School } from '@/lib/types'; +import { Section, sectionStyles as styles } from './sectionShared'; + +const OFSTED_REPORTS = 'https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/'; +const ISI_REPORTS = 'https://www.isi.net/reports/'; + +export function InspectionNotice({ school }: { school: Pick }) { + if (school.inspectorate === 'ISI') { + return ( +
+

+ Inspection + + ISI reports ↗ + +

+

Inspected by the Independent Schools Inspectorate (ISI)

+
+ ); + } + if (school.ofsted_listed) { + return ( +
+

+ Ofsted Rating + + Ofsted reports ↗ + +

+
+ Not yet inspected +
+
+ ); + } + return null; +} +``` + +`components/school/OfstedSection.tsx`: import `FE_FRAMEWORK` from `@/lib/ofstedStatus`; directly after the closing `` add + +```tsx + {ofsted.framework === FE_FRAMEWORK && ( +

{FE_FRAMEWORK}

+ )} +``` + +`PrimarySchoolSections.tsx` and `SecondarySchoolSections.tsx`: import `hasReportCard` from `@/lib/ofstedStatus` and `InspectionNotice` from `./InspectionNotice`; set `const isReportCard = !!ofsted && hasReportCard(ofsted);`; change `{ofsted && ( )}` to `{ofsted ? ( ) : }`. + +`lib/compareLogic.ts`: import `hasReportCard` from `./ofstedStatus`; in `ofstedDisplay` replace the report-card condition with `if (hasReportCard(ofsted)) {`; add + +```ts +/** + * The compare cell when a school has no Ofsted record (audit H4): ISI per GIAS, + * Ofsted's own "not yet inspected", or plainly that we hold nothing. + */ +export function noInspectionLabel(school: Pick): string { + if (school.inspectorate === 'ISI') return 'Inspected by ISI'; + if (school.ofsted_listed) return 'Not yet inspected'; + return 'No inspection in our dataset'; +} +``` + +(import `School` from `./types` if not already imported). + +`components/compare/CompareOfsted.tsx`: give `ResultCell` a `school` prop (`school: School`) and return `{noInspectionLabel(school)}` for `display.kind === 'none'`; pass `school={data[String(school.urn)]?.school_info ?? school}` where it is rendered; in the judgement-detail cell replace `No inspection in our dataset` with `{noInspectionLabel(data[String(school.urn)]?.school_info ?? school)}`; change the `Ofsted page` fallback URL to `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/${school.urn}`. + +`components/compare/CompareAtAGlance.tsx`: replace `No inspection in our dataset` with `{noInspectionLabel(data[String(school.urn)]?.school_info ?? school)}`. + +- [ ] **Step 4: Run the tests, the suite and the type check** + +Run: the Step 2 command → pass. Run: `npm run typecheck && npm test` → all pass. + +- [ ] **Step 5: Commit** + +```bash +git add nextjs-app/components/school/InspectionNotice.tsx nextjs-app/components/school/OfstedSection.tsx nextjs-app/components/school/PrimarySchoolSections.tsx nextjs-app/components/school/SecondarySchoolSections.tsx nextjs-app/lib/compareLogic.ts nextjs-app/components/compare/CompareOfsted.tsx nextjs-app/components/compare/CompareAtAGlance.tsx nextjs-app/__tests__/components/InspectionNotice.test.tsx nextjs-app/__tests__/components/OfstedSection.test.tsx nextjs-app/__tests__/lib/compareLogic.test.ts nextjs-app/__tests__/components/CompareOfsted.test.tsx +git commit -m "fix(school, compare): ISI, not-yet-inspected and further-education states (H4)" +``` + +### Task 13: E2E journeys + +**Files:** +- Modify: `e2e/tests/journeys.spec.ts` (after the Ofsted journeys, `test('a grade confirmed at an ungraded visit says so and is dated by it', …)`) + +- [ ] **Step 1: Add the journeys** + +```ts +test('a further education college shows its Ofsted result, not "Not yet inspected"', async ({ page }) => { + // Queen Mary's College (145175) is inspected under Ofsted's further education + // framework, which the site did not load: its badge said "Not yet inspected" + // (audit H4). Its result may change; the label must never be absent or wrong. + const res = await page.request.get('/api/schools/145175'); + expect(res.ok()).toBeTruthy(); + await page.goto('/school/145175'); + const ofsted = page.locator('#ofsted'); + await expect(ofsted).toBeVisible(); + await expect(ofsted.getByText('Further education and skills')).toBeVisible(); + await expect(ofsted).not.toContainText('Not yet inspected'); +}); + +test('an ISI-inspected school says so instead of "Not yet inspected"', async ({ page }) => { + const res = await page.request.get('/api/schools?school_type=independent&phase=secondary&page_size=100'); + expect(res.ok()).toBeTruthy(); + const school = ((await res.json()).schools ?? []).find( + (s: { inspectorate?: string | null; ofsted_date?: string | null; ofsted_rc_date?: string | null }) => + s.inspectorate === 'ISI' && !s.ofsted_date && !s.ofsted_rc_date); + test.skip(!school, 'no ISI school without an Ofsted record here'); + await page.goto(`/school/${school.urn}`); + await expect(page.getByText('Inspected by the Independent Schools Inspectorate (ISI)')).toBeVisible(); + await expect(page.locator('#ofsted')).not.toContainText('Not yet inspected'); +}); + +test('"Not yet inspected" appears only for a school Ofsted lists without an inspection', async ({ page }) => { + const res = await page.request.get('/api/schools?phase=primary&page_size=500'); + expect(res.ok()).toBeTruthy(); + type Row = { urn: number; ofsted_listed?: boolean | null; ofsted_date?: string | null; ofsted_rc_date?: string | null; ofsted_grade?: number | null }; + const schools: Row[] = (await res.json()).schools ?? []; + const unlisted = schools.find(s => !s.ofsted_listed && !s.ofsted_date && !s.ofsted_rc_date && !s.ofsted_grade); + const listed = schools.find(s => s.ofsted_listed && !s.ofsted_date && !s.ofsted_rc_date && !s.ofsted_grade); + test.skip(!listed, 'no listed, uninspected primary on this page'); + await page.goto(`/school/${listed!.urn}`); + await expect(page.locator('#ofsted').getByText('Not yet inspected')).toBeVisible(); + if (unlisted) { + await page.goto(`/school/${unlisted.urn}`); + await expect(page.getByText('Not yet inspected')).toHaveCount(0); + } +}); +``` + +- [ ] **Step 2: Check the file compiles and lists the tests** + +Run: `cd e2e && npx playwright test --list tests/journeys.spec.ts -g "further education college|ISI-inspected|Ofsted lists without"` → the three tests listed. + +- [ ] **Step 3: Run them against staging to see them fail** + +Only once PR 1 has merged and the daily and monthly Ofsted DAGs have run on staging: `cd e2e && BASE_URL=https://stx.schoolcompare.co.uk npx playwright test tests/journeys.spec.ts -g "further education college|ISI-inspected|Ofsted lists without" --reporter=line --retries=0` +Expected: all three FAIL for the right reason (staging still runs the old site: no FE label, no ISI section, the old badge rule), or skip where the old API lacks the fields; record which. If the DAGs have not run, record that and rely on the post-merge gate. + +- [ ] **Step 4: Commit** + +```bash +git add e2e/tests/journeys.spec.ts +git commit -m "test(e2e): further education, ISI and not-yet-inspected states (H4)" +``` + +### Task 14: Verify and open PR 2 + +- [ ] **Step 1:** `cd nextjs-app && npm run typecheck && npm test`; from the repo root `uv run --no-project --with-requirements requirements.txt python -c "from backend.app import app; print('backend imports OK')"` and `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests pipeline/tests scripts/ci/tests -q` → all pass. +- [ ] **Step 2:** `git status --short` → only `.DS_Store`. `git push -u origin fix/ofsted-inspectorates-site`. +- [ ] **Step 3:** Open the PR through the Gitea API (as Task 9): head `fix/ofsted-inspectorates-site`, title `fix(site): ISI, Ofsted-listed and further-education inspection states (H4, part 2 of 2)`. Body: what changed, including the Ofsted page link fix (`provider/21` 404s for every non-primary school); Task 13 Step 3's result; **merge only after the daily and monthly Ofsted DAGs have run on staging with PR 1; promote only after they have run on production** (before that, the fallback query gives no badge for the ISI and listed groups); ends with the Claude Code line. +- [ ] **Step 4:** Report the PR URL. Do not merge. + +--- + +## After both PRs are live in production (Tudor promotes) + +Validate against GIAS and Ofsted's school and FE MI: every "Not yet inspected" belongs to a school Ofsted's data lists without an inspection; every "Inspected by ISI" matches GIAS; every FE grade, date and report card matches the FE files; schools with no data show no badge; the compare page's Ofsted link resolves for a secondary and a special school. Then mark H4 resolved in the audit report artifact. diff --git a/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md b/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md index fdbe319..82c5dbe 100644 --- a/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md +++ b/docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md @@ -227,3 +227,23 @@ E2E. provider's result; the URN match is always tried first. - **"Not yet inspected" disappears briefly** if PR 2 is promoted before the DAGs run (fallback query). It never appears wrongly. + +## Amendments (implementation plan, 2026-10-13) + +- **UKPRN comes from a new view, `stg_gias_ukprn`,** on the raw GIAS source, + not from a `ukprn` column on `stg_gias_establishments`. A `ref` to + `stg_gias_establishments` from the Ofsted models would put them in the daily + build's `stg_gias_establishments+` selection, where their Ofsted parents are + not built (`pipeline/tests/test_dag_selectors.py`). The daily build selects + `stg_gias_ukprn` by name; `stg_gias_establishments` gains only + `inspectorate_name`. +- **A report card is recognised by `rc_inspection_date` as well as by its area + labels** (`hasReportCard`), on the school page and in compare. An FE report + card carries a safeguarding result and no school areas, so the label test + alone read it as "No overall grade". +- **The compare page's Ofsted link is fixed.** `ofsted_page_url` built + `https://reports.ofsted.gov.uk/provider/21/{urn}`; category 21 is primary + schools, so the link 404s for every secondary, special and FE school + (checked: 137086, 101099, 130457, 145175). It now uses + `…/inspection-reports/find-inspection-report/provider/ELS/{urn}`, which + redirects to the right category and is what the school page already uses.