# Inspection Status Beyond Ofsted's School Data Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Stop labelling schools "Not yet inspected" unless Ofsted's own data lists them uninspected, and show ISI and Ofsted further-education results where the data exists (audit H4). **Architecture:** PR 1 adds two GIAS fields, two Ofsted further-education (FE) streams, an FE status model folded into `int_ofsted_latest` / `fact_ofsted_latest` (framework "Further education and skills"), an `int_ofsted_listed` model, and `dim_school.inspectorate` / `dim_school.ofsted_listed`. PR 2, after the daily and monthly Ofsted DAGs have run on PR 1, publishes the two fields, changes the badge rule, adds the ISI / "Not yet inspected" school-page states and the FE label, updates compare, and fixes the Ofsted page link. **Tech Stack:** Singer SDK taps (Python, pandas, requests), meltanolabs target-postgres, dbt-postgres ~1.10 (YAML unit tests, singular tests), FastAPI + pandas + SQLAlchemy, Next.js / React / TypeScript, Jest + Testing Library, Playwright. **Spec:** `docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md` (read its "Amendments" section: UKPRN comes from a new `stg_gias_ukprn` view; report cards are detected by `rc_inspection_date`; `ofsted_page_url` is fixed). ## Global Constraints - Badge order (search rows and map card): report card → "Report Card · year"; grade in force → grade word · year; inspected, no grade → "Inspected · year"; GIAS inspectorate `ISI` → "Inspected by ISI" (grey, css class `ofstedPending`); `ofsted_listed` → "Not yet inspected" (grey, `ofstedPending`); otherwise no badge. - FE framework value, exactly: `Further education and skills`. - FE rule: a report card on or after the previous-framework inspection (or with none) wins and leaves no grade in force; otherwise a previous-framework grade 1–4 is the current grade, dated by that inspection's first day, basis `graded`; latest visit is the later of that inspection (`graded`) and any short inspection (`short`); grade `-` with no report card gives no FE row. - A school-framework record always wins over an FE record for the same URN. - FE providers match a URN by `Provider URN` first, then by `Provider UKPRN` = GIAS UKPRN. - The Ofsted models must never `ref` `stg_gias_establishments` (the daily build's `stg_gias_establishments+` would select them); they read UKPRN from `stg_gias_ukprn`, a view on the raw GIAS source. - Copy: Ofsted's and GIAS's own terms ("Short inspection", "Not yet inspected", "Inspected by the Independent Schools Inspectorate (ISI)", "Further education and skills"); no derived sentences. - ISI link: `https://www.isi.net/reports/`. Ofsted page link: `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}` (redirects to the right category; `provider/21/{urn}` 404s for every non-primary school). - PR 2 must not merge before the daily DAG (GIAS fields) and `school_data_monthly_ofsted` have run on staging with PR 1, nor be promoted before they have run on production. Promotion is Tudor's; never trigger it. - Never push to `main`. Commits end with the Co-Authored-By trailer the executing session's harness specifies; PR bodies end with the Claude Code line. Do not start a local app server. ## Review Focus 1. A school whose GIAS inspectorate is ISI **and** that Ofsted also lists (dual registration) must show "Inspected by ISI" only when there is no Ofsted inspection record; an Ofsted record always wins. Pinned in Task 11 (`an Ofsted record wins over ISI`). 2. An FE provider whose report card predates its previous-framework inspection (bad data) must not produce a row with both a grade and a report card (that breaks `assert_ofsted_current_grade_consistent`). Pinned in Task 5 (`int_ofsted_fe_report_card_older_than_grade_is_ignored`). 3. A UKPRN shared by an FE provider and a URN that Ofsted also lists under its own URN must take the URN match. Pinned in Task 5 (`int_ofsted_fe_prefers_the_urn_match`). 4. The FE "as at" file missing from the newest academic-year page (a new page with no file yet) must fall back to the previous page, not emit nothing. Pinned in Task 3 (`test_providers_fall_back_to_the_previous_page`). 5. A school-page or compare render for a school with `ofsted_listed` null (marts built before PR 1) must behave as "not listed": no "Not yet inspected". Pinned in Task 11 (null case) and Task 12. ## File Map **PR 1 — branch `fix/ofsted-inspectorates-pipeline`** (exists; spec `5f7528b`) - Modify `pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py` (two fields). - Create `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py` (page/file selection, CSV reading, column maps — pandas-only). - Modify `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (two streams). - Create `pipeline/tests/test_gias_declared_fields.py`, `pipeline/tests/test_ofsted_fe_mi.py`. - Modify `pipeline/transform/models/staging/stg_gias_establishments.sql`, `_stg_sources.yml`. - Create `pipeline/transform/models/staging/stg_gias_ukprn.sql`, `stg_ofsted_fe_providers.sql`, `stg_ofsted_fe_inspections.sql`, `stg_ofsted_fe.yml`. - Create `pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`, `int_ofsted_fe_latest.yml`, `int_ofsted_listed.sql`, `int_ofsted_listed.yml`. - Modify `pipeline/transform/models/intermediate/int_ofsted_latest.sql`, `int_ofsted_latest.yml`. - Modify `pipeline/transform/models/marts/dim_school.sql`, `dim_school.yml`, `_marts_schema.yml`. - Create `pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql`. - Modify `pipeline/dags/school_data_pipeline.py` (daily and monthly selectors), `pipeline/tests/test_dag_selectors.py` (`OPTIONAL_PARENTS`). **PR 2 — branch `fix/ofsted-inspectorates-site`** (from `origin/main` after PR 1 merges; touches no PR 1 file) - Modify `backend/schemas.py` (`SCHOOL_COLUMNS`), `backend/data_loader.py` (main query, optional columns), `backend/app.py` (detail and compare `school_info`), `backend/ofsted_codes.py` (`ofsted_page_url`). - Create `backend/tests/test_inspection_status_payload.py`; modify `backend/tests/test_ofsted_codes.py`, `backend/tests/test_supplementary_enrichment.py`. - Modify `nextjs-app/lib/types.ts`, `nextjs-app/lib/ofstedStatus.ts`, `nextjs-app/lib/utils.ts`, `nextjs-app/lib/compareLogic.ts`. - Modify `nextjs-app/components/SchoolRow.tsx`, `SecondarySchoolRow.tsx`, `LeafletMapInner.tsx`, `school/OfstedSection.tsx`, `school/PrimarySchoolSections.tsx`, `school/SecondarySchoolSections.tsx`, `compare/CompareOfsted.tsx`, `compare/CompareAtAGlance.tsx`. - Create `nextjs-app/components/school/InspectionNotice.tsx`. - Modify tests: `nextjs-app/__tests__/lib/utils.test.ts`, `__tests__/lib/ofstedStatus.test.ts`, `__tests__/lib/compareLogic.test.ts`, `__tests__/components/OfstedSection.test.tsx`, `__tests__/components/CompareOfsted.test.tsx`, `__tests__/components/SchoolRow.test.tsx`; create `__tests__/components/InspectionNotice.test.tsx`. - Modify `e2e/tests/journeys.spec.ts`. **Commands used throughout.** `$SCRATCH` is the session scratchpad directory. Python tests run in CI's environment: ```bash uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest -q ``` Local tap commands use `--with-editable` (a plain `--with ./path` reuses a cached build of the unchanged version and ignores edits). dbt runs through `$SCRATCH/dbt_local/dbt.sh` (Task 1). --- ## PR 1 — pipeline ### Task 1: Local dbt harness `$SCRATCH/dbt_local/pgdata` survives from earlier work; its helper scripts do not. **Files:** none in the repo. - [ ] **Step 1: Recreate the helpers** ```bash mkdir -p "$SCRATCH/dbt_local" && cd "$SCRATCH/dbt_local" cat > start_pg.py <<'EOF' import pgserver, sys srv = pgserver.get_server(sys.argv[1], cleanup_mode=None) print(srv.get_uri()) if "school_compare" not in srv.psql("select datname from pg_database;"): srv.psql("create database school_compare;") EOF uv run --no-project --with pgserver python start_pg.py "$SCRATCH/dbt_local/pgdata" ``` Expected: a URI ending `?host=`. Then, with that socket dir: ```bash cat > "$SCRATCH/dbt_local/dbt.sh" < export PG_HOST= PG_PORT=5432 PG_USER=postgres PG_PASSWORD= PG_DATABASE=school_compare D=$SCRATCH/dbt_local cd /Users/tudor/projects/school_compare/pipeline/transform uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main "\$@" --profiles-dir . --target dev --target-path \$D/target --log-path \$D/logs > \$D/last.log 2>&1 status=\$? sed 's/\x1b\[[0-9;]*m//g' \$D/last.log | grep -E "PASS|FAIL|ERROR|WARN |OK created|Done\.|Error|Compilation|Invalid" exit \$status EOF cat > "$SCRATCH/dbt_local/sql.sh" <<'EOF' #!/bin/bash # Run SQL from stdin against the local school_compare database; print the output. D=$(cd "$(dirname "$0")" && pwd) SQL=$(cat) D="$D" uv run --no-project --with pgserver python -c ' import os, pgserver srv = pgserver.get_server(os.environ["D"] + "/pgdata", cleanup_mode=None) print(srv.psql("\\c school_compare\n" + os.environ["SQL"]))' EOF cat > "$SCRATCH/dbt_local/raw_tables.py" <<'EOF' """Create raw. with a text column per declared field (what target-postgres loads). Usage: raw_tables.py : [ ...]""" import importlib, os, sys, pgserver module, cls = sys.argv[1].split(":") tap = getattr(importlib.import_module(module), cls)(config={}) wanted = set(sys.argv[2:]) stmts = ["create schema if not exists raw;"] for stream in tap.streams.values(): if stream.name in wanted: cols = ", ".join(f'"{c}" text' for c in stream.schema["properties"]) stmts.append(f'drop table if exists raw.{stream.name} cascade; create table raw.{stream.name} ({cols});') print("raw." + stream.name) srv = pgserver.get_server(os.environ["SCRATCH"] + "/dbt_local/pgdata", cleanup_mode=None) srv.psql("\\c school_compare\n" + "\n".join(stmts)) EOF chmod +x "$SCRATCH/dbt_local/dbt.sh" "$SCRATCH/dbt_local/sql.sh" ``` - [ ] **Step 2: Raw tables for GIAS and the school MI** ```bash cd /Users/tudor/projects/school_compare SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-gias --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_gias.tap:TapUKGIAS gias_establishments gias_links SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_ofsted.tap:TapUKOfsted ofsted_inspections ``` Expected: `raw.gias_establishments`, `raw.gias_links`, `raw.ofsted_inspections`. - [ ] **Step 3: Baseline build** Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_ofsted_inspections int_ofsted_latest fact_ofsted_latest dim_school` Expected: exit 0; the 10 `int_ofsted_latest` and 2 `dim_school` unit tests PASS on empty tables. If `pgserver` cannot start, stop and report: dbt tests are then proven only by the staging DAG runs, and the PR description must say so. ### Task 2: GIAS inspectorate and UKPRN **Files:** - Modify: `pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py` (after `th.Property("StatutoryHighAge", th.StringType),`) - Modify: `pipeline/transform/models/staging/stg_gias_establishments.sql` - Create: `pipeline/transform/models/staging/stg_gias_ukprn.sql` - Modify: `pipeline/dags/school_data_pipeline.py` (daily `dbt_build` selection) - Test: `pipeline/tests/test_gias_declared_fields.py` **Interfaces:** - Produces: raw columns `"UKPRN"`, `"InspectorateName (name)"`; `stg_gias_establishments.inspectorate_name text`; `stg_gias_ukprn(urn integer, ukprn text)`. - [ ] **Step 1: Write the failing test** `pipeline/tests/test_gias_declared_fields.py`: ```python """Every field the GIAS establishments stream declares must be a column of the extract GIAS publishes, or it silently loads as null. H4 adds UKPRN (matches Ofsted's further-education providers) and InspectorateName (name) ("ISI" for about 1,340 independent schools). HEADER is GIAS's own, from edubasealldata20261013.csv. """ import re from pathlib import Path TAP = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-gias' / 'tap_uk_gias' / 'tap.py') HEADER = ( 'URN', 'LA (code)', 'LA (name)', 'EstablishmentNumber', 'EstablishmentName', 'TypeOfEstablishment (code)', 'TypeOfEstablishment (name)', 'EstablishmentTypeGroup (code)', 'EstablishmentTypeGroup (name)', 'EstablishmentStatus (code)', 'EstablishmentStatus (name)', 'ReasonEstablishmentOpened (code)', 'ReasonEstablishmentOpened (name)', 'OpenDate', 'ReasonEstablishmentClosed (code)', 'ReasonEstablishmentClosed (name)', 'CloseDate', 'PhaseOfEducation (code)', 'PhaseOfEducation (name)', 'StatutoryLowAge', 'StatutoryHighAge', 'Boarders (code)', 'Boarders (name)', 'NurseryProvision (name)', 'OfficialSixthForm (code)', 'OfficialSixthForm (name)', 'Gender (code)', 'Gender (name)', 'ReligiousCharacter (code)', 'ReligiousCharacter (name)', 'ReligiousEthos (name)', 'Diocese (code)', 'Diocese (name)', 'AdmissionsPolicy (code)', 'AdmissionsPolicy (name)', 'SchoolCapacity', 'SpecialClasses (code)', 'SpecialClasses (name)', 'CensusDate', 'NumberOfPupils', 'NumberOfBoys', 'NumberOfGirls', 'PercentageFSM', 'TrustSchoolFlag (code)', 'TrustSchoolFlag (name)', 'Trusts (code)', 'Trusts (name)', 'SchoolSponsorFlag (name)', 'SchoolSponsors (name)', 'FederationFlag (name)', 'Federations (code)', 'Federations (name)', 'UKPRN', 'FEHEIdentifier', 'FurtherEducationType (name)', 'LastChangedDate', 'Street', 'Locality', 'Address3', 'Town', 'County (name)', 'Postcode', 'SchoolWebsite', 'TelephoneNum', 'HeadTitle (name)', 'HeadFirstName', 'HeadLastName', 'HeadPreferredJobTitle', 'BSOInspectorateName (name)', 'InspectorateReport', 'DateOfLastInspectionVisit', 'NextInspectionVisit', 'TeenMoth (name)', 'TeenMothPlaces', 'CCF (name)', 'SENPRU (name)', 'EBD (name)', 'PlacesPRU', 'FTProv (name)', 'EdByOther (name)', 'Section41Approved (name)', 'SEN1 (name)', 'SEN2 (name)', 'SEN3 (name)', 'SEN4 (name)', 'SEN5 (name)', 'SEN6 (name)', 'SEN7 (name)', 'SEN8 (name)', 'SEN9 (name)', 'SEN10 (name)', 'SEN11 (name)', 'SEN12 (name)', 'SEN13 (name)', 'TypeOfResourcedProvision (name)', 'ResourcedProvisionOnRoll', 'ResourcedProvisionCapacity', 'SenUnitOnRoll', 'SenUnitCapacity', 'GOR (code)', 'GOR (name)', 'DistrictAdministrative (code)', 'DistrictAdministrative (name)', 'AdministrativeWard (code)', 'AdministrativeWard (name)', 'ParliamentaryConstituency (code)', 'ParliamentaryConstituency (name)', 'UrbanRural (code)', 'UrbanRural (name)', 'GSSLACode (name)', 'Easting', 'Northing', 'MSOA (name)', 'LSOA (name)', 'InspectorateName (name)', 'SENStat', 'SENNoStat', 'BoardingEstablishment (name)', 'PropsName', 'PreviousLA (code)', 'PreviousLA (name)', 'PreviousEstablishmentNumber', 'Country (name)', 'UPRN', 'SiteName', 'QABName (code)', 'QABName (name)', 'EstablishmentAccredited (code)', 'EstablishmentAccredited (name)', 'QABReport', 'CHNumber', 'MSOA (code)', 'LSOA (code)', 'FSM', 'AccreditationExpiryDate', ) def declared(): source = TAP.read_text() block = source[source.index('class GIASEstablishmentsStream'):source.index('class GIASLinksStream')] return re.findall(r'th\.Property\("([^"]+)"', block) def test_every_declared_field_is_a_gias_column(): assert set(declared()) - set(HEADER) == set() def test_the_inspection_fields_are_declared(): assert {'UKPRN', 'InspectorateName (name)'} <= set(declared()) ``` - [ ] **Step 2: Run it to see it fail** Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_gias_declared_fields.py -q` Expected: `test_the_inspection_fields_are_declared` FAILS; `test_every_declared_field_is_a_gias_column` passes (a guard for existing fields). - [ ] **Step 3: Declare the fields, stage them, add the UKPRN view** In `tap_uk_gias/tap.py`, after `th.Property("StatutoryHighAge", th.StringType),`: ```python # Matches Ofsted's further-education providers, which some FE # records name by UKPRN only (audit H4). th.Property("UKPRN", th.StringType), # "ISI" for independent schools the Independent Schools Inspectorate # inspects; "Ofsted" or "Not applicable" otherwise (audit H4). th.Property("InspectorateName (name)", th.StringType), ``` In `stg_gias_establishments.sql`, after the line `"NurseryProvision (name)" as nursery_provision,`: ```sql nullif(trim("InspectorateName (name)"), '') as inspectorate_name, ``` `pipeline/transform/models/staging/stg_gias_ukprn.sql`: ```sql -- URN → UKPRN from the raw GIAS extract, for matching Ofsted's further -- education providers (int_ofsted_fe_latest, int_ofsted_listed; audit H4). -- A model on the raw source rather than a column of stg_gias_establishments: -- a ref to that model would put the Ofsted models into the daily build's -- stg_gias_establishments+ selection (pipeline/tests/test_dag_selectors.py). select cast("URN" as integer) as urn, nullif(trim("UKPRN"), '') as ukprn from {{ source('raw', 'gias_establishments') }} where "URN" ~ '^[0-9]+$' ``` In `pipeline/dags/school_data_pipeline.py`, in the daily build, change `--select stg_gias_establishments+ stg_gias_links+ gias_code_names+` to `--select stg_gias_establishments+ stg_gias_links+ gias_code_names+ stg_gias_ukprn`. - [ ] **Step 4: Run the tests and build** Run: the Step 2 pytest command → `2 passed`. Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_dag_selectors.py -q` → all pass. Re-create the GIAS raw tables (Task 1 Step 2, first command) → the table now has `"UKPRN"` and `"InspectorateName (name)"`. Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_gias_ukprn dim_school` → exit 0. - [ ] **Step 5: Commit** ```bash git add pipeline/tests/test_gias_declared_fields.py pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py pipeline/transform/models/staging/stg_gias_establishments.sql pipeline/transform/models/staging/stg_gias_ukprn.sql pipeline/dags/school_data_pipeline.py git commit -m "feat(gias): load UKPRN and the inspectorate GIAS names (H4)" ``` ### Task 3: Ofsted further-education extractor **Files:** - Create: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py` - Modify: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (imports, two stream classes, `discover_streams`) - Test: `pipeline/tests/test_ofsted_fe_mi.py` **Interfaces:** - Produces: `fe_pages(results: list[dict]) -> list[str]`; `renewed_framework_pages(results: list[dict]) -> list[str]`; `latest_as_at_csv(html: str) -> str | None`; `latest_period_csv(html: str) -> str | None`; `read_mi_csv(content: bytes) -> pd.DataFrame`; `records(df, columns: dict[str, str], key: str) -> Iterator[dict]`; `providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None`; `PROVIDER_COLUMNS`, `INSPECTION_COLUMNS`. Streams `ofsted_fe_providers` (key `provider_urn`; fields `provider_urn, provider_ukprn, provider_name, provider_type, renewed_eif_full_inspection, graded_inspection_date, overall_effectiveness, short_inspection_date`) and `ofsted_fe_inspections` (key `inspection_number`; fields `inspection_number, provider_urn, provider_ukprn, first_day_of_inspection, date_published, safeguarding_standards`), all strings. - [ ] **Step 1: Write the failing tests** `pipeline/tests/test_ofsted_fe_mi.py`: ```python """Ofsted publishes further-education inspection outcomes on a gov.uk page per academic year, separate from the school MI the tap already reads (audit H4). The links below are the real ones from the 2025/26 page; the headers are Ofsted's own. """ import importlib.util from pathlib import Path import pandas as pd import pytest MODULE = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-ofsted' / 'tap_uk_ofsted' / 'fe_mi.py') A = 'https://assets.publishing.service.gov.uk/media/' PAGE_2025_26 = f''' x x x x x x x x x ''' SEARCH_RESULTS = [ {'title': 'Further education and skills inspections and outcomes: management information from September 2024 to August 2025', 'link': '/government/statistical-data-sets/fe-2024-25'}, {'title': 'Further education and skills inspections and outcomes: management information from November 2025 to August 2026', 'link': '/government/statistical-data-sets/fe-2025-26'}, {'title': 'Further education and skills inspections and outcomes: management information from September 2023 to August 2024', 'link': '/government/statistical-data-sets/fe-2023-24'}, {'title': 'Monthly management information: Ofsted\'s school inspections outcomes', 'link': '/government/statistical-data-sets/schools'}, ] PROVIDER_HEADER = ( 'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group', 'Local authority', 'Region', 'Ofsted region', 'Has the provider received a full inspection under the renewed EIF?', 'Has the provider received a new provider monitoring inspection?', 'Date of latest short inspection before renewed EIF (if relevant)', 'Inspection number of latest full inspection before renewed EIF', 'First day of inspection of latest inspection before renewed EIF', 'Latest overall effectiveness before renewed EIF', 'Latest quality of education before renewed EIF', 'Latest behaviour and attitudes before renewed EIF', 'Latest personal development before renewed EIF', 'Latest leadership and management before renewed EIF', 'Latest contribution to meeting skills needs before renewed EIF', 'Latest safeguarding before renewed EIF', '', ) INSPECTION_HEADER = ( 'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group', 'Local authority', 'Region', 'Ofsted region', 'Inspection number', 'First day of inspection', 'Date published', 'Safeguarding standards', 'Inclusion', 'Leadership and governance', 'Contribution to meeting skills needs', 'Education programmes for young people - curriculum, teaching and training', 'Education programmes for young people - achievement', 'Education programmes for young people - participation and development', 'Adult learning programmes - curriculum, teaching and training', 'Adult learning programmes - achievement', 'Adult learning programmes - participation and development', 'Apprenticeships - curriculum, teaching and training', 'Apprenticeships - achievement', 'Apprenticeships - participation and development', 'Provision for learners and apprentices with high needs - curriculum, teaching and training', 'Provision for learners and apprentices with high needs - achievement', 'Provision for learners and apprentices with high needs - participation and development', '', ) @pytest.fixture def fe(): spec = importlib.util.spec_from_file_location('fe_mi', MODULE) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) return module def test_the_newest_as_at_file_is_chosen(fe): assert fe.latest_as_at_csv(PAGE_2025_26).endswith('_as_at_31_August_2026.csv') def test_the_latest_cumulative_inspections_file_is_chosen_not_a_monitoring_one(fe): assert fe.latest_period_csv(PAGE_2025_26) == ( A + '6a9feda65f6e942efe37eed6/Management_information_-_further_education_and_skills_' '-_10_November_2025_to_31_August_2026.csv') def test_a_doubled_underscore_file_name_still_parses(fe): page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'August' not in line) assert fe.latest_period_csv(page).endswith('__10_November_2025_to_28_February_2026.csv') def test_monitoring_files_alone_give_nothing(fe): page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'monitoring' in line) assert fe.latest_period_csv(page) is None assert fe.latest_as_at_csv(page) is None def test_pages_are_the_fe_series_newest_first(fe): assert fe.fe_pages(SEARCH_RESULTS) == [ '/government/statistical-data-sets/fe-2025-26', '/government/statistical-data-sets/fe-2024-25', '/government/statistical-data-sets/fe-2023-24', ] def test_report_cards_are_read_only_from_renewed_framework_pages(fe): # Earlier pages hold previous-framework inspection lists, not report cards. assert fe.renewed_framework_pages(SEARCH_RESULTS) == ['/government/statistical-data-sets/fe-2025-26'] def test_providers_fall_back_to_the_previous_page(fe): pages = {'/new': '

No files yet

', '/old': PAGE_2025_26} assert fe.providers_csv_url(['/new', '/old'], pages.__getitem__).endswith('_as_at_31_August_2026.csv') @pytest.mark.parametrize('columns, header', [('PROVIDER_COLUMNS', PROVIDER_HEADER), ('INSPECTION_COLUMNS', INSPECTION_HEADER)]) def test_every_mapped_column_is_in_ofsteds_file(fe, columns, header): assert set(getattr(fe, columns)) - set(header) == set() def test_csv_reading_and_records(fe): content = ('' + ','.join(f'"{c}"' for c in PROVIDER_HEADER[:14]) + '\n' + '145175,10005429,Queen Mary\'s College,Sixth form college,Sixth form colleges,Hampshire,' 'South East,South East,No,No,-,10227183,06/12/2022,2\n' + ',,Blank provider,,,,,,,,,,,\n').encode('utf-8') rows = list(fe.records(fe.read_mi_csv(content), fe.PROVIDER_COLUMNS, 'provider_urn')) assert rows == [{ 'provider_urn': '145175', 'provider_ukprn': '10005429', 'provider_name': "Queen Mary's College", 'provider_type': 'Sixth form college', 'renewed_eif_full_inspection': 'No', 'graded_inspection_date': '06/12/2022', 'overall_effectiveness': '2', 'short_inspection_date': '-', }] ``` - [ ] **Step 2: Run them to see them fail** Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_ofsted_fe_mi.py -q` Expected: every test errors with `FileNotFoundError` for `fe_mi.py`. - [ ] **Step 3: Write the module** `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py`: ```python """Ofsted's further education and skills management information (audit H4). Ofsted publishes FE inspection outcomes on one gov.uk page per academic year ("Further education and skills inspections and outcomes: management information from November 2025 to August 2026"). Each page carries "as at" files (one row per provider, with its latest inspection under the previous framework) and cumulative files of renewed-framework inspections, i.e. report cards ("…_10_November_2025_to_31_January_2026.csv", "…_to_28_February_2026.csv", …). Monitoring-visit files are a different series. Free of the Singer SDK so CI's pytest, which installs only the backend's requirements, can load it. """ from __future__ import annotations import io import re from datetime import date from typing import Callable, Iterator import pandas as pd SEARCH_URL = "https://www.gov.uk/api/search.json" SEARCH_QUERY = "further education and skills inspections and outcomes management information" PAGE_TITLE_PREFIX = "Further education and skills inspections and outcomes: management information from " # Renewed-framework (report card) inspections began on 10 November 2025. RENEWED_FRAMEWORK_START = date(2025, 11, 1) _MONTHS = {name: number for number, name in enumerate( ["january", "february", "march", "april", "may", "june", "july", "august", "september", "october", "november", "december"], start=1)} _TITLE_START = re.compile(r"from ([A-Za-z]+) (\d{4})") _CSV_LINK = re.compile(r'https://assets\.publishing\.service\.gov\.uk/[^"\s<>]+\.csv') _AS_AT = re.compile(r"further_education_and_skills_-_+as_at_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$") _PERIOD = re.compile( r"further_education_and_skills_-_+(\d{1,2})_([A-Za-z]+)_(\d{4})_to_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$") # Ofsted column → Singer field. PROVIDER_COLUMNS = { "Provider URN": "provider_urn", "Provider UKPRN": "provider_ukprn", "Provider name": "provider_name", "Provider type": "provider_type", "Has the provider received a full inspection under the renewed EIF?": "renewed_eif_full_inspection", "First day of inspection of latest inspection before renewed EIF": "graded_inspection_date", "Latest overall effectiveness before renewed EIF": "overall_effectiveness", "Date of latest short inspection before renewed EIF (if relevant)": "short_inspection_date", } INSPECTION_COLUMNS = { "Provider URN": "provider_urn", "Provider UKPRN": "provider_ukprn", "Inspection number": "inspection_number", "First day of inspection": "first_day_of_inspection", "Date published": "date_published", "Safeguarding standards": "safeguarding_standards", } def _date(day: str, month: str, year: str) -> date | None: number = _MONTHS.get(month.lower()) return date(int(year), number, int(day)) if number else None def _page_start(title: str) -> date | None: match = _TITLE_START.search(title) number = _MONTHS.get(match.group(1).lower()) if match else None return date(int(match.group(2)), number, 1) if number else None def _dated_pages(results: list[dict]) -> list[tuple[date, str]]: pages = [] for result in results: title = result.get("title", "") start = _page_start(title) if title.startswith(PAGE_TITLE_PREFIX) else None if start: pages.append((start, result["link"])) return sorted(pages, reverse=True) def fe_pages(results: list[dict]) -> list[str]: """gov.uk paths of the FE MI pages, newest academic year first.""" return [link for _, link in _dated_pages(results)] def renewed_framework_pages(results: list[dict]) -> list[str]: """The pages that can hold report cards: from November 2025 onwards.""" return [link for start, link in _dated_pages(results) if start >= RENEWED_FRAMEWORK_START] def _latest(html: str, pattern: re.Pattern, date_groups: slice) -> str | None: best = None for link in set(_CSV_LINK.findall(html)): match = pattern.search(link) when = _date(*match.groups()[date_groups]) if match else None if when and (best is None or when > best[0]): best = (when, link) return best[1] if best else None def latest_as_at_csv(html: str) -> str | None: """The newest "as at" CSV on a page.""" return _latest(html, _AS_AT, slice(0, 3)) def latest_period_csv(html: str) -> str | None: """The renewed-framework inspections CSV with the latest end date on a page.""" return _latest(html, _PERIOD, slice(3, 6)) def providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None: """The newest "as at" CSV, from the newest page that has one.""" for page in pages: url = latest_as_at_csv(fetch(page)) if url: return url return None def read_mi_csv(content: bytes) -> pd.DataFrame: """Ofsted's MI CSVs: UTF-8 with a BOM, or Windows-1252. The header is the line naming "Provider URN".""" try: text = content.decode("utf-8-sig") except UnicodeDecodeError: text = content.decode("cp1252") lines = text.splitlines() start = next((i for i, line in enumerate(lines[:20]) if "Provider URN" in line), 0) return pd.read_csv(io.StringIO("\n".join(lines[start:])), dtype=str, keep_default_na=False) def records(df: pd.DataFrame, columns: dict[str, str], key: str) -> Iterator[dict]: """Singer records: the mapped columns, stripped. Rows without `key` are skipped.""" present = {source: field for source, field in columns.items() if source in df.columns} for _, row in df.iterrows(): record = {field: str(row[source]).strip() for source, field in present.items()} if record.get(key): yield record ``` - [ ] **Step 4: Add the two streams** In `tap_uk_ofsted/tap.py`, after `from singer_sdk import typing as th`: ```python from tap_uk_ofsted.fe_mi import ( INSPECTION_COLUMNS, PROVIDER_COLUMNS, SEARCH_QUERY, SEARCH_URL, fe_pages, latest_period_csv, providers_csv_url, read_mi_csv, records, renewed_framework_pages, ) ``` Before `class TapUKOfsted(Tap):`: ```python def _fe_search_results() -> list[dict]: """gov.uk search results for Ofsted's FE MI pages (one per academic year).""" resp = requests.get(SEARCH_URL, params={"q": SEARCH_QUERY, "count": 50}, timeout=60) resp.raise_for_status() return resp.json().get("results", []) def _gov_uk_page(path: str) -> str: url = path if path.startswith("http") else "https://www.gov.uk" + path resp = requests.get(url, timeout=60) resp.raise_for_status() return resp.text class OfstedFEProvidersStream(Stream): """Ofsted's further education and skills providers: the newest "as at" file, one row per provider with its latest inspection under the previous framework (audit H4). Page and file selection: fe_mi.py.""" name = "ofsted_fe_providers" primary_keys = ["provider_urn"] replication_key = None schema = th.PropertiesList( th.Property("provider_urn", th.StringType, required=True), *[th.Property(field, th.StringType) for field in PROVIDER_COLUMNS.values() if field != "provider_urn"], ).to_dict() def get_records(self, context): url = providers_csv_url(fe_pages(_fe_search_results()), _gov_uk_page) if not url: self.logger.error("No Ofsted FE 'as at' file found") return self.logger.info("Downloading Ofsted FE providers MI: %s", url) resp = requests.get(url, timeout=120) resp.raise_for_status() yield from records(read_mi_csv(resp.content), PROVIDER_COLUMNS, "provider_urn") class OfstedFEInspectionsStream(Stream): """Ofsted's renewed-framework FE inspections (report cards), from the latest cumulative file on every page since November 2025, so a report card survives the next academic year's page (audit H4).""" name = "ofsted_fe_inspections" primary_keys = ["inspection_number"] replication_key = None schema = th.PropertiesList( th.Property("inspection_number", th.StringType, required=True), *[th.Property(field, th.StringType) for field in INSPECTION_COLUMNS.values() if field != "inspection_number"], ).to_dict() def get_records(self, context): for page in renewed_framework_pages(_fe_search_results()): url = latest_period_csv(_gov_uk_page(page)) if not url: self.logger.warning("No renewed-framework FE inspections file on %s", page) continue self.logger.info("Downloading Ofsted FE inspections MI: %s", url) resp = requests.get(url, timeout=120) resp.raise_for_status() yield from records(read_mi_csv(resp.content), INSPECTION_COLUMNS, "inspection_number") ``` Change `discover_streams` to `return [OfstedInspectionsStream(self), OfstedFEProvidersStream(self), OfstedFEInspectionsStream(self)]`. - [ ] **Step 5: Run the tests, then both streams against gov.uk** Run: the Step 2 pytest command → `11 passed`. ```bash uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted python -c " from tap_uk_ofsted.tap import TapUKOfsted tap = TapUKOfsted(config={}) p = list(tap.streams['ofsted_fe_providers'].get_records(None)) i = list(tap.streams['ofsted_fe_inspections'].get_records(None)) print(len(p), [r for r in p if r['provider_urn'] in ('145175', '143526', '130645')]) print(len(i), [r for r in i if r['provider_urn'] == '143526'])" ``` Expected: about 1,979 providers (Ofsted's count as at 31 August 2026; a newer file may differ slightly) with Queen Mary's College graded `2` on `06/12/2022`, The Autism Project – CareTrade `2` on `16/11/2021`, Exeter and North Devon Colleges Group `-`; about 1,472 inspections with 143526 inspected `12/05/2026`, safeguarding `Met`. - [ ] **Step 6: Commit** ```bash git add pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py pipeline/tests/test_ofsted_fe_mi.py git commit -m "feat(ofsted): extract Ofsted's further education and skills MI (H4)" ``` ### Task 4: FE staging models **Files:** - Modify: `pipeline/transform/models/staging/_stg_sources.yml` - Create: `pipeline/transform/models/staging/stg_ofsted_fe_providers.sql`, `stg_ofsted_fe_inspections.sql`, `stg_ofsted_fe.yml` **Interfaces:** - Consumes: raw `ofsted_fe_providers`, `ofsted_fe_inspections` (Task 3 fields). - Produces: `stg_ofsted_fe_providers(provider_urn integer, provider_ukprn text, provider_name text, provider_type text, renewed_eif_full_inspection boolean, graded_inspection_date date, overall_effectiveness integer, short_inspection_date date)`; `stg_ofsted_fe_inspections(inspection_number text, provider_urn integer, provider_ukprn text, rc_inspection_date date, published_date date, rc_safeguarding_met boolean)`. - [ ] **Step 1: Declare the sources, create the raw tables, write the unit tests** In `_stg_sources.yml`, after the `ofsted_inspections` entry: ```yaml - name: ofsted_fe_providers description: Ofsted further education and skills MI, newest "as at" file — one row per provider with its latest inspection under the previous framework - name: ofsted_fe_inspections description: Ofsted further education and skills MI — renewed-framework inspections (report cards) since 10 November 2025, one row per inspection ``` Create the raw tables: `SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_ofsted.tap:TapUKOfsted ofsted_fe_providers ofsted_fe_inspections` (from the repo root) → two `raw.…` lines. `pipeline/transform/models/staging/stg_ofsted_fe.yml`: ```yaml version: 2 unit_tests: - name: stg_ofsted_fe_providers_parses_grades_and_dates description: Ofsted writes "-" for no grade or date; grades are 1-4. model: stg_ofsted_fe_providers given: - input: source('raw', 'ofsted_fe_providers') rows: - {provider_urn: '145175', provider_ukprn: '10005429', provider_name: "Queen Mary's College", renewed_eif_full_inspection: 'No', graded_inspection_date: '06/12/2022', overall_effectiveness: '2', short_inspection_date: '-'} - {provider_urn: '130645', provider_ukprn: '-', provider_name: 'Exeter and North Devon Colleges Group', renewed_eif_full_inspection: 'No', graded_inspection_date: '-', overall_effectiveness: '-', short_inspection_date: '-'} - {provider_urn: '', provider_ukprn: '10000001', provider_name: 'No URN'} expect: rows: - {provider_urn: 145175, provider_ukprn: '10005429', renewed_eif_full_inspection: false, graded_inspection_date: '2022-12-06', overall_effectiveness: 2, short_inspection_date: null} - {provider_urn: 130645, provider_ukprn: null, renewed_eif_full_inspection: false, graded_inspection_date: null, overall_effectiveness: null, short_inspection_date: null} - name: stg_ofsted_fe_inspections_parses_dates_and_safeguarding model: stg_ofsted_fe_inspections given: - input: source('raw', 'ofsted_fe_inspections') rows: - {inspection_number: '10380001', provider_urn: '143526', provider_ukprn: '10000002', first_day_of_inspection: '12/05/2026', date_published: '16/06/2026', safeguarding_standards: 'Met'} - {inspection_number: '10380002', provider_urn: '150042', provider_ukprn: '10000003', first_day_of_inspection: '20/01/2026', date_published: '24/02/2026', safeguarding_standards: 'Not met'} - {inspection_number: '10380003', provider_urn: '1', provider_ukprn: '', first_day_of_inspection: '', date_published: '', safeguarding_standards: ''} expect: rows: - {inspection_number: '10380001', provider_urn: 143526, rc_inspection_date: '2026-05-12', published_date: '2026-06-16', rc_safeguarding_met: true} - {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', published_date: '2026-02-24', rc_safeguarding_met: false} ``` - [ ] **Step 2: Run them to see them fail** Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "stg_ofsted_fe_providers stg_ofsted_fe_inspections,test_type:unit"` Expected: non-zero exit; parsing fails because the two models do not exist. - [ ] **Step 3: Write the models** `pipeline/transform/models/staging/stg_ofsted_fe_providers.sql`: ```sql -- Staging model: Ofsted further education and skills providers — the "as at" -- MI file, one row per provider with its latest inspection under the previous -- framework (audit H4). Ofsted writes '-' for no grade or date. select cast(trim(provider_urn) as integer) as provider_urn, nullif(nullif(trim(provider_ukprn), ''), '-') as provider_ukprn, provider_name, provider_type, lower(trim(renewed_eif_full_inspection)) = 'yes' as renewed_eif_full_inspection, case when trim(graded_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$' then to_date(trim(graded_inspection_date), 'DD/MM/YYYY') end as graded_inspection_date, case when trim(overall_effectiveness) in ('1', '2', '3', '4') then trim(overall_effectiveness)::integer end as overall_effectiveness, case when trim(short_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$' then to_date(trim(short_inspection_date), 'DD/MM/YYYY') end as short_inspection_date from {{ source('raw', 'ofsted_fe_providers') }} where trim(provider_urn) ~ '^[0-9]+$' ``` `pipeline/transform/models/staging/stg_ofsted_fe_inspections.sql`: ```sql -- Staging model: Ofsted's renewed-framework FE inspections (report cards), -- one row per inspection (audit H4). Only the headline is kept: the date and -- the safeguarding result. select trim(inspection_number) as inspection_number, cast(trim(provider_urn) as integer) as provider_urn, nullif(nullif(trim(provider_ukprn), ''), '-') as provider_ukprn, to_date(trim(first_day_of_inspection), 'DD/MM/YYYY') as rc_inspection_date, case when trim(date_published) ~ '^\d{2}/\d{2}/\d{4}$' then to_date(trim(date_published), 'DD/MM/YYYY') end as published_date, case lower(trim(safeguarding_standards)) when 'met' then true when 'not met' then false end as rc_safeguarding_met from {{ source('raw', 'ofsted_fe_inspections') }} where trim(provider_urn) ~ '^[0-9]+$' and trim(first_day_of_inspection) ~ '^\d{2}/\d{2}/\d{4}$' ``` - [ ] **Step 4: Run the unit tests and build** Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_ofsted_fe_providers stg_ofsted_fe_inspections` Expected: exit 0; both unit tests PASS; both views created. - [ ] **Step 5: Commit** ```bash git add pipeline/transform/models/staging/_stg_sources.yml pipeline/transform/models/staging/stg_ofsted_fe_providers.sql pipeline/transform/models/staging/stg_ofsted_fe_inspections.sql pipeline/transform/models/staging/stg_ofsted_fe.yml git commit -m "feat(dbt): stage Ofsted's further education and skills MI (H4)" ``` ### Task 5: `int_ofsted_fe_latest` **Files:** - Create: `pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`, `int_ofsted_fe_latest.yml` **Interfaces:** - Consumes: `stg_gias_ukprn`, `stg_ofsted_fe_providers`, `stg_ofsted_fe_inspections`. - Produces: `int_ofsted_fe_latest` with exactly `int_ofsted_latest`'s columns, in its order and types: `urn, latest_visit_date, latest_visit_kind, latest_visit_outcome, current_grade, current_grade_date, current_grade_basis, graded_inspection_date, ungraded_inspection_date, inspection_date, inspection_type, framework, overall_effectiveness, quality_of_education, behaviour_attitudes, personal_development, leadership_management, early_years_provision, sixth_form_provision, ungraded_outcome, ungraded_grade, rc_safeguarding_met, rc_inclusion, rc_curriculum_teaching, rc_achievement, rc_attendance_behaviour, rc_personal_development, rc_leadership_governance, rc_early_years, rc_sixth_form, rc_inspection_date, report_url`. `latest_visit_kind` ∈ `report_card`, `graded`, `short`. - [ ] **Step 1: Write the unit tests** `pipeline/transform/models/intermediate/int_ofsted_fe_latest.yml`: ```yaml version: 2 unit_tests: - name: int_ofsted_fe_grade_only description: Queen Mary's College (145175), Good on 6 December 2022, no report card. model: int_ofsted_fe_latest given: - input: ref('stg_gias_ukprn') rows: - {urn: 145175, ukprn: '10005429'} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 145175, provider_ukprn: '10005429', graded_inspection_date: '2022-12-06', overall_effectiveness: 2} - input: ref('stg_ofsted_fe_inspections') rows: [] expect: rows: - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, rc_inspection_date: null, framework: 'Further education and skills'} - name: int_ofsted_fe_short_inspection_is_a_visit_not_a_grade model: int_ofsted_fe_latest given: - input: ref('stg_gias_ukprn') rows: - {urn: 1, ukprn: null} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 1, graded_inspection_date: '2019-03-05', overall_effectiveness: 2, short_inspection_date: '2023-02-01'} - input: ref('stg_ofsted_fe_inspections') rows: [] expect: rows: - {urn: 1, latest_visit_date: '2023-02-01', latest_visit_kind: short, current_grade: 2, current_grade_date: '2019-03-05', current_grade_basis: graded} - name: int_ofsted_fe_newer_report_card_wins description: The Autism Project - CareTrade (143526). Good 2021; report card 12 May 2026, safeguarding met. model: int_ofsted_fe_latest given: - input: ref('stg_gias_ukprn') rows: - {urn: 143526, ukprn: '10000002'} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 143526, provider_ukprn: '10000002', graded_inspection_date: '2021-11-16', overall_effectiveness: 2} - input: ref('stg_ofsted_fe_inspections') rows: - {inspection_number: '10380001', provider_urn: 143526, provider_ukprn: '10000002', rc_inspection_date: '2026-05-12', rc_safeguarding_met: true} expect: rows: - {urn: 143526, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, current_grade: null, current_grade_date: null, current_grade_basis: null, rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, overall_effectiveness: 2} - name: int_ofsted_fe_report_card_only model: int_ofsted_fe_latest given: - input: ref('stg_gias_ukprn') rows: - {urn: 150042, ukprn: null} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 150042, graded_inspection_date: null, overall_effectiveness: null} - input: ref('stg_ofsted_fe_inspections') rows: - {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false} expect: rows: - {urn: 150042, latest_visit_date: '2026-01-20', latest_visit_kind: report_card, current_grade: null, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false} - name: int_ofsted_fe_listed_never_inspected_has_no_row model: int_ofsted_fe_latest given: - input: ref('stg_gias_ukprn') rows: - {urn: 130645, ukprn: null} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 130645, graded_inspection_date: null, overall_effectiveness: null} - input: ref('stg_ofsted_fe_inspections') rows: [] expect: rows: [] - name: int_ofsted_fe_matches_by_ukprn_when_the_urn_differs model: int_ofsted_fe_latest given: - input: ref('stg_gias_ukprn') rows: - {urn: 5, ukprn: '10099999'} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 9999999, provider_ukprn: '10099999', graded_inspection_date: '2024-01-30', overall_effectiveness: 3} - input: ref('stg_ofsted_fe_inspections') rows: [] expect: rows: - {urn: 5, current_grade: 3, current_grade_date: '2024-01-30'} - name: int_ofsted_fe_prefers_the_urn_match model: int_ofsted_fe_latest given: - input: ref('stg_gias_ukprn') rows: - {urn: 6, ukprn: '10088888'} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 6, provider_ukprn: '10077777', graded_inspection_date: '2023-05-01', overall_effectiveness: 1} - {provider_urn: 8888888, provider_ukprn: '10088888', graded_inspection_date: '2024-05-01', overall_effectiveness: 4} - input: ref('stg_ofsted_fe_inspections') rows: [] expect: rows: - {urn: 6, current_grade: 1, current_grade_date: '2023-05-01'} - name: int_ofsted_fe_report_card_older_than_grade_is_ignored description: Bad data guard. A report card dated before the graded inspection cannot be the latest, and a row must never carry both a grade and a report card. model: int_ofsted_fe_latest given: - input: ref('stg_gias_ukprn') rows: - {urn: 7, ukprn: null} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 7, graded_inspection_date: '2026-03-01', overall_effectiveness: 2} - input: ref('stg_ofsted_fe_inspections') rows: - {inspection_number: '1', provider_urn: 7, rc_inspection_date: '2026-01-10', rc_safeguarding_met: true} expect: rows: - {urn: 7, latest_visit_kind: graded, current_grade: 2, rc_inspection_date: null, rc_safeguarding_met: null} ``` - [ ] **Step 2: Run them to see them fail** Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_fe_latest,test_type:unit"` Expected: non-zero exit; parsing fails (model missing). - [ ] **Step 3: Write the model** `pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`: ```sql -- Intermediate model: Ofsted further education and skills status per URN -- (audit H4), in int_ofsted_latest's columns so it can be unioned there. -- A provider matches a school by Provider URN, else by UKPRN (stg_gias_ukprn). -- Rule, in the C1 status rule's terms -- (docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md): -- a report card on or after the previous-framework inspection wins and -- leaves no grade in force; otherwise a previous-framework grade 1-4 is the -- current grade, dated by that inspection; the latest visit is the later of -- it and any short inspection (whose outcome the MI does not give). with gias as ( select urn, ukprn from {{ ref('stg_gias_ukprn') }} ), providers as ( select distinct on (g.urn) g.urn, p.graded_inspection_date, p.overall_effectiveness, p.short_inspection_date from gias g join {{ ref('stg_ofsted_fe_providers') }} p on p.provider_urn = g.urn or (g.ukprn is not null and p.provider_ukprn = g.ukprn) order by g.urn, (p.provider_urn = g.urn) desc, p.graded_inspection_date desc nulls last ), report_cards as ( select distinct on (g.urn) g.urn, c.rc_inspection_date, c.rc_safeguarding_met from gias g join {{ ref('stg_ofsted_fe_inspections') }} c on c.provider_urn = g.urn or (g.ukprn is not null and c.provider_ukprn = g.ukprn) order by g.urn, (c.provider_urn = g.urn) desc, c.rc_inspection_date desc, c.inspection_number desc ), combined as ( select coalesce(p.urn, c.urn) as urn, p.graded_inspection_date, p.overall_effectiveness, p.short_inspection_date, c.rc_inspection_date as card_date, c.rc_safeguarding_met as card_safeguarding_met from providers p full outer join report_cards c on c.urn = p.urn ), decided as ( select *, card_date is not null and card_date >= coalesce(greatest(graded_inspection_date, short_inspection_date), card_date) as report_card_latest from combined ) select urn, case when report_card_latest then card_date else greatest(graded_inspection_date, short_inspection_date) end as latest_visit_date, case when report_card_latest then 'report_card' when short_inspection_date > graded_inspection_date then 'short' else 'graded' end as latest_visit_kind, null::text as latest_visit_outcome, case when not report_card_latest then overall_effectiveness end as current_grade, case when not report_card_latest then graded_inspection_date end as current_grade_date, case when not report_card_latest then 'graded' end as current_grade_basis, graded_inspection_date, null::date as ungraded_inspection_date, coalesce(graded_inspection_date, card_date) as inspection_date, null::text as inspection_type, 'Further education and skills'::text as framework, overall_effectiveness, null::integer as quality_of_education, null::integer as behaviour_attitudes, null::integer as personal_development, null::integer as leadership_management, null::integer as early_years_provision, null::integer as sixth_form_provision, null::text as ungraded_outcome, null::integer as ungraded_grade, case when report_card_latest then card_safeguarding_met end as rc_safeguarding_met, null::integer as rc_inclusion, null::integer as rc_curriculum_teaching, null::integer as rc_achievement, null::integer as rc_attendance_behaviour, null::integer as rc_personal_development, null::integer as rc_leadership_governance, null::integer as rc_early_years, null::integer as rc_sixth_form, case when report_card_latest then card_date end as rc_inspection_date, null::text as report_url from decided where report_card_latest or overall_effectiveness between 1 and 4 ``` - [ ] **Step 4: Run the unit tests and build** Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_ukprn stg_ofsted_fe_providers stg_ofsted_fe_inspections int_ofsted_fe_latest` Expected: exit 0; all 8 `int_ofsted_fe_latest` unit tests PASS. - [ ] **Step 5: Commit** ```bash git add pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql pipeline/transform/models/intermediate/int_ofsted_fe_latest.yml git commit -m "feat(dbt): further education status per URN under the C1 rule (H4)" ``` ### Task 6: Fold FE into `int_ofsted_latest` / `fact_ofsted_latest` **Files:** - Modify: `pipeline/transform/models/intermediate/int_ofsted_latest.sql`, `int_ofsted_latest.yml` - Modify: `pipeline/transform/models/marts/_marts_schema.yml` (`fact_ofsted_latest.latest_visit_kind` accepted values) - Create: `pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql` **Interfaces:** - Consumes: `int_ofsted_fe_latest` (Task 5; same columns and order as `int_ofsted_latest`). - Produces: `int_ofsted_latest` / `fact_ofsted_latest` rows with `framework = 'Further education and skills'` for URNs that have no school-framework record; `latest_visit_kind` may be `short`. - [ ] **Step 1: Write the failing unit tests, and give the existing ones the new input** In `int_ofsted_latest.yml`, under `given:` of **each** of the 10 existing tests, after its `stg_ofsted_inspections` input, add: ```yaml - input: ref('int_ofsted_fe_latest') rows: [] ``` Append: ```yaml - name: fe_status_fills_a_school_with_no_school_record description: Queen Mary's College (145175) has no school-framework record; its FE status passes through (audit H4). model: int_ofsted_latest given: - input: ref('stg_ofsted_inspections') rows: [] - input: ref('int_ofsted_fe_latest') rows: - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'} expect: rows: - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'} - name: a_school_record_wins_over_an_fe_record model: int_ofsted_latest given: - input: ref('stg_ofsted_inspections') rows: - {urn: 1, graded_inspection_date: '2019-06-01', overall_effectiveness: 2, framework: 'Schools - S5'} - input: ref('int_ofsted_fe_latest') rows: - {urn: 1, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, rc_inspection_date: '2026-05-12', framework: 'Further education and skills'} expect: rows: - {urn: 1, latest_visit_date: '2019-06-01', latest_visit_kind: graded, current_grade: 2, framework: 'Schools - S5'} ``` - [ ] **Step 2: Run them to see them fail** Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_latest,test_type:unit"` Expected: non-zero exit — `int_ofsted_latest` does not reference `int_ofsted_fe_latest`, so the given input is rejected (or, if dbt accepts unused inputs, `fe_status_fills_a_school_with_no_school_record` FAILS with no rows). - [ ] **Step 3: Union the FE rows** In `int_ofsted_latest.sql`, add to the header comment: ```sql -- Schools with no school-framework record take their further education and -- skills status from int_ofsted_fe_latest (same columns, same order; audit H4). ``` Replace the final `select … from graded` with the same select wrapped in a CTE, followed by the union: ```sql school_status as ( select urn, -- (… the existing select list, unchanged, through report_url …) report_url from graded ), fe_status as ( -- Same columns in the same order as school_status. select f.* from {{ ref('int_ofsted_fe_latest') }} f where not exists (select 1 from school_status s where s.urn = f.urn) ) select * from school_status union all select * from fe_status ``` (The `school_status` select list is the existing final select list verbatim; only `select` becomes `school_status as (select` and a `)` closes it. Remember the comma after the `graded` CTE's closing parenthesis.) In `_marts_schema.yml`, `fact_ofsted_latest` → `latest_visit_kind` → `accepted_values` → `values: ['report_card', 'graded', 'ungraded', 'short']`. `pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql`: ```sql -- A school-framework record always wins over a further education one for the -- same URN (audit H4): no FE row may stand where Ofsted's school MI has one. select f.urn from {{ ref('fact_ofsted_latest') }} f join {{ ref('stg_ofsted_inspections') }} s on s.urn = f.urn where f.framework = 'Further education and skills' ``` - [ ] **Step 4: Run the unit tests and build** Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select int_ofsted_fe_latest int_ofsted_latest fact_ofsted_latest assert_ofsted_fe_never_overrides_school assert_ofsted_current_grade_consistent` Expected: exit 0; 12 `int_ofsted_latest` unit tests PASS; schema and data tests PASS on empty tables. - [ ] **Step 5: Commit** ```bash git add pipeline/transform/models/intermediate/int_ofsted_latest.sql pipeline/transform/models/intermediate/int_ofsted_latest.yml pipeline/transform/models/marts/_marts_schema.yml pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql git commit -m "feat(dbt): further education status where a school has no school record (H4)" ``` ### Task 7: `int_ofsted_listed`, `dim_school` columns, DAG selectors **Files:** - Create: `pipeline/transform/models/intermediate/int_ofsted_listed.sql`, `int_ofsted_listed.yml` - Modify: `pipeline/transform/models/marts/dim_school.sql`, `dim_school.yml` - Modify: `pipeline/dags/school_data_pipeline.py` (monthly Ofsted selection), `pipeline/tests/test_dag_selectors.py` **Interfaces:** - Consumes: raw `ofsted_inspections`, `stg_gias_ukprn`, `stg_ofsted_fe_providers`, `stg_gias_establishments.inspectorate_name`. - Produces: `int_ofsted_listed(urn integer)`; `dim_school.inspectorate text`, `dim_school.ofsted_listed boolean` (null when `int_ofsted_listed` does not exist yet). - [ ] **Step 1: Write the unit tests** `pipeline/transform/models/intermediate/int_ofsted_listed.yml`: ```yaml version: 2 unit_tests: - name: int_ofsted_listed_includes_uninspected_schools_and_fe_providers description: > Abbas and Templecombe (150031) is in Ofsted's school MI with no inspection; Exeter and North Devon Colleges Group (130645) is in the FE MI with none; a UKPRN-only FE match counts too. A school in neither is absent (audit H4). model: int_ofsted_listed given: - input: source('raw', 'ofsted_inspections') rows: - {urn: '150031', inspection_date: 'NULL'} - {urn: '102408', inspection_date: '17/06/2025'} - input: ref('stg_gias_ukprn') rows: - {urn: 130645, ukprn: null} - {urn: 5, ukprn: '10099999'} - {urn: 133058, ukprn: null} - input: ref('stg_ofsted_fe_providers') rows: - {provider_urn: 130645, provider_ukprn: null} - {provider_urn: 9999999, provider_ukprn: '10099999'} expect: rows: - {urn: 150031} - {urn: 102408} - {urn: 130645} - {urn: 5} ``` In `dim_school.yml`, under `given:` of both existing tests add: ```yaml - input: ref('int_ofsted_listed') rows: [] ``` and append: ```yaml - name: dim_school_carries_inspectorate_and_ofsted_listing description: > Audit H4. GIAS names ISI for Abbey College Cambridge; Ofsted lists Abbas and Templecombe without an inspection; a third school is in neither. model: dim_school given: - input: ref('stg_gias_establishments') rows: - {urn: 136083, school_name: 'Abbey College Cambridge', status_code: 1, school_type_code: 11, local_authority_code: 873, phase_code: 4, inspectorate_name: 'ISI'} - {urn: 150031, school_name: 'Abbas and Templecombe Church School', status_code: 1, school_type_code: 28, local_authority_code: 933, phase_code: 2, inspectorate_name: 'Not applicable'} - {urn: 1, school_name: 'No data school', status_code: 1, school_type_code: 1, local_authority_code: 312, phase_code: 2, inspectorate_name: null} - input: ref('int_ofsted_latest') rows: [] - input: ref('int_ofsted_listed') rows: - {urn: 150031} expect: rows: - {urn: 136083, inspectorate: 'ISI', ofsted_listed: false} - {urn: 150031, inspectorate: 'Not applicable', ofsted_listed: true} - {urn: 1, inspectorate: null, ofsted_listed: false} ``` - [ ] **Step 2: Run them to see them fail** Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_listed dim_school,test_type:unit"` Expected: non-zero exit (model `int_ofsted_listed` missing; `dim_school` has no such columns). - [ ] **Step 3: Write `int_ofsted_listed` and extend `dim_school`** `pipeline/transform/models/intermediate/int_ofsted_listed.sql`: ```sql -- Intermediate model: every URN Ofsted's own data lists, inspected or not -- (audit H4). Only these may be called "Not yet inspected". Reads the raw -- school MI because stg_ofsted_inspections drops rows with no inspection date. with school_mi as ( select distinct cast(urn as integer) as urn from {{ source('raw', 'ofsted_inspections') }} where urn is not null ), fe as ( select distinct g.urn from {{ ref('stg_gias_ukprn') }} g join {{ ref('stg_ofsted_fe_providers') }} p on p.provider_urn = g.urn or (g.ukprn is not null and p.provider_ukprn = g.ukprn) ) select urn from school_mi union select urn from fe ``` In `dim_school.sql`: 1. Below the existing `-- depends_on: {{ ref('int_ofsted_latest') }}` line add `-- depends_on: {{ ref('int_ofsted_listed') }}` and extend the comment above it: `int_ofsted_latest and int_ofsted_listed are only ref()'d inside conditional blocks below (they may not exist before the Ofsted pipeline first runs), so declare the dependencies explicitly for dbt's DAG.` 2. After the `{% set ofsted_relation = … %}` block add: ```sql {% set listed_relation = adapter.get_relation( database=target.database, schema='intermediate', identifier='int_ofsted_listed' ) %} ``` 3. After `s.admissions_policy_code,` add: ```sql -- The inspectorate GIAS names ("ISI" for most independent schools) and -- whether Ofsted's own data lists the school, inspected or not (audit H4). s.inspectorate_name as inspectorate, {% if listed_relation is not none %} l.urn is not null as ofsted_listed, {% else %} null::boolean as ofsted_listed, {% endif %} ``` 4. After the `{% if ofsted_relation is not none %} left join … {% endif %}` block add: ```sql {% if listed_relation is not none %} left join {{ ref('int_ofsted_listed') }} l on s.urn = l.urn {% endif %} ``` In `pipeline/dags/school_data_pipeline.py`, in the monthly Ofsted build change `--select stg_ofsted_inspections+ int_ofsted_latest+` to `--select stg_ofsted_inspections+ stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ int_ofsted_latest+`. In `pipeline/tests/test_dag_selectors.py`: ```python # dim_school reads int_ofsted_latest and int_ofsted_listed only when the # relations exist (adapter.get_relation), so a missing table is not a failure. OPTIONAL_PARENTS = {'int_ofsted_latest', 'int_ofsted_listed'} ``` - [ ] **Step 4: Run the tests and build** Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select int_ofsted_listed dim_school` Expected: exit 0; the `int_ofsted_listed` unit test and 3 `dim_school` unit tests PASS. Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests -q` → all pass. - [ ] **Step 5: Commit** ```bash git add pipeline/transform/models/intermediate/int_ofsted_listed.sql pipeline/transform/models/intermediate/int_ofsted_listed.yml pipeline/transform/models/marts/dim_school.sql pipeline/transform/models/marts/dim_school.yml pipeline/dags/school_data_pipeline.py pipeline/tests/test_dag_selectors.py git commit -m "feat(dbt): dim_school.inspectorate and ofsted_listed (H4)" ``` ### Task 8: Load the real data end to end and verify PR 1 **Files:** none in the repo. - [ ] **Step 1: Load GIAS and Ofsted through the changed taps** ```bash mkdir -p "$SCRATCH/e2e_h4" && cd "$SCRATCH/e2e_h4" echo "drop table if exists raw.gias_establishments, raw.gias_links, raw.ofsted_inspections, raw.ofsted_fe_providers, raw.ofsted_fe_inspections cascade;" | "$SCRATCH/dbt_local/sql.sh" SOCK=$(grep -o 'PG_HOST=[^ ]*' "$SCRATCH/dbt_local/dbt.sh" | cut -d= -f2) echo "{\"sqlalchemy_url\": \"postgresql+psycopg://postgres@/school_compare?host=$SOCK\", \"default_target_schema\": \"raw\"}" > target.json echo '{}' > tap.json cat > run_load.sh <<'EOF' #!/bin/bash R=/Users/tudor/projects/school_compare/pipeline/plugins/extractors for tap in gias ofsted; do uv run --no-project --with-editable "$R/tap-uk-$tap" tap-uk-$tap --config tap.json 2> tap-$tap.log \ | uv run --no-project --with meltanolabs-target-postgres target-postgres --config target.json > target-$tap.log 2>&1 echo "$tap exit ${PIPESTATUS[*]}" done EOF bash run_load.sh ``` Expected: `gias exit 0 0` and `ofsted exit 0 0`. - [ ] **Step 2: Build and test everything this PR touches** Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_gias_ukprn stg_ofsted_inspections stg_ofsted_fe_providers stg_ofsted_fe_inspections int_ofsted_fe_latest int_ofsted_latest int_ofsted_listed fact_ofsted_latest dim_school assert_ofsted_fe_never_overrides_school assert_ofsted_current_grade_consistent` Expected: exit 0; all unit, schema and data tests PASS. - [ ] **Step 3: Check the values** Through `"$SCRATCH/dbt_local/sql.sh" <<'SQL' … SQL`: ```sql select urn, latest_visit_date, latest_visit_kind, current_grade, current_grade_date, rc_inspection_date, rc_safeguarding_met, framework from marts.fact_ofsted_latest where urn in (145175, 143526, 130457) order by urn; select count(*) from marts.fact_ofsted_latest where framework = 'Further education and skills'; select count(*) filter (where inspectorate = 'ISI') as isi, count(*) filter (where ofsted_listed) as listed from marts.dim_school; select d.urn, d.ofsted_listed, f.urn is not null as has_record from marts.dim_school d left join marts.fact_ofsted_latest f using (urn) where d.urn in (150031, 136083); select count(*) as fe_rows_for_listed_schools from marts.fact_ofsted_latest f join marts.dim_school d using (urn) where f.framework = 'Further education and skills'; ``` Expected: 145175 graded 2 on 2022-12-06; 143526 report card 2026-05-12 with safeguarding true; 130457 graded 2 on 2024-01-30; about 490 FE rows for listed schools (443 + 45 + 2 as at 31 August 2026); about 1,341 ISI schools; 150031 listed with no record (unless Ofsted has inspected it since); 136083 not listed, no record. Record any drift from these numbers and its cause (newer MI files) in the PR description. - [ ] **Step 4: Run CI's Python suite** Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests pipeline/tests scripts/ci/tests -q` → all pass. ### Task 9: Open PR 1 - [ ] **Step 1:** `git status --short` → only `.DS_Store`. - [ ] **Step 2:** `git push -u origin fix/ofsted-inspectorates-pipeline`. - [ ] **Step 3:** Open the PR through the Gitea API with basic auth from `git credential fill` (host `privaterepo.sitaru.org`; never print the secret): `POST /api/v1/repos/tudor/school_compare/pulls`, head `fix/ofsted-inspectorates-pipeline`, base `main`, title `fix(pipeline): inspectorate, Ofsted listing and further-education status (H4, part 1 of 2)`. Body: what changed; Task 8's numbers; **After merge:** run the daily DAG (GIAS fields) and then `school_data_monthly_ofsted` on staging — the monthly build reads `raw.gias_establishments."UKPRN"`, which exists only after a daily run with this PR; then merge PR 2; production the same order before PR 2 is promoted; ends with the Claude Code line. - [ ] **Step 4:** Report the PR URL. Do not merge. --- ## PR 2 — site After PR 1 merges: `git fetch origin && git switch -c fix/ofsted-inspectorates-site origin/main`. ### Task 10: Backend payloads, optional columns, Ofsted page link **Files:** - Modify: `backend/schemas.py` (`SCHOOL_COLUMNS`), `backend/data_loader.py` (`_MAIN_QUERY`, `_EXTRA_OPTIONAL_COLUMNS`, the fallback loop), `backend/app.py` (detail and compare `school_info`), `backend/ofsted_codes.py` (`ofsted_page_url`) - Test: create `backend/tests/test_inspection_status_payload.py`; modify `backend/tests/test_ofsted_codes.py:45`, `backend/tests/test_supplementary_enrichment.py:36` **Interfaces:** - Produces: list rows, detail `school_info` and compare `school_info` carry `inspectorate: str | null` and `ofsted_listed: bool | null`; `ofsted_page_url(urn)` returns `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}`. - [ ] **Step 1: Write the failing tests** `backend/tests/test_inspection_status_payload.py`: ```python """Audit H4: every payload that feeds an Ofsted badge carries the inspectorate GIAS names and whether Ofsted's own data lists the school, so the site never says "Not yet inspected" on the strength of missing data.""" import pandas as pd import pytest from fastapi.testclient import TestClient def _df(): base = dict(year=202425, local_authority='Cambridgeshire', local_authority_code=873, postcode='CB1 1AA', latitude=52.2, longitude=0.12, address='1 High St', status='Open', town='Cambridge') return pd.DataFrame([ dict(base, urn=136083, school_name='Abbey College Cambridge', phase='Secondary', school_type='Other independent school', inspectorate='ISI', ofsted_listed=False), dict(base, urn=150031, school_name='Abbas and Templecombe Church School', phase='Primary', school_type='Academy converter', inspectorate='Not applicable', ofsted_listed=True), ]) @pytest.fixture() def client(monkeypatch): from backend import app as app_module from backend import database as database_module def _no_db(): raise RuntimeError('no database in this test') monkeypatch.setattr(app_module, 'load_latest_school_data', _df) monkeypatch.setattr(app_module, 'load_school_data', _df) monkeypatch.setattr(app_module, 'get_supplementary_data', lambda db, urn: {}) monkeypatch.setattr(app_module, '_place_registry', None) monkeypatch.setattr(database_module, 'SessionLocal', _no_db) return TestClient(app_module.app, raise_server_exceptions=False) def test_the_list_carries_inspectorate_and_listing(client): resp = client.get('/api/schools?page_size=50') assert resp.status_code == 200, resp.text by_urn = {s['urn']: s for s in resp.json()['schools']} assert by_urn[136083]['inspectorate'] == 'ISI' assert by_urn[150031]['ofsted_listed'] is True def test_the_school_page_carries_them(client): info = client.get('/api/schools/136083').json()['school_info'] assert info['inspectorate'] == 'ISI' assert info['ofsted_listed'] is False def test_compare_carries_them(client): body = client.get('/api/compare?urns=136083,150031').json() assert body['comparison']['150031']['school_info']['ofsted_listed'] is True assert body['comparison']['136083']['school_info']['inspectorate'] == 'ISI' def test_a_mart_built_before_the_columns_still_loads(): # Missing columns make the main query retry with NULLs (data_loader). from backend import data_loader assert {'inspectorate', 'ofsted_listed'} <= set(data_loader._EXTRA_OPTIONAL_COLUMNS) fallback = str(data_loader._MAIN_QUERY_NO_EXTRA_COLS) assert 'NULL AS inspectorate,' in fallback and 'NULL AS ofsted_listed,' in fallback ``` (`/api/compare` returns `{"comparison": {: {"school_info": …}}}`.) Change `backend/tests/test_ofsted_codes.py:45` and `backend/tests/test_supplementary_enrichment.py:36` to expect `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/` (keep each test's URN). - [ ] **Step 2: Run them to see them fail** Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests/test_inspection_status_payload.py backend/tests/test_ofsted_codes.py backend/tests/test_supplementary_enrichment.py -q` Expected: the four new tests fail (`KeyError: 'inspectorate'`; the optional-columns assertion); the two link tests fail on the old `provider/21` URL. - [ ] **Step 3: Implement** - `backend/schemas.py`, `SCHOOL_COLUMNS`: after `"ofsted_rc_date",` add `"inspectorate",` and `"ofsted_listed",`. - `backend/data_loader.py`, `_MAIN_QUERY`: after ` s.nursery_provision,` add ` s.inspectorate,` and ` s.ofsted_listed,`. Add `"inspectorate",` and `"ofsted_listed",` to `_EXTRA_OPTIONAL_COLUMNS`, and `("inspectorate", "s"),` and `("ofsted_listed", "s"),` to the tuple of `(column, alias)` pairs the fallback loop replaces. Extend the comment above `_EXTRA_OPTIONAL_COLUMNS`: `… parliamentary constituency, and the inspectorate / Ofsted-listing columns added for audit H4) …`. - `backend/app.py`, detail `school_info` (after `"nursery_provision": latest.get("nursery_provision"),`): ```python # What the Ofsted badge may say when there is no inspection record # (audit H4): the inspectorate GIAS names, and whether Ofsted's own # data lists the school at all. "inspectorate": latest.get("inspectorate"), "ofsted_listed": latest.get("ofsted_listed"), ``` - `backend/app.py`, compare `school_info` (after `"has_sixth_form": convert_to_native(latest.get("has_sixth_form")),`): ```python # Audit H4: what compare may say when there is no inspection record. "inspectorate": convert_to_native(latest.get("inspectorate")), "ofsted_listed": convert_to_native(latest.get("ofsted_listed")), ``` - `backend/ofsted_codes.py`: ```python def ofsted_page_url(urn: int) -> str: """The school's page on ofsted.gov.uk (all its reports live there — we never deep-link an individual report). Ofsted's provider pages sit under a category (21 primary, 23 secondary, 25 special, 40 FE colleges, …); this address redirects to the right one, where provider/21 404s for every non-primary school.""" return f"https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}" ``` - [ ] **Step 4: Run the tests and the backend suite** Run: the Step 2 command → all pass. Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests -q` → all pass. - [ ] **Step 5: Commit** ```bash git add backend/schemas.py backend/data_loader.py backend/app.py backend/ofsted_codes.py backend/tests/test_inspection_status_payload.py backend/tests/test_ofsted_codes.py backend/tests/test_supplementary_enrichment.py git commit -m "fix(api): inspectorate and Ofsted listing in payloads; Ofsted links that resolve (H4)" ``` ### Task 11: The badge rule and its callers **Files:** - Modify: `nextjs-app/lib/types.ts` (`School`, `OfstedLatestVisit`), `nextjs-app/lib/ofstedStatus.ts`, `nextjs-app/lib/utils.ts` (`buildOfstedListBadge`) - Modify: `nextjs-app/components/SchoolRow.tsx`, `SecondarySchoolRow.tsx`, `LeafletMapInner.tsx` - Test: `nextjs-app/__tests__/lib/utils.test.ts`, `__tests__/lib/ofstedStatus.test.ts`, `__tests__/components/SchoolRow.test.tsx` **Interfaces:** - Produces: `School.inspectorate?: string | null`, `School.ofsted_listed?: boolean | null`; `OfstedLatestVisit.kind` adds `'short'`; `FE_FRAMEWORK = 'Further education and skills'`; `hasReportCard(ofsted: Pick): boolean`; `buildOfstedListBadge(school): OfstedListBadge | null`. - [ ] **Step 1: Write the failing tests** In `__tests__/lib/utils.test.ts`, inside `describe('buildOfstedListBadge', …)`: replace the test `returns pending badge when no grade and no inspection on record` with the block below, and in every other test in that describe use `buildOfstedListBadge(…)!` (the function may now return null): ```ts it('says "Not yet inspected" only when Ofsted lists the school uninspected (audit H4)', () => { const badge = buildOfstedListBadge({ ofsted_grade: null, ofsted_date: null, ofsted_listed: true }); expect(badge).toEqual({ label: 'Not yet inspected', cssClass: 'ofstedPending' }); }); it('names ISI for a school GIAS says ISI inspects', () => { expect(buildOfstedListBadge({ ofsted_date: null, inspectorate: 'ISI', ofsted_listed: false })) .toEqual({ label: 'Inspected by ISI', cssClass: 'ofstedPending' }); }); it('an Ofsted record wins over ISI', () => { expect(buildOfstedListBadge({ ofsted_grade: 2, ofsted_grade_date: '2022-03-01', inspectorate: 'ISI' })!.label) .toBe('Good · 2022'); }); it('gives no badge without any inspection data', () => { expect(buildOfstedListBadge({ ofsted_grade: null, ofsted_date: null, ofsted_framework: null })).toBeNull(); expect(buildOfstedListBadge({ ofsted_listed: null, inspectorate: 'Not applicable' })).toBeNull(); }); ``` In `__tests__/lib/ofstedStatus.test.ts` add: ```ts it('names a short inspection as Ofsted does', () => { expect(latestVisitLine({ date: '2023-02-01', kind: 'short', outcome: null })).toBe('Short inspection, 1 February 2023'); }); it('treats a report card with only a safeguarding result as a report card (FE)', () => { expect(hasReportCard({ rc_inspection_date: '2026-05-12', report_card: {} })).toBe(true); expect(hasReportCard({ rc_inspection_date: null, report_card: {} })).toBe(false); }); ``` (change the file's first line to `import { gradeSourceLine, hasReportCard, latestVisitLine, showLatestVisitLine } from '@/lib/ofstedStatus';`). In `__tests__/components/SchoolRow.test.tsx` add (the file's school fixture is `base`): ```tsx it('renders no Ofsted badge for a school with no inspection data (audit H4)', () => { const { container } = render(); expect(container.textContent).not.toMatch(/Not yet inspected|Inspected/); }); ``` - [ ] **Step 2: Run them to see them fail** Run: `cd nextjs-app && npx jest __tests__/lib/utils.test.ts __tests__/lib/ofstedStatus.test.ts __tests__/components/SchoolRow.test.tsx` Expected: the new tests fail (no ISI rule; `hasReportCard` missing; "Not yet inspected" for no data). - [ ] **Step 3: Implement** `lib/types.ts`: in `School`, after `ofsted_framework?: string | null;` add ```ts /** The inspectorate GIAS names ("ISI", "Ofsted", "Not applicable"). Audit H4. */ inspectorate?: string | null; /** Whether Ofsted's own data lists the school, inspected or not. Audit H4. */ ofsted_listed?: boolean | null; ``` and in `OfstedLatestVisit` change `kind` to `'report_card' | 'graded' | 'ungraded' | 'short';`. `lib/ofstedStatus.ts`: add `short: 'Short inspection',` to `VISIT_KIND`, and ```ts /** fact_ofsted_latest.framework for Ofsted's further education and skills records (audit H4). */ export const FE_FRAMEWORK = 'Further education and skills'; /** * A report card is in force. Checked by its date as well as its area labels: * an FE report card carries a safeguarding result and no school areas. */ export function hasReportCard( ofsted: Pick, ): boolean { return !!ofsted.rc_inspection_date || Object.keys(ofsted.report_card ?? {}).length > 0; } ``` `lib/utils.ts`, `buildOfstedListBadge`: add `inspectorate?: string | null; ofsted_listed?: boolean | null;` to the parameter type, change the return type to `OfstedListBadge | null`, replace the last bullet of the doc comment with ```ts * - No inspection on record (audit H4): "Inspected by ISI" when GIAS names * ISI; "Not yet inspected" only when Ofsted's own data lists the school; * otherwise no badge, never a claim the data cannot support. ``` and replace `return { label: 'Not yet inspected', cssClass: 'ofstedPending' };` with ```ts if (school.inspectorate === 'ISI') return { label: 'Inspected by ISI', cssClass: 'ofstedPending' }; if (school.ofsted_listed) return { label: 'Not yet inspected', cssClass: 'ofstedPending' }; return null; ``` `components/SchoolRow.tsx` and `components/SecondarySchoolRow.tsx`: wrap the badge span in `{ofstedBadge && ( … )}`. `components/LeafletMapInner.tsx`, `cardHtml`: replace the tags line with ```ts
${badge ? `${escapeHtml(badge.label)}` : ''}${type}
``` - [ ] **Step 4: Run the tests, the suite and the type check** Run: `npx jest __tests__/lib/utils.test.ts __tests__/lib/ofstedStatus.test.ts __tests__/components/SchoolRow.test.tsx` → pass. Run: `npm run typecheck && npm test` → all pass (fix any other caller the type check names by handling `null`, never by asserting it away). - [ ] **Step 5: Commit** ```bash git add nextjs-app/lib/types.ts nextjs-app/lib/ofstedStatus.ts nextjs-app/lib/utils.ts nextjs-app/components/SchoolRow.tsx nextjs-app/components/SecondarySchoolRow.tsx nextjs-app/components/LeafletMapInner.tsx nextjs-app/__tests__/lib/utils.test.ts nextjs-app/__tests__/lib/ofstedStatus.test.ts nextjs-app/__tests__/components/SchoolRow.test.tsx git commit -m "fix(search): no \"Not yet inspected\" without Ofsted's word; ISI badge (H4)" ``` ### Task 12: School page and compare **Files:** - Create: `nextjs-app/components/school/InspectionNotice.tsx`; test `nextjs-app/__tests__/components/InspectionNotice.test.tsx` - Modify: `nextjs-app/components/school/OfstedSection.tsx`, `PrimarySchoolSections.tsx`, `SecondarySchoolSections.tsx`, `nextjs-app/lib/compareLogic.ts`, `nextjs-app/components/compare/CompareOfsted.tsx`, `CompareAtAGlance.tsx` - Test: `nextjs-app/__tests__/components/OfstedSection.test.tsx`, `__tests__/components/CompareOfsted.test.tsx`, `__tests__/lib/compareLogic.test.ts` **Interfaces:** - Consumes: `FE_FRAMEWORK`, `hasReportCard`, `School.inspectorate`, `School.ofsted_listed` (Task 11). - Produces: `InspectionNotice({ school }: { school: Pick })` — renders the ISI or "Not yet inspected" section, or nothing; `noInspectionLabel(school): string` in `lib/compareLogic.ts`. - [ ] **Step 1: Write the failing tests** `__tests__/components/InspectionNotice.test.tsx`: ```tsx import { render, screen } from '@testing-library/react'; import { InspectionNotice } from '@/components/school/InspectionNotice'; it('names ISI and links its reports for an ISI school (audit H4)', () => { render(); expect(screen.getByText('Inspected by the Independent Schools Inspectorate (ISI)')).toBeInTheDocument(); expect(screen.getByRole('link', { name: /ISI reports/ })).toHaveAttribute('href', 'https://www.isi.net/reports/'); }); it('says "Not yet inspected" for a school Ofsted lists without an inspection', () => { render(); expect(screen.getByText('Not yet inspected')).toBeInTheDocument(); expect(screen.getByRole('link', { name: /Ofsted reports/ })).toHaveAttribute( 'href', 'https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/150031'); }); it('renders nothing without inspection data, including an unknown listing', () => { const { container } = render(); expect(container).toBeEmptyDOMElement(); }); ``` In `__tests__/components/OfstedSection.test.tsx` add (uses the file's `renderSection`): ```tsx it('labels a further education record and prints its short inspection (audit H4)', () => { renderSection({ framework: 'Further education and skills', current_grade: { grade: 2, date: '2019-03-05', basis: 'graded' }, latest_visit: { date: '2023-02-01', kind: 'short', outcome: null }, }); expect(screen.getByText('Further education and skills')).toBeInTheDocument(); expect(screen.getByText('Good')).toBeInTheDocument(); expect(screen.getByText('Latest visit: Short inspection, 1 February 2023')).toBeInTheDocument(); }); ``` In `__tests__/lib/compareLogic.test.ts` add: ```ts it('shows an FE report card with only a safeguarding result as a report card (audit H4)', () => { const display = ofstedDisplay(ofsted({ rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, report_card: {} })); expect(display.kind).toBe('report_card'); }); it('words the no-record compare cell from the evidence', () => { expect(noInspectionLabel({ inspectorate: 'ISI' })).toBe('Inspected by ISI'); expect(noInspectionLabel({ ofsted_listed: true })).toBe('Not yet inspected'); expect(noInspectionLabel({ ofsted_listed: null })).toBe('No inspection in our dataset'); }); ``` (`ofsted(partial)` is the file's existing fixture helper; add `noInspectionLabel` to its import from `@/lib/compareLogic`.) In `__tests__/components/CompareOfsted.test.tsx`, inside `describe('CompareOfsted', …)`: ```tsx it('words a school without an Ofsted record from the evidence (audit H4)', () => { const isi = { ...school(4, 'ISI School'), inspectorate: 'ISI' } as School; const listed = { ...school(5, 'New Academy'), ofsted_listed: true } as School; render(); expect(screen.getAllByText('Inspected by ISI').length).toBeGreaterThan(0); expect(screen.getAllByText('Not yet inspected').length).toBeGreaterThan(0); expect(screen.queryByText(/No inspection outcome in our dataset/)).toBeNull(); }); ``` - [ ] **Step 2: Run them to see them fail** Run: `npx jest __tests__/components/InspectionNotice.test.tsx __tests__/components/OfstedSection.test.tsx __tests__/lib/compareLogic.test.ts __tests__/components/CompareOfsted.test.tsx` Expected: the new tests fail (component and helper missing; no FE label; FE report card read as `no_overall_grade`). - [ ] **Step 3: Implement** `components/school/InspectionNotice.tsx`: ```tsx /** * InspectionNotice — the school page's inspection section when there is no * Ofsted inspection record (audit H4): the inspectorate GIAS names (ISI), or * Ofsted's own "not yet inspected" when its data lists the school. Nothing * otherwise: no claim the data cannot support. Server component. */ import type { School } from '@/lib/types'; import { Section, sectionStyles as styles } from './sectionShared'; const OFSTED_REPORTS = 'https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/'; const ISI_REPORTS = 'https://www.isi.net/reports/'; export function InspectionNotice({ school }: { school: Pick }) { if (school.inspectorate === 'ISI') { return (

Inspection ISI reports ↗

Inspected by the Independent Schools Inspectorate (ISI)

); } if (school.ofsted_listed) { return (

Ofsted Rating Ofsted reports ↗

Not yet inspected
); } return null; } ``` `components/school/OfstedSection.tsx`: import `FE_FRAMEWORK` from `@/lib/ofstedStatus`; directly after the closing `` add ```tsx {ofsted.framework === FE_FRAMEWORK && (

{FE_FRAMEWORK}

)} ``` `PrimarySchoolSections.tsx` and `SecondarySchoolSections.tsx`: import `hasReportCard` from `@/lib/ofstedStatus` and `InspectionNotice` from `./InspectionNotice`; set `const isReportCard = !!ofsted && hasReportCard(ofsted);`; change `{ofsted && ( )}` to `{ofsted ? ( ) : }`. `lib/compareLogic.ts`: import `hasReportCard` from `./ofstedStatus`; in `ofstedDisplay` replace the report-card condition with `if (hasReportCard(ofsted)) {`; add ```ts /** * The compare cell when a school has no Ofsted record (audit H4): ISI per GIAS, * Ofsted's own "not yet inspected", or plainly that we hold nothing. */ export function noInspectionLabel(school: Pick): string { if (school.inspectorate === 'ISI') return 'Inspected by ISI'; if (school.ofsted_listed) return 'Not yet inspected'; return 'No inspection in our dataset'; } ``` (import `School` from `./types` if not already imported). `components/compare/CompareOfsted.tsx`: give `ResultCell` a `school` prop (`school: School`) and return `{noInspectionLabel(school)}` for `display.kind === 'none'`; pass `school={data[String(school.urn)]?.school_info ?? school}` where it is rendered; in the judgement-detail cell replace `No inspection in our dataset` with `{noInspectionLabel(data[String(school.urn)]?.school_info ?? school)}`; change the `Ofsted page` fallback URL to `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/${school.urn}`. `components/compare/CompareAtAGlance.tsx`: replace `No inspection in our dataset` with `{noInspectionLabel(data[String(school.urn)]?.school_info ?? school)}`. - [ ] **Step 4: Run the tests, the suite and the type check** Run: the Step 2 command → pass. Run: `npm run typecheck && npm test` → all pass. - [ ] **Step 5: Commit** ```bash git add nextjs-app/components/school/InspectionNotice.tsx nextjs-app/components/school/OfstedSection.tsx nextjs-app/components/school/PrimarySchoolSections.tsx nextjs-app/components/school/SecondarySchoolSections.tsx nextjs-app/lib/compareLogic.ts nextjs-app/components/compare/CompareOfsted.tsx nextjs-app/components/compare/CompareAtAGlance.tsx nextjs-app/__tests__/components/InspectionNotice.test.tsx nextjs-app/__tests__/components/OfstedSection.test.tsx nextjs-app/__tests__/lib/compareLogic.test.ts nextjs-app/__tests__/components/CompareOfsted.test.tsx git commit -m "fix(school, compare): ISI, not-yet-inspected and further-education states (H4)" ``` ### Task 13: E2E journeys **Files:** - Modify: `e2e/tests/journeys.spec.ts` (after the Ofsted journeys, `test('a grade confirmed at an ungraded visit says so and is dated by it', …)`) - [ ] **Step 1: Add the journeys** ```ts test('a further education college shows its Ofsted result, not "Not yet inspected"', async ({ page }) => { // Queen Mary's College (145175) is inspected under Ofsted's further education // framework, which the site did not load: its badge said "Not yet inspected" // (audit H4). Its result may change; the label must never be absent or wrong. const res = await page.request.get('/api/schools/145175'); expect(res.ok()).toBeTruthy(); await page.goto('/school/145175'); const ofsted = page.locator('#ofsted'); await expect(ofsted).toBeVisible(); await expect(ofsted.getByText('Further education and skills')).toBeVisible(); await expect(ofsted).not.toContainText('Not yet inspected'); }); test('an ISI-inspected school says so instead of "Not yet inspected"', async ({ page }) => { const res = await page.request.get('/api/schools?school_type=independent&phase=secondary&page_size=100'); expect(res.ok()).toBeTruthy(); const school = ((await res.json()).schools ?? []).find( (s: { inspectorate?: string | null; ofsted_date?: string | null; ofsted_rc_date?: string | null }) => s.inspectorate === 'ISI' && !s.ofsted_date && !s.ofsted_rc_date); test.skip(!school, 'no ISI school without an Ofsted record here'); await page.goto(`/school/${school.urn}`); await expect(page.getByText('Inspected by the Independent Schools Inspectorate (ISI)')).toBeVisible(); await expect(page.locator('#ofsted')).not.toContainText('Not yet inspected'); }); test('"Not yet inspected" appears only for a school Ofsted lists without an inspection', async ({ page }) => { const res = await page.request.get('/api/schools?phase=primary&page_size=500'); expect(res.ok()).toBeTruthy(); type Row = { urn: number; ofsted_listed?: boolean | null; ofsted_date?: string | null; ofsted_rc_date?: string | null; ofsted_grade?: number | null }; const schools: Row[] = (await res.json()).schools ?? []; const unlisted = schools.find(s => !s.ofsted_listed && !s.ofsted_date && !s.ofsted_rc_date && !s.ofsted_grade); const listed = schools.find(s => s.ofsted_listed && !s.ofsted_date && !s.ofsted_rc_date && !s.ofsted_grade); test.skip(!listed, 'no listed, uninspected primary on this page'); await page.goto(`/school/${listed!.urn}`); await expect(page.locator('#ofsted').getByText('Not yet inspected')).toBeVisible(); if (unlisted) { await page.goto(`/school/${unlisted.urn}`); await expect(page.getByText('Not yet inspected')).toHaveCount(0); } }); ``` - [ ] **Step 2: Check the file compiles and lists the tests** Run: `cd e2e && npx playwright test --list tests/journeys.spec.ts -g "further education college|ISI-inspected|Ofsted lists without"` → the three tests listed. - [ ] **Step 3: Run them against staging to see them fail** Only once PR 1 has merged and the daily and monthly Ofsted DAGs have run on staging: `cd e2e && BASE_URL=https://stx.schoolcompare.co.uk npx playwright test tests/journeys.spec.ts -g "further education college|ISI-inspected|Ofsted lists without" --reporter=line --retries=0` Expected: all three FAIL for the right reason (staging still runs the old site: no FE label, no ISI section, the old badge rule), or skip where the old API lacks the fields; record which. If the DAGs have not run, record that and rely on the post-merge gate. - [ ] **Step 4: Commit** ```bash git add e2e/tests/journeys.spec.ts git commit -m "test(e2e): further education, ISI and not-yet-inspected states (H4)" ``` ### Task 14: Verify and open PR 2 - [ ] **Step 1:** `cd nextjs-app && npm run typecheck && npm test`; from the repo root `uv run --no-project --with-requirements requirements.txt python -c "from backend.app import app; print('backend imports OK')"` and `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests pipeline/tests scripts/ci/tests -q` → all pass. - [ ] **Step 2:** `git status --short` → only `.DS_Store`. `git push -u origin fix/ofsted-inspectorates-site`. - [ ] **Step 3:** Open the PR through the Gitea API (as Task 9): head `fix/ofsted-inspectorates-site`, title `fix(site): ISI, Ofsted-listed and further-education inspection states (H4, part 2 of 2)`. Body: what changed, including the Ofsted page link fix (`provider/21` 404s for every non-primary school); Task 13 Step 3's result; **merge only after the daily and monthly Ofsted DAGs have run on staging with PR 1; promote only after they have run on production** (before that, the fallback query gives no badge for the ISI and listed groups); ends with the Claude Code line. - [ ] **Step 4:** Report the PR URL. Do not merge. --- ## After both PRs are live in production (Tudor promotes) Validate against GIAS and Ofsted's school and FE MI: every "Not yet inspected" belongs to a school Ofsted's data lists without an inspection; every "Inspected by ISI" matches GIAS; every FE grade, date and report card matches the FE files; schools with no data show no badge; the compare page's Ofsted link resolves for a secondary and a special school. Then mark H4 resolved in the audit report artifact.