school_compare/docs/superpowers/plans/2026-10-13-inspection-status-beyond-ofsted-schools.md

Inspection Status Beyond Ofsted's School Data Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Stop labelling schools "Not yet inspected" unless Ofsted's own data lists them uninspected, and show ISI and Ofsted further-education results where the data exists (audit H4).

Architecture: PR 1 adds two GIAS fields, two Ofsted further-education (FE) streams, an FE status model folded into int_ofsted_latest / fact_ofsted_latest (framework "Further education and skills"), an int_ofsted_listed model, and dim_school.inspectorate / dim_school.ofsted_listed. PR 2, after the daily and monthly Ofsted DAGs have run on PR 1, publishes the two fields, changes the badge rule, adds the ISI / "Not yet inspected" school-page states and the FE label, updates compare, and fixes the Ofsted page link.

Tech Stack: Singer SDK taps (Python, pandas, requests), meltanolabs target-postgres, dbt-postgres ~1.10 (YAML unit tests, singular tests), FastAPI + pandas + SQLAlchemy, Next.js / React / TypeScript, Jest + Testing Library, Playwright.

Spec: docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md (read its "Amendments" section: UKPRN comes from a new stg_gias_ukprn view; report cards are detected by rc_inspection_date; ofsted_page_url is fixed).

Global Constraints

• Badge order (search rows and map card): report card → "Report Card · year"; grade in force → grade word · year; inspected, no grade → "Inspected · year"; GIAS inspectorate ISI → "Inspected by ISI" (grey, css class ofstedPending); ofsted_listed → "Not yet inspected" (grey, ofstedPending); otherwise no badge.
• FE framework value, exactly: Further education and skills.
• FE rule: a report card on or after the previous-framework inspection (or with none) wins and leaves no grade in force; otherwise a previous-framework grade 1–4 is the current grade, dated by that inspection's first day, basis graded; latest visit is the later of that inspection (graded) and any short inspection (short); grade - with no report card gives no FE row.
• A school-framework record always wins over an FE record for the same URN.
• FE providers match a URN by Provider URN first, then by Provider UKPRN = GIAS UKPRN.
• The Ofsted models must never ref stg_gias_establishments (the daily build's stg_gias_establishments+ would select them); they read UKPRN from stg_gias_ukprn, a view on the raw GIAS source.
• Copy: Ofsted's and GIAS's own terms ("Short inspection", "Not yet inspected", "Inspected by the Independent Schools Inspectorate (ISI)", "Further education and skills"); no derived sentences.
• ISI link: https://www.isi.net/reports/. Ofsted page link: https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn} (redirects to the right category; provider/21/{urn} 404s for every non-primary school).
• PR 2 must not merge before the daily DAG (GIAS fields) and school_data_monthly_ofsted have run on staging with PR 1, nor be promoted before they have run on production. Promotion is Tudor's; never trigger it.
• Never push to main. Commits end with the Co-Authored-By trailer the executing session's harness specifies; PR bodies end with the Claude Code line. Do not start a local app server.

Review Focus

1. A school whose GIAS inspectorate is ISI and that Ofsted also lists (dual registration) must show "Inspected by ISI" only when there is no Ofsted inspection record; an Ofsted record always wins. Pinned in Task 11 (an Ofsted record wins over ISI).
2. An FE provider whose report card predates its previous-framework inspection (bad data) must not produce a row with both a grade and a report card (that breaks assert_ofsted_current_grade_consistent). Pinned in Task 5 (int_ofsted_fe_report_card_older_than_grade_is_ignored).
3. A UKPRN shared by an FE provider and a URN that Ofsted also lists under its own URN must take the URN match. Pinned in Task 5 (int_ofsted_fe_prefers_the_urn_match).
4. The FE "as at" file missing from the newest academic-year page (a new page with no file yet) must fall back to the previous page, not emit nothing. Pinned in Task 3 (test_providers_fall_back_to_the_previous_page).
5. A school-page or compare render for a school with ofsted_listed null (marts built before PR 1) must behave as "not listed": no "Not yet inspected". Pinned in Task 11 (null case) and Task 12.

File Map

PR 1 — branch fix/ofsted-inspectorates-pipeline (exists; spec 5f7528b)

• Modify pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py (two fields).
• Create pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py (page/file selection, CSV reading, column maps — pandas-only).
• Modify pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py (two streams).
• Create pipeline/tests/test_gias_declared_fields.py, pipeline/tests/test_ofsted_fe_mi.py.
• Modify pipeline/transform/models/staging/stg_gias_establishments.sql, _stg_sources.yml.
• Create pipeline/transform/models/staging/stg_gias_ukprn.sql, stg_ofsted_fe_providers.sql, stg_ofsted_fe_inspections.sql, stg_ofsted_fe.yml.
• Create pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql, int_ofsted_fe_latest.yml, int_ofsted_listed.sql, int_ofsted_listed.yml.
• Modify pipeline/transform/models/intermediate/int_ofsted_latest.sql, int_ofsted_latest.yml.
• Modify pipeline/transform/models/marts/dim_school.sql, dim_school.yml, _marts_schema.yml.
• Create pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql.
• Modify pipeline/dags/school_data_pipeline.py (daily and monthly selectors), pipeline/tests/test_dag_selectors.py (OPTIONAL_PARENTS).

PR 2 — branch fix/ofsted-inspectorates-site (from origin/main after PR 1 merges; touches no PR 1 file)

• Modify backend/schemas.py (SCHOOL_COLUMNS), backend/data_loader.py (main query, optional columns), backend/app.py (detail and compare school_info), backend/ofsted_codes.py (ofsted_page_url).
• Create backend/tests/test_inspection_status_payload.py; modify backend/tests/test_ofsted_codes.py, backend/tests/test_supplementary_enrichment.py.
• Modify nextjs-app/lib/types.ts, nextjs-app/lib/ofstedStatus.ts, nextjs-app/lib/utils.ts, nextjs-app/lib/compareLogic.ts.
• Modify nextjs-app/components/SchoolRow.tsx, SecondarySchoolRow.tsx, LeafletMapInner.tsx, school/OfstedSection.tsx, school/PrimarySchoolSections.tsx, school/SecondarySchoolSections.tsx, compare/CompareOfsted.tsx, compare/CompareAtAGlance.tsx.
• Create nextjs-app/components/school/InspectionNotice.tsx.
• Modify tests: nextjs-app/__tests__/lib/utils.test.ts, __tests__/lib/ofstedStatus.test.ts, __tests__/lib/compareLogic.test.ts, __tests__/components/OfstedSection.test.tsx, __tests__/components/CompareOfsted.test.tsx, __tests__/components/SchoolRow.test.tsx; create __tests__/components/InspectionNotice.test.tsx.
• Modify e2e/tests/journeys.spec.ts.

Commands used throughout. $SCRATCH is the session scratchpad directory. Python tests run in CI's environment:

uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest <paths> -q

Local tap commands use --with-editable (a plain --with ./path reuses a cached build of the unchanged version and ignores edits). dbt runs through $SCRATCH/dbt_local/dbt.sh (Task 1).

---

PR 1 — pipeline

Task 1: Local dbt harness

$SCRATCH/dbt_local/pgdata survives from earlier work; its helper scripts do not.

Files: none in the repo.

• Step 1: Recreate the helpers

mkdir -p "$SCRATCH/dbt_local" && cd "$SCRATCH/dbt_local"
cat > start_pg.py <<'EOF'
import pgserver, sys
srv = pgserver.get_server(sys.argv[1], cleanup_mode=None)
print(srv.get_uri())
if "school_compare" not in srv.psql("select datname from pg_database;"):
    srv.psql("create database school_compare;")
EOF
uv run --no-project --with pgserver python start_pg.py "$SCRATCH/dbt_local/pgdata"

Expected: a URI ending ?host=<socket dir>. Then, with that socket dir:

cat > "$SCRATCH/dbt_local/dbt.sh" <<EOF
#!/bin/bash
# Run dbt against the throwaway pgserver database. Usage: dbt.sh <dbt args...>
export PG_HOST=<socket dir> PG_PORT=5432 PG_USER=postgres PG_PASSWORD= PG_DATABASE=school_compare
D=$SCRATCH/dbt_local
cd /Users/tudor/projects/school_compare/pipeline/transform
uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main "\$@" --profiles-dir . --target dev --target-path \$D/target --log-path \$D/logs > \$D/last.log 2>&1
status=\$?
sed 's/\x1b\[[0-9;]*m//g' \$D/last.log | grep -E "PASS|FAIL|ERROR|WARN |OK created|Done\.|Error|Compilation|Invalid"
exit \$status
EOF
cat > "$SCRATCH/dbt_local/sql.sh" <<'EOF'
#!/bin/bash
# Run SQL from stdin against the local school_compare database; print the output.
D=$(cd "$(dirname "$0")" && pwd)
SQL=$(cat) D="$D" uv run --no-project --with pgserver python -c '
import os, pgserver
srv = pgserver.get_server(os.environ["D"] + "/pgdata", cleanup_mode=None)
print(srv.psql("\\c school_compare\n" + os.environ["SQL"]))'
EOF
cat > "$SCRATCH/dbt_local/raw_tables.py" <<'EOF'
"""Create raw.<stream> with a text column per declared field (what target-postgres loads).
Usage: raw_tables.py <module>:<TapClass> <stream> [<stream> ...]"""
import importlib, os, sys, pgserver
module, cls = sys.argv[1].split(":")
tap = getattr(importlib.import_module(module), cls)(config={})
wanted = set(sys.argv[2:])
stmts = ["create schema if not exists raw;"]
for stream in tap.streams.values():
    if stream.name in wanted:
        cols = ", ".join(f'"{c}" text' for c in stream.schema["properties"])
        stmts.append(f'drop table if exists raw.{stream.name} cascade; create table raw.{stream.name} ({cols});')
        print("raw." + stream.name)
srv = pgserver.get_server(os.environ["SCRATCH"] + "/dbt_local/pgdata", cleanup_mode=None)
srv.psql("\\c school_compare\n" + "\n".join(stmts))
EOF
chmod +x "$SCRATCH/dbt_local/dbt.sh" "$SCRATCH/dbt_local/sql.sh"

• Step 2: Raw tables for GIAS and the school MI

cd /Users/tudor/projects/school_compare
SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-gias --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_gias.tap:TapUKGIAS gias_establishments gias_links
SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_ofsted.tap:TapUKOfsted ofsted_inspections

Expected: raw.gias_establishments, raw.gias_links, raw.ofsted_inspections.

• Step 3: Baseline build

Run: $SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_ofsted_inspections int_ofsted_latest fact_ofsted_latest dim_school Expected: exit 0; the 10 int_ofsted_latest and 2 dim_school unit tests PASS on empty tables. If pgserver cannot start, stop and report: dbt tests are then proven only by the staging DAG runs, and the PR description must say so.

Task 2: GIAS inspectorate and UKPRN

Files:

• Modify: pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py (after th.Property("StatutoryHighAge", th.StringType),)
• Modify: pipeline/transform/models/staging/stg_gias_establishments.sql
• Create: pipeline/transform/models/staging/stg_gias_ukprn.sql
• Modify: pipeline/dags/school_data_pipeline.py (daily dbt_build selection)
• Test: pipeline/tests/test_gias_declared_fields.py

Interfaces:

• Produces: raw columns "UKPRN", "InspectorateName (name)"; stg_gias_establishments.inspectorate_name text; stg_gias_ukprn(urn integer, ukprn text).

• Step 1: Write the failing test

pipeline/tests/test_gias_declared_fields.py:

"""Every field the GIAS establishments stream declares must be a column of the
extract GIAS publishes, or it silently loads as null. H4 adds UKPRN (matches
Ofsted's further-education providers) and InspectorateName (name) ("ISI" for
about 1,340 independent schools). HEADER is GIAS's own, from
edubasealldata20261013.csv.
"""
import re
from pathlib import Path

TAP = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-gias'
       / 'tap_uk_gias' / 'tap.py')

HEADER = (
    'URN', 'LA (code)', 'LA (name)', 'EstablishmentNumber', 'EstablishmentName',
    'TypeOfEstablishment (code)', 'TypeOfEstablishment (name)', 'EstablishmentTypeGroup (code)',
    'EstablishmentTypeGroup (name)', 'EstablishmentStatus (code)', 'EstablishmentStatus (name)',
    'ReasonEstablishmentOpened (code)', 'ReasonEstablishmentOpened (name)', 'OpenDate',
    'ReasonEstablishmentClosed (code)', 'ReasonEstablishmentClosed (name)', 'CloseDate',
    'PhaseOfEducation (code)', 'PhaseOfEducation (name)', 'StatutoryLowAge', 'StatutoryHighAge',
    'Boarders (code)', 'Boarders (name)', 'NurseryProvision (name)', 'OfficialSixthForm (code)',
    'OfficialSixthForm (name)', 'Gender (code)', 'Gender (name)', 'ReligiousCharacter (code)',
    'ReligiousCharacter (name)', 'ReligiousEthos (name)', 'Diocese (code)', 'Diocese (name)',
    'AdmissionsPolicy (code)', 'AdmissionsPolicy (name)', 'SchoolCapacity', 'SpecialClasses (code)',
    'SpecialClasses (name)', 'CensusDate', 'NumberOfPupils', 'NumberOfBoys', 'NumberOfGirls',
    'PercentageFSM', 'TrustSchoolFlag (code)', 'TrustSchoolFlag (name)', 'Trusts (code)',
    'Trusts (name)', 'SchoolSponsorFlag (name)', 'SchoolSponsors (name)', 'FederationFlag (name)',
    'Federations (code)', 'Federations (name)', 'UKPRN', 'FEHEIdentifier',
    'FurtherEducationType (name)', 'LastChangedDate', 'Street', 'Locality', 'Address3', 'Town',
    'County (name)', 'Postcode', 'SchoolWebsite', 'TelephoneNum', 'HeadTitle (name)',
    'HeadFirstName', 'HeadLastName', 'HeadPreferredJobTitle', 'BSOInspectorateName (name)',
    'InspectorateReport', 'DateOfLastInspectionVisit', 'NextInspectionVisit', 'TeenMoth (name)',
    'TeenMothPlaces', 'CCF (name)', 'SENPRU (name)', 'EBD (name)', 'PlacesPRU', 'FTProv (name)',
    'EdByOther (name)', 'Section41Approved (name)', 'SEN1 (name)', 'SEN2 (name)', 'SEN3 (name)',
    'SEN4 (name)', 'SEN5 (name)', 'SEN6 (name)', 'SEN7 (name)', 'SEN8 (name)', 'SEN9 (name)',
    'SEN10 (name)', 'SEN11 (name)', 'SEN12 (name)', 'SEN13 (name)',
    'TypeOfResourcedProvision (name)', 'ResourcedProvisionOnRoll', 'ResourcedProvisionCapacity',
    'SenUnitOnRoll', 'SenUnitCapacity', 'GOR (code)', 'GOR (name)',
    'DistrictAdministrative (code)', 'DistrictAdministrative (name)', 'AdministrativeWard (code)',
    'AdministrativeWard (name)', 'ParliamentaryConstituency (code)',
    'ParliamentaryConstituency (name)', 'UrbanRural (code)', 'UrbanRural (name)',
    'GSSLACode (name)', 'Easting', 'Northing', 'MSOA (name)', 'LSOA (name)',
    'InspectorateName (name)', 'SENStat', 'SENNoStat', 'BoardingEstablishment (name)', 'PropsName',
    'PreviousLA (code)', 'PreviousLA (name)', 'PreviousEstablishmentNumber', 'Country (name)',
    'UPRN', 'SiteName', 'QABName (code)', 'QABName (name)', 'EstablishmentAccredited (code)',
    'EstablishmentAccredited (name)', 'QABReport', 'CHNumber', 'MSOA (code)', 'LSOA (code)', 'FSM',
    'AccreditationExpiryDate',
)


def declared():
    source = TAP.read_text()
    block = source[source.index('class GIASEstablishmentsStream'):source.index('class GIASLinksStream')]
    return re.findall(r'th\.Property\("([^"]+)"', block)


def test_every_declared_field_is_a_gias_column():
    assert set(declared()) - set(HEADER) == set()


def test_the_inspection_fields_are_declared():
    assert {'UKPRN', 'InspectorateName (name)'} <= set(declared())

• Step 2: Run it to see it fail

Run: uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_gias_declared_fields.py -q Expected: test_the_inspection_fields_are_declared FAILS; test_every_declared_field_is_a_gias_column passes (a guard for existing fields).

• Step 3: Declare the fields, stage them, add the UKPRN view

In tap_uk_gias/tap.py, after th.Property("StatutoryHighAge", th.StringType),:

        # Matches Ofsted's further-education providers, which some FE
        # records name by UKPRN only (audit H4).
        th.Property("UKPRN", th.StringType),
        # "ISI" for independent schools the Independent Schools Inspectorate
        # inspects; "Ofsted" or "Not applicable" otherwise (audit H4).
        th.Property("InspectorateName (name)", th.StringType),

In stg_gias_establishments.sql, after the line "NurseryProvision (name)" as nursery_provision,:

        nullif(trim("InspectorateName (name)"), '')     as inspectorate_name,

pipeline/transform/models/staging/stg_gias_ukprn.sql:

-- URN → UKPRN from the raw GIAS extract, for matching Ofsted's further
-- education providers (int_ofsted_fe_latest, int_ofsted_listed; audit H4).
-- A model on the raw source rather than a column of stg_gias_establishments:
-- a ref to that model would put the Ofsted models into the daily build's
-- stg_gias_establishments+ selection (pipeline/tests/test_dag_selectors.py).

select
    cast("URN" as integer)          as urn,
    nullif(trim("UKPRN"), '')       as ukprn
from {{ source('raw', 'gias_establishments') }}
where "URN" ~ '^[0-9]+$'

In pipeline/dags/school_data_pipeline.py, in the daily build, change --select stg_gias_establishments+ stg_gias_links+ gias_code_names+ to --select stg_gias_establishments+ stg_gias_links+ gias_code_names+ stg_gias_ukprn.

• Step 4: Run the tests and build

Run: the Step 2 pytest command → 2 passed. Run: uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_dag_selectors.py -q → all pass. Re-create the GIAS raw tables (Task 1 Step 2, first command) → the table now has "UKPRN" and "InspectorateName (name)". Run: $SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_gias_ukprn dim_school → exit 0.

• Step 5: Commit

git add pipeline/tests/test_gias_declared_fields.py pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py pipeline/transform/models/staging/stg_gias_establishments.sql pipeline/transform/models/staging/stg_gias_ukprn.sql pipeline/dags/school_data_pipeline.py
git commit -m "feat(gias): load UKPRN and the inspectorate GIAS names (H4)"

Task 3: Ofsted further-education extractor

Files:

• Create: pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py
• Modify: pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py (imports, two stream classes, discover_streams)
• Test: pipeline/tests/test_ofsted_fe_mi.py

Interfaces:

• Produces: fe_pages(results: list[dict]) -> list[str]; renewed_framework_pages(results: list[dict]) -> list[str]; latest_as_at_csv(html: str) -> str | None; latest_period_csv(html: str) -> str | None; read_mi_csv(content: bytes) -> pd.DataFrame; records(df, columns: dict[str, str], key: str) -> Iterator[dict]; providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None; PROVIDER_COLUMNS, INSPECTION_COLUMNS. Streams ofsted_fe_providers (key provider_urn; fields provider_urn, provider_ukprn, provider_name, provider_type, renewed_eif_full_inspection, graded_inspection_date, overall_effectiveness, short_inspection_date) and ofsted_fe_inspections (key inspection_number; fields inspection_number, provider_urn, provider_ukprn, first_day_of_inspection, date_published, safeguarding_standards), all strings.

• Step 1: Write the failing tests

pipeline/tests/test_ofsted_fe_mi.py:

"""Ofsted publishes further-education inspection outcomes on a gov.uk page per
academic year, separate from the school MI the tap already reads (audit H4).
The links below are the real ones from the 2025/26 page; the headers are
Ofsted's own.
"""
import importlib.util
from pathlib import Path

import pandas as pd
import pytest

MODULE = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-ofsted'
          / 'tap_uk_ofsted' / 'fe_mi.py')

A = 'https://assets.publishing.service.gov.uk/media/'
PAGE_2025_26 = f'''
<a href="{A}6989b16dc106cc5501b6e4c3/Management_information_-_further_education_and_skills_-_as_at_31_January_2026.csv">x</a>
<a href="{A}6989b18cd3f57710b50a9b92/Management_information_-_further_education_and_skills_-_10_November_2025_to_31_January_2026.csv">x</a>
<a href="{A}69aec8d6d620c14fa183eeb6/Management_information_-_further_education_and_skills_-__10_November_2025_to_28_February_2026.csv">x</a>
<a href="{A}69aec8fcc78869bf8eb8a54a/Management_information_-_further_education_and_skills_-_as_at_28_February_2026.csv">x</a>
<a href="{A}69aef091bde9c3f213c89a1e/Management_information_-_further_education_and_skills_-_as_at_31_January_2026.ods">x</a>
<a href="{A}6a9fec1e92e72b8ac437eef3/Management_information_-_further_education_and_skills_-_as_at_31_August_2026.csv">x</a>
<a href="{A}6a9fed6b5f6e942efe37eed4/Management_information_-_new_provider_monitoring_inspections_-_10_November_2025_to_31_August_2026.csv">x</a>
<a href="{A}6a9fed815f6e942efe37eed5/Management_information_-_monitoring_inspections_-_10_November_2025_to_31_August_2026.csv">x</a>
<a href="{A}6a9feda65f6e942efe37eed6/Management_information_-_further_education_and_skills_-_10_November_2025_to_31_August_2026.csv">x</a>
'''

SEARCH_RESULTS = [
    {'title': 'Further education and skills inspections and outcomes: management information from September 2024 to August 2025',
     'link': '/government/statistical-data-sets/fe-2024-25'},
    {'title': 'Further education and skills inspections and outcomes: management information from November 2025 to August 2026',
     'link': '/government/statistical-data-sets/fe-2025-26'},
    {'title': 'Further education and skills inspections and outcomes: management information from September 2023 to August 2024',
     'link': '/government/statistical-data-sets/fe-2023-24'},
    {'title': 'Monthly management information: Ofsted\'s school inspections outcomes',
     'link': '/government/statistical-data-sets/schools'},
]

PROVIDER_HEADER = (
    'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group',
    'Local authority', 'Region', 'Ofsted region',
    'Has the provider received a full inspection under the renewed EIF?',
    'Has the provider received a new provider monitoring inspection?',
    'Date of latest short inspection before renewed EIF (if relevant)',
    'Inspection number of latest full inspection before renewed EIF',
    'First day of inspection of latest inspection before renewed EIF',
    'Latest overall effectiveness before renewed EIF',
    'Latest quality of education before renewed EIF',
    'Latest behaviour and attitudes before renewed EIF',
    'Latest personal development before renewed EIF',
    'Latest leadership and management before renewed EIF',
    'Latest contribution to meeting skills needs before renewed EIF',
    'Latest safeguarding before renewed EIF', '',
)

INSPECTION_HEADER = (
    'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group',
    'Local authority', 'Region', 'Ofsted region', 'Inspection number', 'First day of inspection',
    'Date published', 'Safeguarding standards', 'Inclusion', 'Leadership and governance',
    'Contribution to meeting skills needs',
    'Education programmes for young people - curriculum, teaching and training',
    'Education programmes for young people - achievement',
    'Education programmes for young people - participation and development',
    'Adult learning programmes - curriculum, teaching and training',
    'Adult learning programmes - achievement',
    'Adult learning programmes - participation and development',
    'Apprenticeships - curriculum, teaching and training', 'Apprenticeships - achievement',
    'Apprenticeships - participation and development',
    'Provision for learners and apprentices with high needs - curriculum, teaching and training',
    'Provision for learners and apprentices with high needs - achievement',
    'Provision for learners and apprentices with high needs - participation and development', '',
)


@pytest.fixture
def fe():
    spec = importlib.util.spec_from_file_location('fe_mi', MODULE)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module


def test_the_newest_as_at_file_is_chosen(fe):
    assert fe.latest_as_at_csv(PAGE_2025_26).endswith('_as_at_31_August_2026.csv')


def test_the_latest_cumulative_inspections_file_is_chosen_not_a_monitoring_one(fe):
    assert fe.latest_period_csv(PAGE_2025_26) == (
        A + '6a9feda65f6e942efe37eed6/Management_information_-_further_education_and_skills_'
            '-_10_November_2025_to_31_August_2026.csv')


def test_a_doubled_underscore_file_name_still_parses(fe):
    page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'August' not in line)
    assert fe.latest_period_csv(page).endswith('__10_November_2025_to_28_February_2026.csv')


def test_monitoring_files_alone_give_nothing(fe):
    page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'monitoring' in line)
    assert fe.latest_period_csv(page) is None
    assert fe.latest_as_at_csv(page) is None


def test_pages_are_the_fe_series_newest_first(fe):
    assert fe.fe_pages(SEARCH_RESULTS) == [
        '/government/statistical-data-sets/fe-2025-26',
        '/government/statistical-data-sets/fe-2024-25',
        '/government/statistical-data-sets/fe-2023-24',
    ]


def test_report_cards_are_read_only_from_renewed_framework_pages(fe):
    # Earlier pages hold previous-framework inspection lists, not report cards.
    assert fe.renewed_framework_pages(SEARCH_RESULTS) == ['/government/statistical-data-sets/fe-2025-26']


def test_providers_fall_back_to_the_previous_page(fe):
    pages = {'/new': '<p>No files yet</p>', '/old': PAGE_2025_26}
    assert fe.providers_csv_url(['/new', '/old'], pages.__getitem__).endswith('_as_at_31_August_2026.csv')


@pytest.mark.parametrize('columns, header', [('PROVIDER_COLUMNS', PROVIDER_HEADER),
                                             ('INSPECTION_COLUMNS', INSPECTION_HEADER)])
def test_every_mapped_column_is_in_ofsteds_file(fe, columns, header):
    assert set(getattr(fe, columns)) - set(header) == set()


def test_csv_reading_and_records(fe):
    content = ('' + ','.join(f'"{c}"' for c in PROVIDER_HEADER[:14]) + '\n'
               + '145175,10005429,Queen Mary\'s College,Sixth form college,Sixth form colleges,Hampshire,'
                 'South East,South East,No,No,-,10227183,06/12/2022,2\n'
               + ',,Blank provider,,,,,,,,,,,\n').encode('utf-8')
    rows = list(fe.records(fe.read_mi_csv(content), fe.PROVIDER_COLUMNS, 'provider_urn'))
    assert rows == [{
        'provider_urn': '145175', 'provider_ukprn': '10005429', 'provider_name': "Queen Mary's College",
        'provider_type': 'Sixth form college', 'renewed_eif_full_inspection': 'No',
        'graded_inspection_date': '06/12/2022', 'overall_effectiveness': '2', 'short_inspection_date': '-',
    }]

• Step 2: Run them to see them fail

Run: uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_ofsted_fe_mi.py -q Expected: every test errors with FileNotFoundError for fe_mi.py.

• Step 3: Write the module

pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py:

"""Ofsted's further education and skills management information (audit H4).

Ofsted publishes FE inspection outcomes on one gov.uk page per academic year
("Further education and skills inspections and outcomes: management
information from November 2025 to August 2026"). Each page carries "as at"
files (one row per provider, with its latest inspection under the previous
framework) and cumulative files of renewed-framework inspections, i.e. report
cards ("…_10_November_2025_to_31_January_2026.csv", "…_to_28_February_2026.csv",
…). Monitoring-visit files are a different series.

Free of the Singer SDK so CI's pytest, which installs only the backend's
requirements, can load it.
"""
from __future__ import annotations

import io
import re
from datetime import date
from typing import Callable, Iterator

import pandas as pd

SEARCH_URL = "https://www.gov.uk/api/search.json"
SEARCH_QUERY = "further education and skills inspections and outcomes management information"
PAGE_TITLE_PREFIX = "Further education and skills inspections and outcomes: management information from "
# Renewed-framework (report card) inspections began on 10 November 2025.
RENEWED_FRAMEWORK_START = date(2025, 11, 1)

_MONTHS = {name: number for number, name in enumerate(
    ["january", "february", "march", "april", "may", "june", "july",
     "august", "september", "october", "november", "december"], start=1)}
_TITLE_START = re.compile(r"from ([A-Za-z]+) (\d{4})")
_CSV_LINK = re.compile(r'https://assets\.publishing\.service\.gov\.uk/[^"\s<>]+\.csv')
_AS_AT = re.compile(r"further_education_and_skills_-_+as_at_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$")
_PERIOD = re.compile(
    r"further_education_and_skills_-_+(\d{1,2})_([A-Za-z]+)_(\d{4})_to_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$")

# Ofsted column → Singer field.
PROVIDER_COLUMNS = {
    "Provider URN": "provider_urn",
    "Provider UKPRN": "provider_ukprn",
    "Provider name": "provider_name",
    "Provider type": "provider_type",
    "Has the provider received a full inspection under the renewed EIF?": "renewed_eif_full_inspection",
    "First day of inspection of latest inspection before renewed EIF": "graded_inspection_date",
    "Latest overall effectiveness before renewed EIF": "overall_effectiveness",
    "Date of latest short inspection before renewed EIF (if relevant)": "short_inspection_date",
}
INSPECTION_COLUMNS = {
    "Provider URN": "provider_urn",
    "Provider UKPRN": "provider_ukprn",
    "Inspection number": "inspection_number",
    "First day of inspection": "first_day_of_inspection",
    "Date published": "date_published",
    "Safeguarding standards": "safeguarding_standards",
}


def _date(day: str, month: str, year: str) -> date | None:
    number = _MONTHS.get(month.lower())
    return date(int(year), number, int(day)) if number else None


def _page_start(title: str) -> date | None:
    match = _TITLE_START.search(title)
    number = _MONTHS.get(match.group(1).lower()) if match else None
    return date(int(match.group(2)), number, 1) if number else None


def _dated_pages(results: list[dict]) -> list[tuple[date, str]]:
    pages = []
    for result in results:
        title = result.get("title", "")
        start = _page_start(title) if title.startswith(PAGE_TITLE_PREFIX) else None
        if start:
            pages.append((start, result["link"]))
    return sorted(pages, reverse=True)


def fe_pages(results: list[dict]) -> list[str]:
    """gov.uk paths of the FE MI pages, newest academic year first."""
    return [link for _, link in _dated_pages(results)]


def renewed_framework_pages(results: list[dict]) -> list[str]:
    """The pages that can hold report cards: from November 2025 onwards."""
    return [link for start, link in _dated_pages(results) if start >= RENEWED_FRAMEWORK_START]


def _latest(html: str, pattern: re.Pattern, date_groups: slice) -> str | None:
    best = None
    for link in set(_CSV_LINK.findall(html)):
        match = pattern.search(link)
        when = _date(*match.groups()[date_groups]) if match else None
        if when and (best is None or when > best[0]):
            best = (when, link)
    return best[1] if best else None


def latest_as_at_csv(html: str) -> str | None:
    """The newest "as at" CSV on a page."""
    return _latest(html, _AS_AT, slice(0, 3))


def latest_period_csv(html: str) -> str | None:
    """The renewed-framework inspections CSV with the latest end date on a page."""
    return _latest(html, _PERIOD, slice(3, 6))


def providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None:
    """The newest "as at" CSV, from the newest page that has one."""
    for page in pages:
        url = latest_as_at_csv(fetch(page))
        if url:
            return url
    return None


def read_mi_csv(content: bytes) -> pd.DataFrame:
    """Ofsted's MI CSVs: UTF-8 with a BOM, or Windows-1252. The header is the
    line naming "Provider URN"."""
    try:
        text = content.decode("utf-8-sig")
    except UnicodeDecodeError:
        text = content.decode("cp1252")
    lines = text.splitlines()
    start = next((i for i, line in enumerate(lines[:20]) if "Provider URN" in line), 0)
    return pd.read_csv(io.StringIO("\n".join(lines[start:])), dtype=str, keep_default_na=False)


def records(df: pd.DataFrame, columns: dict[str, str], key: str) -> Iterator[dict]:
    """Singer records: the mapped columns, stripped. Rows without `key` are skipped."""
    present = {source: field for source, field in columns.items() if source in df.columns}
    for _, row in df.iterrows():
        record = {field: str(row[source]).strip() for source, field in present.items()}
        if record.get(key):
            yield record

• Step 4: Add the two streams

In tap_uk_ofsted/tap.py, after from singer_sdk import typing as th:

from tap_uk_ofsted.fe_mi import (
    INSPECTION_COLUMNS,
    PROVIDER_COLUMNS,
    SEARCH_QUERY,
    SEARCH_URL,
    fe_pages,
    latest_period_csv,
    providers_csv_url,
    read_mi_csv,
    records,
    renewed_framework_pages,
)

Before class TapUKOfsted(Tap)::

def _fe_search_results() -> list[dict]:
    """gov.uk search results for Ofsted's FE MI pages (one per academic year)."""
    resp = requests.get(SEARCH_URL, params={"q": SEARCH_QUERY, "count": 50}, timeout=60)
    resp.raise_for_status()
    return resp.json().get("results", [])


def _gov_uk_page(path: str) -> str:
    url = path if path.startswith("http") else "https://www.gov.uk" + path
    resp = requests.get(url, timeout=60)
    resp.raise_for_status()
    return resp.text


class OfstedFEProvidersStream(Stream):
    """Ofsted's further education and skills providers: the newest "as at"
    file, one row per provider with its latest inspection under the previous
    framework (audit H4). Page and file selection: fe_mi.py."""

    name = "ofsted_fe_providers"
    primary_keys = ["provider_urn"]
    replication_key = None

    schema = th.PropertiesList(
        th.Property("provider_urn", th.StringType, required=True),
        *[th.Property(field, th.StringType) for field in PROVIDER_COLUMNS.values() if field != "provider_urn"],
    ).to_dict()

    def get_records(self, context):
        url = providers_csv_url(fe_pages(_fe_search_results()), _gov_uk_page)
        if not url:
            self.logger.error("No Ofsted FE 'as at' file found")
            return
        self.logger.info("Downloading Ofsted FE providers MI: %s", url)
        resp = requests.get(url, timeout=120)
        resp.raise_for_status()
        yield from records(read_mi_csv(resp.content), PROVIDER_COLUMNS, "provider_urn")


class OfstedFEInspectionsStream(Stream):
    """Ofsted's renewed-framework FE inspections (report cards), from the
    latest cumulative file on every page since November 2025, so a report
    card survives the next academic year's page (audit H4)."""

    name = "ofsted_fe_inspections"
    primary_keys = ["inspection_number"]
    replication_key = None

    schema = th.PropertiesList(
        th.Property("inspection_number", th.StringType, required=True),
        *[th.Property(field, th.StringType) for field in INSPECTION_COLUMNS.values() if field != "inspection_number"],
    ).to_dict()

    def get_records(self, context):
        for page in renewed_framework_pages(_fe_search_results()):
            url = latest_period_csv(_gov_uk_page(page))
            if not url:
                self.logger.warning("No renewed-framework FE inspections file on %s", page)
                continue
            self.logger.info("Downloading Ofsted FE inspections MI: %s", url)
            resp = requests.get(url, timeout=120)
            resp.raise_for_status()
            yield from records(read_mi_csv(resp.content), INSPECTION_COLUMNS, "inspection_number")

Change discover_streams to return [OfstedInspectionsStream(self), OfstedFEProvidersStream(self), OfstedFEInspectionsStream(self)].

• Step 5: Run the tests, then both streams against gov.uk

Run: the Step 2 pytest command → 11 passed.

uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted python -c "
from tap_uk_ofsted.tap import TapUKOfsted
tap = TapUKOfsted(config={})
p = list(tap.streams['ofsted_fe_providers'].get_records(None))
i = list(tap.streams['ofsted_fe_inspections'].get_records(None))
print(len(p), [r for r in p if r['provider_urn'] in ('145175', '143526', '130645')])
print(len(i), [r for r in i if r['provider_urn'] == '143526'])"

Expected: about 1,979 providers (Ofsted's count as at 31 August 2026; a newer file may differ slightly) with Queen Mary's College graded 2 on 06/12/2022, The Autism Project – CareTrade 2 on 16/11/2021, Exeter and North Devon Colleges Group -; about 1,472 inspections with 143526 inspected 12/05/2026, safeguarding Met.

• Step 6: Commit

git add pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py pipeline/tests/test_ofsted_fe_mi.py
git commit -m "feat(ofsted): extract Ofsted's further education and skills MI (H4)"

Task 4: FE staging models

Files:

• Modify: pipeline/transform/models/staging/_stg_sources.yml
• Create: pipeline/transform/models/staging/stg_ofsted_fe_providers.sql, stg_ofsted_fe_inspections.sql, stg_ofsted_fe.yml

Interfaces:

• Consumes: raw ofsted_fe_providers, ofsted_fe_inspections (Task 3 fields).

• Produces: stg_ofsted_fe_providers(provider_urn integer, provider_ukprn text, provider_name text, provider_type text, renewed_eif_full_inspection boolean, graded_inspection_date date, overall_effectiveness integer, short_inspection_date date); stg_ofsted_fe_inspections(inspection_number text, provider_urn integer, provider_ukprn text, rc_inspection_date date, published_date date, rc_safeguarding_met boolean).

• Step 1: Declare the sources, create the raw tables, write the unit tests

In _stg_sources.yml, after the ofsted_inspections entry:

      - name: ofsted_fe_providers
        description: Ofsted further education and skills MI, newest "as at" file — one row per provider with its latest inspection under the previous framework

      - name: ofsted_fe_inspections
        description: Ofsted further education and skills MI — renewed-framework inspections (report cards) since 10 November 2025, one row per inspection

Create the raw tables: SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_ofsted.tap:TapUKOfsted ofsted_fe_providers ofsted_fe_inspections (from the repo root) → two raw.… lines.

pipeline/transform/models/staging/stg_ofsted_fe.yml:

version: 2

unit_tests:
  - name: stg_ofsted_fe_providers_parses_grades_and_dates
    description: Ofsted writes "-" for no grade or date; grades are 1-4.
    model: stg_ofsted_fe_providers
    given:
      - input: source('raw', 'ofsted_fe_providers')
        rows:
          - {provider_urn: '145175', provider_ukprn: '10005429', provider_name: "Queen Mary's College", renewed_eif_full_inspection: 'No', graded_inspection_date: '06/12/2022', overall_effectiveness: '2', short_inspection_date: '-'}
          - {provider_urn: '130645', provider_ukprn: '-', provider_name: 'Exeter and North Devon Colleges Group', renewed_eif_full_inspection: 'No', graded_inspection_date: '-', overall_effectiveness: '-', short_inspection_date: '-'}
          - {provider_urn: '', provider_ukprn: '10000001', provider_name: 'No URN'}
    expect:
      rows:
        - {provider_urn: 145175, provider_ukprn: '10005429', renewed_eif_full_inspection: false, graded_inspection_date: '2022-12-06', overall_effectiveness: 2, short_inspection_date: null}
        - {provider_urn: 130645, provider_ukprn: null, renewed_eif_full_inspection: false, graded_inspection_date: null, overall_effectiveness: null, short_inspection_date: null}

  - name: stg_ofsted_fe_inspections_parses_dates_and_safeguarding
    model: stg_ofsted_fe_inspections
    given:
      - input: source('raw', 'ofsted_fe_inspections')
        rows:
          - {inspection_number: '10380001', provider_urn: '143526', provider_ukprn: '10000002', first_day_of_inspection: '12/05/2026', date_published: '16/06/2026', safeguarding_standards: 'Met'}
          - {inspection_number: '10380002', provider_urn: '150042', provider_ukprn: '10000003', first_day_of_inspection: '20/01/2026', date_published: '24/02/2026', safeguarding_standards: 'Not met'}
          - {inspection_number: '10380003', provider_urn: '1', provider_ukprn: '', first_day_of_inspection: '', date_published: '', safeguarding_standards: ''}
    expect:
      rows:
        - {inspection_number: '10380001', provider_urn: 143526, rc_inspection_date: '2026-05-12', published_date: '2026-06-16', rc_safeguarding_met: true}
        - {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', published_date: '2026-02-24', rc_safeguarding_met: false}

• Step 2: Run them to see them fail

Run: $SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "stg_ofsted_fe_providers stg_ofsted_fe_inspections,test_type:unit" Expected: non-zero exit; parsing fails because the two models do not exist.

• Step 3: Write the models

pipeline/transform/models/staging/stg_ofsted_fe_providers.sql:

-- Staging model: Ofsted further education and skills providers — the "as at"
-- MI file, one row per provider with its latest inspection under the previous
-- framework (audit H4). Ofsted writes '-' for no grade or date.

select
    cast(trim(provider_urn) as integer)                                          as provider_urn,
    nullif(nullif(trim(provider_ukprn), ''), '-')                                as provider_ukprn,
    provider_name,
    provider_type,
    lower(trim(renewed_eif_full_inspection)) = 'yes'                             as renewed_eif_full_inspection,
    case when trim(graded_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$'
        then to_date(trim(graded_inspection_date), 'DD/MM/YYYY') end             as graded_inspection_date,
    case when trim(overall_effectiveness) in ('1', '2', '3', '4')
        then trim(overall_effectiveness)::integer end                            as overall_effectiveness,
    case when trim(short_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$'
        then to_date(trim(short_inspection_date), 'DD/MM/YYYY') end              as short_inspection_date
from {{ source('raw', 'ofsted_fe_providers') }}
where trim(provider_urn) ~ '^[0-9]+$'

pipeline/transform/models/staging/stg_ofsted_fe_inspections.sql:

-- Staging model: Ofsted's renewed-framework FE inspections (report cards),
-- one row per inspection (audit H4). Only the headline is kept: the date and
-- the safeguarding result.

select
    trim(inspection_number)                                             as inspection_number,
    cast(trim(provider_urn) as integer)                                 as provider_urn,
    nullif(nullif(trim(provider_ukprn), ''), '-')                       as provider_ukprn,
    to_date(trim(first_day_of_inspection), 'DD/MM/YYYY')                as rc_inspection_date,
    case when trim(date_published) ~ '^\d{2}/\d{2}/\d{4}$'
        then to_date(trim(date_published), 'DD/MM/YYYY') end            as published_date,
    case lower(trim(safeguarding_standards))
        when 'met' then true
        when 'not met' then false
    end                                                                 as rc_safeguarding_met
from {{ source('raw', 'ofsted_fe_inspections') }}
where trim(provider_urn) ~ '^[0-9]+$'
  and trim(first_day_of_inspection) ~ '^\d{2}/\d{2}/\d{4}$'

• Step 4: Run the unit tests and build

Run: $SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_ofsted_fe_providers stg_ofsted_fe_inspections Expected: exit 0; both unit tests PASS; both views created.

• Step 5: Commit

git add pipeline/transform/models/staging/_stg_sources.yml pipeline/transform/models/staging/stg_ofsted_fe_providers.sql pipeline/transform/models/staging/stg_ofsted_fe_inspections.sql pipeline/transform/models/staging/stg_ofsted_fe.yml
git commit -m "feat(dbt): stage Ofsted's further education and skills MI (H4)"

Task 5: int_ofsted_fe_latest

Files:

• Create: pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql, int_ofsted_fe_latest.yml

Interfaces:

• Consumes: stg_gias_ukprn, stg_ofsted_fe_providers, stg_ofsted_fe_inspections.

• Produces: int_ofsted_fe_latest with exactly int_ofsted_latest's columns, in its order and types: urn, latest_visit_date, latest_visit_kind, latest_visit_outcome, current_grade, current_grade_date, current_grade_basis, graded_inspection_date, ungraded_inspection_date, inspection_date, inspection_type, framework, overall_effectiveness, quality_of_education, behaviour_attitudes, personal_development, leadership_management, early_years_provision, sixth_form_provision, ungraded_outcome, ungraded_grade, rc_safeguarding_met, rc_inclusion, rc_curriculum_teaching, rc_achievement, rc_attendance_behaviour, rc_personal_development, rc_leadership_governance, rc_early_years, rc_sixth_form, rc_inspection_date, report_url. latest_visit_kind ∈ report_card, graded, short.

• Step 1: Write the unit tests

pipeline/transform/models/intermediate/int_ofsted_fe_latest.yml:

version: 2

unit_tests:
  - name: int_ofsted_fe_grade_only
    description: Queen Mary's College (145175), Good on 6 December 2022, no report card.
    model: int_ofsted_fe_latest
    given:
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 145175, ukprn: '10005429'}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 145175, provider_ukprn: '10005429', graded_inspection_date: '2022-12-06', overall_effectiveness: 2}
      - input: ref('stg_ofsted_fe_inspections')
        rows: []
    expect:
      rows:
        - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, rc_inspection_date: null, framework: 'Further education and skills'}

  - name: int_ofsted_fe_short_inspection_is_a_visit_not_a_grade
    model: int_ofsted_fe_latest
    given:
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 1, ukprn: null}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 1, graded_inspection_date: '2019-03-05', overall_effectiveness: 2, short_inspection_date: '2023-02-01'}
      - input: ref('stg_ofsted_fe_inspections')
        rows: []
    expect:
      rows:
        - {urn: 1, latest_visit_date: '2023-02-01', latest_visit_kind: short, current_grade: 2, current_grade_date: '2019-03-05', current_grade_basis: graded}

  - name: int_ofsted_fe_newer_report_card_wins
    description: The Autism Project - CareTrade (143526). Good 2021; report card 12 May 2026, safeguarding met.
    model: int_ofsted_fe_latest
    given:
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 143526, ukprn: '10000002'}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 143526, provider_ukprn: '10000002', graded_inspection_date: '2021-11-16', overall_effectiveness: 2}
      - input: ref('stg_ofsted_fe_inspections')
        rows:
          - {inspection_number: '10380001', provider_urn: 143526, provider_ukprn: '10000002', rc_inspection_date: '2026-05-12', rc_safeguarding_met: true}
    expect:
      rows:
        - {urn: 143526, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, current_grade: null, current_grade_date: null, current_grade_basis: null, rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, overall_effectiveness: 2}

  - name: int_ofsted_fe_report_card_only
    model: int_ofsted_fe_latest
    given:
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 150042, ukprn: null}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 150042, graded_inspection_date: null, overall_effectiveness: null}
      - input: ref('stg_ofsted_fe_inspections')
        rows:
          - {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false}
    expect:
      rows:
        - {urn: 150042, latest_visit_date: '2026-01-20', latest_visit_kind: report_card, current_grade: null, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false}

  - name: int_ofsted_fe_listed_never_inspected_has_no_row
    model: int_ofsted_fe_latest
    given:
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 130645, ukprn: null}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 130645, graded_inspection_date: null, overall_effectiveness: null}
      - input: ref('stg_ofsted_fe_inspections')
        rows: []
    expect:
      rows: []

  - name: int_ofsted_fe_matches_by_ukprn_when_the_urn_differs
    model: int_ofsted_fe_latest
    given:
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 5, ukprn: '10099999'}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 9999999, provider_ukprn: '10099999', graded_inspection_date: '2024-01-30', overall_effectiveness: 3}
      - input: ref('stg_ofsted_fe_inspections')
        rows: []
    expect:
      rows:
        - {urn: 5, current_grade: 3, current_grade_date: '2024-01-30'}

  - name: int_ofsted_fe_prefers_the_urn_match
    model: int_ofsted_fe_latest
    given:
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 6, ukprn: '10088888'}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 6, provider_ukprn: '10077777', graded_inspection_date: '2023-05-01', overall_effectiveness: 1}
          - {provider_urn: 8888888, provider_ukprn: '10088888', graded_inspection_date: '2024-05-01', overall_effectiveness: 4}
      - input: ref('stg_ofsted_fe_inspections')
        rows: []
    expect:
      rows:
        - {urn: 6, current_grade: 1, current_grade_date: '2023-05-01'}

  - name: int_ofsted_fe_report_card_older_than_grade_is_ignored
    description: Bad data guard. A report card dated before the graded inspection cannot be the latest, and a row must never carry both a grade and a report card.
    model: int_ofsted_fe_latest
    given:
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 7, ukprn: null}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 7, graded_inspection_date: '2026-03-01', overall_effectiveness: 2}
      - input: ref('stg_ofsted_fe_inspections')
        rows:
          - {inspection_number: '1', provider_urn: 7, rc_inspection_date: '2026-01-10', rc_safeguarding_met: true}
    expect:
      rows:
        - {urn: 7, latest_visit_kind: graded, current_grade: 2, rc_inspection_date: null, rc_safeguarding_met: null}

• Step 2: Run them to see them fail

Run: $SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_fe_latest,test_type:unit" Expected: non-zero exit; parsing fails (model missing).

• Step 3: Write the model

pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql:

-- Intermediate model: Ofsted further education and skills status per URN
-- (audit H4), in int_ofsted_latest's columns so it can be unioned there.
-- A provider matches a school by Provider URN, else by UKPRN (stg_gias_ukprn).
-- Rule, in the C1 status rule's terms
-- (docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md):
--   a report card on or after the previous-framework inspection wins and
--   leaves no grade in force; otherwise a previous-framework grade 1-4 is the
--   current grade, dated by that inspection; the latest visit is the later of
--   it and any short inspection (whose outcome the MI does not give).

with gias as (
    select urn, ukprn from {{ ref('stg_gias_ukprn') }}
),

providers as (
    select distinct on (g.urn)
        g.urn,
        p.graded_inspection_date,
        p.overall_effectiveness,
        p.short_inspection_date
    from gias g
    join {{ ref('stg_ofsted_fe_providers') }} p
      on p.provider_urn = g.urn
      or (g.ukprn is not null and p.provider_ukprn = g.ukprn)
    order by g.urn, (p.provider_urn = g.urn) desc, p.graded_inspection_date desc nulls last
),

report_cards as (
    select distinct on (g.urn)
        g.urn,
        c.rc_inspection_date,
        c.rc_safeguarding_met
    from gias g
    join {{ ref('stg_ofsted_fe_inspections') }} c
      on c.provider_urn = g.urn
      or (g.ukprn is not null and c.provider_ukprn = g.ukprn)
    order by g.urn, (c.provider_urn = g.urn) desc, c.rc_inspection_date desc, c.inspection_number desc
),

combined as (
    select
        coalesce(p.urn, c.urn)                                          as urn,
        p.graded_inspection_date,
        p.overall_effectiveness,
        p.short_inspection_date,
        c.rc_inspection_date                                            as card_date,
        c.rc_safeguarding_met                                           as card_safeguarding_met
    from providers p
    full outer join report_cards c on c.urn = p.urn
),

decided as (
    select
        *,
        card_date is not null
            and card_date >= coalesce(greatest(graded_inspection_date, short_inspection_date), card_date)
            as report_card_latest
    from combined
)

select
    urn,
    case when report_card_latest then card_date
         else greatest(graded_inspection_date, short_inspection_date) end              as latest_visit_date,
    case when report_card_latest then 'report_card'
         when short_inspection_date > graded_inspection_date then 'short'
         else 'graded' end                                                              as latest_visit_kind,
    null::text                                                                          as latest_visit_outcome,
    case when not report_card_latest then overall_effectiveness end                     as current_grade,
    case when not report_card_latest then graded_inspection_date end                    as current_grade_date,
    case when not report_card_latest then 'graded' end                                  as current_grade_basis,
    graded_inspection_date,
    null::date                                                                          as ungraded_inspection_date,
    coalesce(graded_inspection_date, card_date)                                         as inspection_date,
    null::text                                                                          as inspection_type,
    'Further education and skills'::text                                                as framework,
    overall_effectiveness,
    null::integer                                                                       as quality_of_education,
    null::integer                                                                       as behaviour_attitudes,
    null::integer                                                                       as personal_development,
    null::integer                                                                       as leadership_management,
    null::integer                                                                       as early_years_provision,
    null::integer                                                                       as sixth_form_provision,
    null::text                                                                          as ungraded_outcome,
    null::integer                                                                       as ungraded_grade,
    case when report_card_latest then card_safeguarding_met end                         as rc_safeguarding_met,
    null::integer                                                                       as rc_inclusion,
    null::integer                                                                       as rc_curriculum_teaching,
    null::integer                                                                       as rc_achievement,
    null::integer                                                                       as rc_attendance_behaviour,
    null::integer                                                                       as rc_personal_development,
    null::integer                                                                       as rc_leadership_governance,
    null::integer                                                                       as rc_early_years,
    null::integer                                                                       as rc_sixth_form,
    case when report_card_latest then card_date end                                     as rc_inspection_date,
    null::text                                                                          as report_url
from decided
where report_card_latest or overall_effectiveness between 1 and 4

• Step 4: Run the unit tests and build

Run: $SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_ukprn stg_ofsted_fe_providers stg_ofsted_fe_inspections int_ofsted_fe_latest Expected: exit 0; all 8 int_ofsted_fe_latest unit tests PASS.

• Step 5: Commit

git add pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql pipeline/transform/models/intermediate/int_ofsted_fe_latest.yml
git commit -m "feat(dbt): further education status per URN under the C1 rule (H4)"

Task 6: Fold FE into int_ofsted_latest / fact_ofsted_latest

Files:

• Modify: pipeline/transform/models/intermediate/int_ofsted_latest.sql, int_ofsted_latest.yml
• Modify: pipeline/transform/models/marts/_marts_schema.yml (fact_ofsted_latest.latest_visit_kind accepted values)
• Create: pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql

Interfaces:

• Consumes: int_ofsted_fe_latest (Task 5; same columns and order as int_ofsted_latest).

• Produces: int_ofsted_latest / fact_ofsted_latest rows with framework = 'Further education and skills' for URNs that have no school-framework record; latest_visit_kind may be short.

• Step 1: Write the failing unit tests, and give the existing ones the new input

In int_ofsted_latest.yml, under given: of each of the 10 existing tests, after its stg_ofsted_inspections input, add:

      - input: ref('int_ofsted_fe_latest')
        rows: []

Append:

  - name: fe_status_fills_a_school_with_no_school_record
    description: Queen Mary's College (145175) has no school-framework record; its FE status passes through (audit H4).
    model: int_ofsted_latest
    given:
      - input: ref('stg_ofsted_inspections')
        rows: []
      - input: ref('int_ofsted_fe_latest')
        rows:
          - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'}
    expect:
      rows:
        - {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'}

  - name: a_school_record_wins_over_an_fe_record
    model: int_ofsted_latest
    given:
      - input: ref('stg_ofsted_inspections')
        rows:
          - {urn: 1, graded_inspection_date: '2019-06-01', overall_effectiveness: 2, framework: 'Schools - S5'}
      - input: ref('int_ofsted_fe_latest')
        rows:
          - {urn: 1, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, rc_inspection_date: '2026-05-12', framework: 'Further education and skills'}
    expect:
      rows:
        - {urn: 1, latest_visit_date: '2019-06-01', latest_visit_kind: graded, current_grade: 2, framework: 'Schools - S5'}

• Step 2: Run them to see them fail

Run: $SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_latest,test_type:unit" Expected: non-zero exit — int_ofsted_latest does not reference int_ofsted_fe_latest, so the given input is rejected (or, if dbt accepts unused inputs, fe_status_fills_a_school_with_no_school_record FAILS with no rows).

• Step 3: Union the FE rows

In int_ofsted_latest.sql, add to the header comment:

-- Schools with no school-framework record take their further education and
-- skills status from int_ofsted_fe_latest (same columns, same order; audit H4).

Replace the final select … from graded with the same select wrapped in a CTE, followed by the union:

school_status as (
    select
        urn,
        -- (… the existing select list, unchanged, through report_url …)
        report_url
    from graded
),

fe_status as (
    -- Same columns in the same order as school_status.
    select f.*
    from {{ ref('int_ofsted_fe_latest') }} f
    where not exists (select 1 from school_status s where s.urn = f.urn)
)

select * from school_status
union all
select * from fe_status

(The school_status select list is the existing final select list verbatim; only select becomes school_status as (select and a ) closes it. Remember the comma after the graded CTE's closing parenthesis.)

In _marts_schema.yml, fact_ofsted_latest → latest_visit_kind → accepted_values → values: ['report_card', 'graded', 'ungraded', 'short'].

pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql:

-- A school-framework record always wins over a further education one for the
-- same URN (audit H4): no FE row may stand where Ofsted's school MI has one.

select f.urn
from {{ ref('fact_ofsted_latest') }} f
join {{ ref('stg_ofsted_inspections') }} s on s.urn = f.urn
where f.framework = 'Further education and skills'

• Step 4: Run the unit tests and build

Run: $SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select int_ofsted_fe_latest int_ofsted_latest fact_ofsted_latest assert_ofsted_fe_never_overrides_school assert_ofsted_current_grade_consistent Expected: exit 0; 12 int_ofsted_latest unit tests PASS; schema and data tests PASS on empty tables.

• Step 5: Commit

git add pipeline/transform/models/intermediate/int_ofsted_latest.sql pipeline/transform/models/intermediate/int_ofsted_latest.yml pipeline/transform/models/marts/_marts_schema.yml pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql
git commit -m "feat(dbt): further education status where a school has no school record (H4)"

Task 7: int_ofsted_listed, dim_school columns, DAG selectors

Files:

• Create: pipeline/transform/models/intermediate/int_ofsted_listed.sql, int_ofsted_listed.yml
• Modify: pipeline/transform/models/marts/dim_school.sql, dim_school.yml
• Modify: pipeline/dags/school_data_pipeline.py (monthly Ofsted selection), pipeline/tests/test_dag_selectors.py

Interfaces:

• Consumes: raw ofsted_inspections, stg_gias_ukprn, stg_ofsted_fe_providers, stg_gias_establishments.inspectorate_name.

• Produces: int_ofsted_listed(urn integer); dim_school.inspectorate text, dim_school.ofsted_listed boolean (null when int_ofsted_listed does not exist yet).

• Step 1: Write the unit tests

pipeline/transform/models/intermediate/int_ofsted_listed.yml:

version: 2

unit_tests:
  - name: int_ofsted_listed_includes_uninspected_schools_and_fe_providers
    description: >
      Abbas and Templecombe (150031) is in Ofsted's school MI with no inspection;
      Exeter and North Devon Colleges Group (130645) is in the FE MI with none;
      a UKPRN-only FE match counts too. A school in neither is absent (audit H4).
    model: int_ofsted_listed
    given:
      - input: source('raw', 'ofsted_inspections')
        rows:
          - {urn: '150031', inspection_date: 'NULL'}
          - {urn: '102408', inspection_date: '17/06/2025'}
      - input: ref('stg_gias_ukprn')
        rows:
          - {urn: 130645, ukprn: null}
          - {urn: 5, ukprn: '10099999'}
          - {urn: 133058, ukprn: null}
      - input: ref('stg_ofsted_fe_providers')
        rows:
          - {provider_urn: 130645, provider_ukprn: null}
          - {provider_urn: 9999999, provider_ukprn: '10099999'}
    expect:
      rows:
        - {urn: 150031}
        - {urn: 102408}
        - {urn: 130645}
        - {urn: 5}

In dim_school.yml, under given: of both existing tests add:

      - input: ref('int_ofsted_listed')
        rows: []

and append:

  - name: dim_school_carries_inspectorate_and_ofsted_listing
    description: >
      Audit H4. GIAS names ISI for Abbey College Cambridge; Ofsted lists Abbas
      and Templecombe without an inspection; a third school is in neither.
    model: dim_school
    given:
      - input: ref('stg_gias_establishments')
        rows:
          - {urn: 136083, school_name: 'Abbey College Cambridge', status_code: 1, school_type_code: 11, local_authority_code: 873, phase_code: 4, inspectorate_name: 'ISI'}
          - {urn: 150031, school_name: 'Abbas and Templecombe Church School', status_code: 1, school_type_code: 28, local_authority_code: 933, phase_code: 2, inspectorate_name: 'Not applicable'}
          - {urn: 1, school_name: 'No data school', status_code: 1, school_type_code: 1, local_authority_code: 312, phase_code: 2, inspectorate_name: null}
      - input: ref('int_ofsted_latest')
        rows: []
      - input: ref('int_ofsted_listed')
        rows:
          - {urn: 150031}
    expect:
      rows:
        - {urn: 136083, inspectorate: 'ISI', ofsted_listed: false}
        - {urn: 150031, inspectorate: 'Not applicable', ofsted_listed: true}
        - {urn: 1, inspectorate: null, ofsted_listed: false}

• Step 2: Run them to see them fail

Run: $SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_listed dim_school,test_type:unit" Expected: non-zero exit (model int_ofsted_listed missing; dim_school has no such columns).

• Step 3: Write int_ofsted_listed and extend dim_school

pipeline/transform/models/intermediate/int_ofsted_listed.sql:

-- Intermediate model: every URN Ofsted's own data lists, inspected or not
-- (audit H4). Only these may be called "Not yet inspected". Reads the raw
-- school MI because stg_ofsted_inspections drops rows with no inspection date.

with school_mi as (
    select distinct cast(urn as integer) as urn
    from {{ source('raw', 'ofsted_inspections') }}
    where urn is not null
),

fe as (
    select distinct g.urn
    from {{ ref('stg_gias_ukprn') }} g
    join {{ ref('stg_ofsted_fe_providers') }} p
      on p.provider_urn = g.urn
      or (g.ukprn is not null and p.provider_ukprn = g.ukprn)
)

select urn from school_mi
union
select urn from fe

In dim_school.sql:

1. Below the existing -- depends_on: {{ ref('int_ofsted_latest') }} line add -- depends_on: {{ ref('int_ofsted_listed') }} and extend the comment above it: int_ofsted_latest and int_ofsted_listed are only ref()'d inside conditional blocks below (they may not exist before the Ofsted pipeline first runs), so declare the dependencies explicitly for dbt's DAG.
2. After the {% set ofsted_relation = … %} block add:

{% set listed_relation = adapter.get_relation(
    database=target.database,
    schema='intermediate',
    identifier='int_ofsted_listed'
) %}

3. After s.admissions_policy_code, add:

    -- The inspectorate GIAS names ("ISI" for most independent schools) and
    -- whether Ofsted's own data lists the school, inspected or not (audit H4).
    s.inspectorate_name as inspectorate,
    {% if listed_relation is not none %}
    l.urn is not null as ofsted_listed,
    {% else %}
    null::boolean as ofsted_listed,
    {% endif %}

4. After the {% if ofsted_relation is not none %} left join … {% endif %} block add:

{% if listed_relation is not none %}
left join {{ ref('int_ofsted_listed') }} l on s.urn = l.urn
{% endif %}

In pipeline/dags/school_data_pipeline.py, in the monthly Ofsted build change --select stg_ofsted_inspections+ int_ofsted_latest+ to --select stg_ofsted_inspections+ stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ int_ofsted_latest+.

In pipeline/tests/test_dag_selectors.py:

# dim_school reads int_ofsted_latest and int_ofsted_listed only when the
# relations exist (adapter.get_relation), so a missing table is not a failure.
OPTIONAL_PARENTS = {'int_ofsted_latest', 'int_ofsted_listed'}

• Step 4: Run the tests and build

Run: $SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select int_ofsted_listed dim_school Expected: exit 0; the int_ofsted_listed unit test and 3 dim_school unit tests PASS. Run: uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests -q → all pass.

• Step 5: Commit

git add pipeline/transform/models/intermediate/int_ofsted_listed.sql pipeline/transform/models/intermediate/int_ofsted_listed.yml pipeline/transform/models/marts/dim_school.sql pipeline/transform/models/marts/dim_school.yml pipeline/dags/school_data_pipeline.py pipeline/tests/test_dag_selectors.py
git commit -m "feat(dbt): dim_school.inspectorate and ofsted_listed (H4)"

Task 8: Load the real data end to end and verify PR 1

Files: none in the repo.

• Step 1: Load GIAS and Ofsted through the changed taps

mkdir -p "$SCRATCH/e2e_h4" && cd "$SCRATCH/e2e_h4"
echo "drop table if exists raw.gias_establishments, raw.gias_links, raw.ofsted_inspections, raw.ofsted_fe_providers, raw.ofsted_fe_inspections cascade;" | "$SCRATCH/dbt_local/sql.sh"
SOCK=$(grep -o 'PG_HOST=[^ ]*' "$SCRATCH/dbt_local/dbt.sh" | cut -d= -f2)
echo "{\"sqlalchemy_url\": \"postgresql+psycopg://postgres@/school_compare?host=$SOCK\", \"default_target_schema\": \"raw\"}" > target.json
echo '{}' > tap.json
cat > run_load.sh <<'EOF'
#!/bin/bash
R=/Users/tudor/projects/school_compare/pipeline/plugins/extractors
for tap in gias ofsted; do
  uv run --no-project --with-editable "$R/tap-uk-$tap" tap-uk-$tap --config tap.json 2> tap-$tap.log \
    | uv run --no-project --with meltanolabs-target-postgres target-postgres --config target.json > target-$tap.log 2>&1
  echo "$tap exit ${PIPESTATUS[*]}"
done
EOF
bash run_load.sh

Expected: gias exit 0 0 and ofsted exit 0 0.

• Step 2: Build and test everything this PR touches

Run: $SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_gias_ukprn stg_ofsted_inspections stg_ofsted_fe_providers stg_ofsted_fe_inspections int_ofsted_fe_latest int_ofsted_latest int_ofsted_listed fact_ofsted_latest dim_school assert_ofsted_fe_never_overrides_school assert_ofsted_current_grade_consistent Expected: exit 0; all unit, schema and data tests PASS.

• Step 3: Check the values

Through "$SCRATCH/dbt_local/sql.sh" <<'SQL' … SQL:

select urn, latest_visit_date, latest_visit_kind, current_grade, current_grade_date, rc_inspection_date, rc_safeguarding_met, framework
from marts.fact_ofsted_latest where urn in (145175, 143526, 130457) order by urn;
select count(*) from marts.fact_ofsted_latest where framework = 'Further education and skills';
select count(*) filter (where inspectorate = 'ISI') as isi, count(*) filter (where ofsted_listed) as listed from marts.dim_school;
select d.urn, d.ofsted_listed, f.urn is not null as has_record from marts.dim_school d left join marts.fact_ofsted_latest f using (urn) where d.urn in (150031, 136083);
select count(*) as fe_rows_for_listed_schools from marts.fact_ofsted_latest f join marts.dim_school d using (urn) where f.framework = 'Further education and skills';

Expected: 145175 graded 2 on 2022-12-06; 143526 report card 2026-05-12 with safeguarding true; 130457 graded 2 on 2024-01-30; about 490 FE rows for listed schools (443 + 45 + 2 as at 31 August 2026); about 1,341 ISI schools; 150031 listed with no record (unless Ofsted has inspected it since); 136083 not listed, no record. Record any drift from these numbers and its cause (newer MI files) in the PR description.

• Step 4: Run CI's Python suite

Run: uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests pipeline/tests scripts/ci/tests -q → all pass.

Task 9: Open PR 1

• Step 1: git status --short → only .DS_Store.
• Step 2: git push -u origin fix/ofsted-inspectorates-pipeline.
• Step 3: Open the PR through the Gitea API with basic auth from git credential fill (host privaterepo.sitaru.org; never print the secret): POST /api/v1/repos/tudor/school_compare/pulls, head fix/ofsted-inspectorates-pipeline, base main, title fix(pipeline): inspectorate, Ofsted listing and further-education status (H4, part 1 of 2). Body: what changed; Task 8's numbers; After merge: run the daily DAG (GIAS fields) and then school_data_monthly_ofsted on staging — the monthly build reads raw.gias_establishments."UKPRN", which exists only after a daily run with this PR; then merge PR 2; production the same order before PR 2 is promoted; ends with the Claude Code line.
• Step 4: Report the PR URL. Do not merge.

---

PR 2 — site

After PR 1 merges: git fetch origin && git switch -c fix/ofsted-inspectorates-site origin/main.

Task 10: Backend payloads, optional columns, Ofsted page link

Files:

• Modify: backend/schemas.py (SCHOOL_COLUMNS), backend/data_loader.py (_MAIN_QUERY, _EXTRA_OPTIONAL_COLUMNS, the fallback loop), backend/app.py (detail and compare school_info), backend/ofsted_codes.py (ofsted_page_url)
• Test: create backend/tests/test_inspection_status_payload.py; modify backend/tests/test_ofsted_codes.py:45, backend/tests/test_supplementary_enrichment.py:36

Interfaces:

• Produces: list rows, detail school_info and compare school_info carry inspectorate: str | null and ofsted_listed: bool | null; ofsted_page_url(urn) returns https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}.

• Step 1: Write the failing tests

backend/tests/test_inspection_status_payload.py:

"""Audit H4: every payload that feeds an Ofsted badge carries the inspectorate
GIAS names and whether Ofsted's own data lists the school, so the site never
says "Not yet inspected" on the strength of missing data."""

import pandas as pd
import pytest
from fastapi.testclient import TestClient


def _df():
    base = dict(year=202425, local_authority='Cambridgeshire', local_authority_code=873,
                postcode='CB1 1AA', latitude=52.2, longitude=0.12, address='1 High St',
                status='Open', town='Cambridge')
    return pd.DataFrame([
        dict(base, urn=136083, school_name='Abbey College Cambridge', phase='Secondary',
             school_type='Other independent school', inspectorate='ISI', ofsted_listed=False),
        dict(base, urn=150031, school_name='Abbas and Templecombe Church School', phase='Primary',
             school_type='Academy converter', inspectorate='Not applicable', ofsted_listed=True),
    ])


@pytest.fixture()
def client(monkeypatch):
    from backend import app as app_module
    from backend import database as database_module

    def _no_db():
        raise RuntimeError('no database in this test')

    monkeypatch.setattr(app_module, 'load_latest_school_data', _df)
    monkeypatch.setattr(app_module, 'load_school_data', _df)
    monkeypatch.setattr(app_module, 'get_supplementary_data', lambda db, urn: {})
    monkeypatch.setattr(app_module, '_place_registry', None)
    monkeypatch.setattr(database_module, 'SessionLocal', _no_db)
    return TestClient(app_module.app, raise_server_exceptions=False)


def test_the_list_carries_inspectorate_and_listing(client):
    resp = client.get('/api/schools?page_size=50')
    assert resp.status_code == 200, resp.text
    by_urn = {s['urn']: s for s in resp.json()['schools']}
    assert by_urn[136083]['inspectorate'] == 'ISI'
    assert by_urn[150031]['ofsted_listed'] is True


def test_the_school_page_carries_them(client):
    info = client.get('/api/schools/136083').json()['school_info']
    assert info['inspectorate'] == 'ISI'
    assert info['ofsted_listed'] is False


def test_compare_carries_them(client):
    body = client.get('/api/compare?urns=136083,150031').json()
    assert body['comparison']['150031']['school_info']['ofsted_listed'] is True
    assert body['comparison']['136083']['school_info']['inspectorate'] == 'ISI'


def test_a_mart_built_before_the_columns_still_loads():
    # Missing columns make the main query retry with NULLs (data_loader).
    from backend import data_loader
    assert {'inspectorate', 'ofsted_listed'} <= set(data_loader._EXTRA_OPTIONAL_COLUMNS)
    fallback = str(data_loader._MAIN_QUERY_NO_EXTRA_COLS)
    assert 'NULL AS inspectorate,' in fallback and 'NULL AS ofsted_listed,' in fallback

(/api/compare returns {"comparison": {<urn>: {"school_info": …}}}.) Change backend/tests/test_ofsted_codes.py:45 and backend/tests/test_supplementary_enrichment.py:36 to expect https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/<urn> (keep each test's URN).

• Step 2: Run them to see them fail

Run: uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests/test_inspection_status_payload.py backend/tests/test_ofsted_codes.py backend/tests/test_supplementary_enrichment.py -q Expected: the four new tests fail (KeyError: 'inspectorate'; the optional-columns assertion); the two link tests fail on the old provider/21 URL.

• Step 3: Implement

• backend/schemas.py, SCHOOL_COLUMNS: after "ofsted_rc_date", add "inspectorate", and "ofsted_listed",.

• backend/data_loader.py, _MAIN_QUERY: after s.nursery_provision, add s.inspectorate, and s.ofsted_listed,. Add "inspectorate", and "ofsted_listed", to _EXTRA_OPTIONAL_COLUMNS, and ("inspectorate", "s"), and ("ofsted_listed", "s"), to the tuple of (column, alias) pairs the fallback loop replaces. Extend the comment above _EXTRA_OPTIONAL_COLUMNS: … parliamentary constituency, and the inspectorate / Ofsted-listing columns added for audit H4) ….

• backend/app.py, detail school_info (after "nursery_provision": latest.get("nursery_provision"),):

            # What the Ofsted badge may say when there is no inspection record
            # (audit H4): the inspectorate GIAS names, and whether Ofsted's own
            # data lists the school at all.
            "inspectorate": latest.get("inspectorate"),
            "ofsted_listed": latest.get("ofsted_listed"),

• backend/app.py, compare school_info (after "has_sixth_form": convert_to_native(latest.get("has_sixth_form")),):

                    # Audit H4: what compare may say when there is no inspection record.
                    "inspectorate": convert_to_native(latest.get("inspectorate")),
                    "ofsted_listed": convert_to_native(latest.get("ofsted_listed")),

• backend/ofsted_codes.py:

def ofsted_page_url(urn: int) -> str:
    """The school's page on ofsted.gov.uk (all its reports live there — we
    never deep-link an individual report). Ofsted's provider pages sit under a
    category (21 primary, 23 secondary, 25 special, 40 FE colleges, …); this
    address redirects to the right one, where provider/21 404s for every
    non-primary school."""
    return f"https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}"

• Step 4: Run the tests and the backend suite

Run: the Step 2 command → all pass. Run: uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests -q → all pass.

• Step 5: Commit

git add backend/schemas.py backend/data_loader.py backend/app.py backend/ofsted_codes.py backend/tests/test_inspection_status_payload.py backend/tests/test_ofsted_codes.py backend/tests/test_supplementary_enrichment.py
git commit -m "fix(api): inspectorate and Ofsted listing in payloads; Ofsted links that resolve (H4)"

Task 11: The badge rule and its callers

Files:

• Modify: nextjs-app/lib/types.ts (School, OfstedLatestVisit), nextjs-app/lib/ofstedStatus.ts, nextjs-app/lib/utils.ts (buildOfstedListBadge)
• Modify: nextjs-app/components/SchoolRow.tsx, SecondarySchoolRow.tsx, LeafletMapInner.tsx
• Test: nextjs-app/__tests__/lib/utils.test.ts, __tests__/lib/ofstedStatus.test.ts, __tests__/components/SchoolRow.test.tsx

Interfaces:

• Produces: School.inspectorate?: string | null, School.ofsted_listed?: boolean | null; OfstedLatestVisit.kind adds 'short'; FE_FRAMEWORK = 'Further education and skills'; hasReportCard(ofsted: Pick<OfstedInspection, 'rc_inspection_date' | 'report_card'>): boolean; buildOfstedListBadge(school): OfstedListBadge | null.

• Step 1: Write the failing tests

In __tests__/lib/utils.test.ts, inside describe('buildOfstedListBadge', …): replace the test returns pending badge when no grade and no inspection on record with the block below, and in every other test in that describe use buildOfstedListBadge(…)! (the function may now return null):

  it('says "Not yet inspected" only when Ofsted lists the school uninspected (audit H4)', () => {
    const badge = buildOfstedListBadge({ ofsted_grade: null, ofsted_date: null, ofsted_listed: true });
    expect(badge).toEqual({ label: 'Not yet inspected', cssClass: 'ofstedPending' });
  });

  it('names ISI for a school GIAS says ISI inspects', () => {
    expect(buildOfstedListBadge({ ofsted_date: null, inspectorate: 'ISI', ofsted_listed: false }))
      .toEqual({ label: 'Inspected by ISI', cssClass: 'ofstedPending' });
  });

  it('an Ofsted record wins over ISI', () => {
    expect(buildOfstedListBadge({ ofsted_grade: 2, ofsted_grade_date: '2022-03-01', inspectorate: 'ISI' })!.label)
      .toBe('Good · 2022');
  });

  it('gives no badge without any inspection data', () => {
    expect(buildOfstedListBadge({ ofsted_grade: null, ofsted_date: null, ofsted_framework: null })).toBeNull();
    expect(buildOfstedListBadge({ ofsted_listed: null, inspectorate: 'Not applicable' })).toBeNull();
  });

In __tests__/lib/ofstedStatus.test.ts add:

it('names a short inspection as Ofsted does', () => {
  expect(latestVisitLine({ date: '2023-02-01', kind: 'short', outcome: null })).toBe('Short inspection, 1 February 2023');
});

it('treats a report card with only a safeguarding result as a report card (FE)', () => {
  expect(hasReportCard({ rc_inspection_date: '2026-05-12', report_card: {} })).toBe(true);
  expect(hasReportCard({ rc_inspection_date: null, report_card: {} })).toBe(false);
});

(change the file's first line to import { gradeSourceLine, hasReportCard, latestVisitLine, showLatestVisitLine } from '@/lib/ofstedStatus';).

In __tests__/components/SchoolRow.test.tsx add (the file's school fixture is base):

it('renders no Ofsted badge for a school with no inspection data (audit H4)', () => {
  const { container } = render(<SchoolRow school={{ ...base, ofsted_grade: null, ofsted_date: null, ofsted_listed: false, inspectorate: null }} />);
  expect(container.textContent).not.toMatch(/Not yet inspected|Inspected/);
});

• Step 2: Run them to see them fail

Run: cd nextjs-app && npx jest __tests__/lib/utils.test.ts __tests__/lib/ofstedStatus.test.ts __tests__/components/SchoolRow.test.tsx Expected: the new tests fail (no ISI rule; hasReportCard missing; "Not yet inspected" for no data).

• Step 3: Implement

lib/types.ts: in School, after ofsted_framework?: string | null; add

  /** The inspectorate GIAS names ("ISI", "Ofsted", "Not applicable"). Audit H4. */
  inspectorate?: string | null;
  /** Whether Ofsted's own data lists the school, inspected or not. Audit H4. */
  ofsted_listed?: boolean | null;

and in OfstedLatestVisit change kind to 'report_card' | 'graded' | 'ungraded' | 'short';.

lib/ofstedStatus.ts: add short: 'Short inspection', to VISIT_KIND, and

/** fact_ofsted_latest.framework for Ofsted's further education and skills records (audit H4). */
export const FE_FRAMEWORK = 'Further education and skills';

/**
 * A report card is in force. Checked by its date as well as its area labels:
 * an FE report card carries a safeguarding result and no school areas.
 */
export function hasReportCard(
  ofsted: Pick<OfstedInspection, 'rc_inspection_date' | 'report_card'>,
): boolean {
  return !!ofsted.rc_inspection_date || Object.keys(ofsted.report_card ?? {}).length > 0;
}

lib/utils.ts, buildOfstedListBadge: add inspectorate?: string | null; ofsted_listed?: boolean | null; to the parameter type, change the return type to OfstedListBadge | null, replace the last bullet of the doc comment with

 *   - No inspection on record (audit H4): "Inspected by ISI" when GIAS names
 *     ISI; "Not yet inspected" only when Ofsted's own data lists the school;
 *     otherwise no badge, never a claim the data cannot support.

and replace return { label: 'Not yet inspected', cssClass: 'ofstedPending' }; with

  if (school.inspectorate === 'ISI') return { label: 'Inspected by ISI', cssClass: 'ofstedPending' };
  if (school.ofsted_listed) return { label: 'Not yet inspected', cssClass: 'ofstedPending' };
  return null;

components/SchoolRow.tsx and components/SecondarySchoolRow.tsx: wrap the badge span in {ofstedBadge && ( … )}.

components/LeafletMapInner.tsx, cardHtml: replace the tags line with

  <div class="sc-card-tags">${badge ? `<span class="sc-ofsted sc-${badge.cssClass}">${escapeHtml(badge.label)}</span>` : ''}${type}</div>

• Step 4: Run the tests, the suite and the type check

Run: npx jest __tests__/lib/utils.test.ts __tests__/lib/ofstedStatus.test.ts __tests__/components/SchoolRow.test.tsx → pass. Run: npm run typecheck && npm test → all pass (fix any other caller the type check names by handling null, never by asserting it away).

• Step 5: Commit

git add nextjs-app/lib/types.ts nextjs-app/lib/ofstedStatus.ts nextjs-app/lib/utils.ts nextjs-app/components/SchoolRow.tsx nextjs-app/components/SecondarySchoolRow.tsx nextjs-app/components/LeafletMapInner.tsx nextjs-app/__tests__/lib/utils.test.ts nextjs-app/__tests__/lib/ofstedStatus.test.ts nextjs-app/__tests__/components/SchoolRow.test.tsx
git commit -m "fix(search): no \"Not yet inspected\" without Ofsted's word; ISI badge (H4)"

Task 12: School page and compare

Files:

• Create: nextjs-app/components/school/InspectionNotice.tsx; test nextjs-app/__tests__/components/InspectionNotice.test.tsx
• Modify: nextjs-app/components/school/OfstedSection.tsx, PrimarySchoolSections.tsx, SecondarySchoolSections.tsx, nextjs-app/lib/compareLogic.ts, nextjs-app/components/compare/CompareOfsted.tsx, CompareAtAGlance.tsx
• Test: nextjs-app/__tests__/components/OfstedSection.test.tsx, __tests__/components/CompareOfsted.test.tsx, __tests__/lib/compareLogic.test.ts

Interfaces:

• Consumes: FE_FRAMEWORK, hasReportCard, School.inspectorate, School.ofsted_listed (Task 11).

• Produces: InspectionNotice({ school }: { school: Pick<School, 'urn' | 'inspectorate' | 'ofsted_listed'> }) — renders the ISI or "Not yet inspected" section, or nothing; noInspectionLabel(school): string in lib/compareLogic.ts.

• Step 1: Write the failing tests

__tests__/components/InspectionNotice.test.tsx:

import { render, screen } from '@testing-library/react';
import { InspectionNotice } from '@/components/school/InspectionNotice';

it('names ISI and links its reports for an ISI school (audit H4)', () => {
  render(<InspectionNotice school={{ urn: 136083, inspectorate: 'ISI', ofsted_listed: false }} />);
  expect(screen.getByText('Inspected by the Independent Schools Inspectorate (ISI)')).toBeInTheDocument();
  expect(screen.getByRole('link', { name: /ISI reports/ })).toHaveAttribute('href', 'https://www.isi.net/reports/');
});

it('says "Not yet inspected" for a school Ofsted lists without an inspection', () => {
  render(<InspectionNotice school={{ urn: 150031, inspectorate: 'Not applicable', ofsted_listed: true }} />);
  expect(screen.getByText('Not yet inspected')).toBeInTheDocument();
  expect(screen.getByRole('link', { name: /Ofsted reports/ })).toHaveAttribute(
    'href', 'https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/150031');
});

it('renders nothing without inspection data, including an unknown listing', () => {
  const { container } = render(<InspectionNotice school={{ urn: 1, inspectorate: null, ofsted_listed: null }} />);
  expect(container).toBeEmptyDOMElement();
});

In __tests__/components/OfstedSection.test.tsx add (uses the file's renderSection):

it('labels a further education record and prints its short inspection (audit H4)', () => {
  renderSection({
    framework: 'Further education and skills',
    current_grade: { grade: 2, date: '2019-03-05', basis: 'graded' },
    latest_visit: { date: '2023-02-01', kind: 'short', outcome: null },
  });
  expect(screen.getByText('Further education and skills')).toBeInTheDocument();
  expect(screen.getByText('Good')).toBeInTheDocument();
  expect(screen.getByText('Latest visit: Short inspection, 1 February 2023')).toBeInTheDocument();
});

In __tests__/lib/compareLogic.test.ts add:

it('shows an FE report card with only a safeguarding result as a report card (audit H4)', () => {
  const display = ofstedDisplay(ofsted({ rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, report_card: {} }));
  expect(display.kind).toBe('report_card');
});

it('words the no-record compare cell from the evidence', () => {
  expect(noInspectionLabel({ inspectorate: 'ISI' })).toBe('Inspected by ISI');
  expect(noInspectionLabel({ ofsted_listed: true })).toBe('Not yet inspected');
  expect(noInspectionLabel({ ofsted_listed: null })).toBe('No inspection in our dataset');
});

(ofsted(partial) is the file's existing fixture helper; add noInspectionLabel to its import from @/lib/compareLogic.)

In __tests__/components/CompareOfsted.test.tsx, inside describe('CompareOfsted', …):

  it('words a school without an Ofsted record from the evidence (audit H4)', () => {
    const isi = { ...school(4, 'ISI School'), inspectorate: 'ISI' } as School;
    const listed = { ...school(5, 'New Academy'), ofsted_listed: true } as School;
    render(<CompareOfsted schools={[isi, listed]} data={{
      '4': { school_info: isi, yearly_data: [] } as unknown as ComparisonData,
      '5': { school_info: listed, yearly_data: [] } as unknown as ComparisonData,
    }} />);
    expect(screen.getAllByText('Inspected by ISI').length).toBeGreaterThan(0);
    expect(screen.getAllByText('Not yet inspected').length).toBeGreaterThan(0);
    expect(screen.queryByText(/No inspection outcome in our dataset/)).toBeNull();
  });

• Step 2: Run them to see them fail

Run: npx jest __tests__/components/InspectionNotice.test.tsx __tests__/components/OfstedSection.test.tsx __tests__/lib/compareLogic.test.ts __tests__/components/CompareOfsted.test.tsx Expected: the new tests fail (component and helper missing; no FE label; FE report card read as no_overall_grade).

• Step 3: Implement

components/school/InspectionNotice.tsx:

/**
 * InspectionNotice — the school page's inspection section when there is no
 * Ofsted inspection record (audit H4): the inspectorate GIAS names (ISI), or
 * Ofsted's own "not yet inspected" when its data lists the school. Nothing
 * otherwise: no claim the data cannot support. Server component.
 */

import type { School } from '@/lib/types';
import { Section, sectionStyles as styles } from './sectionShared';

const OFSTED_REPORTS = 'https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/';
const ISI_REPORTS = 'https://www.isi.net/reports/';

export function InspectionNotice({ school }: { school: Pick<School, 'urn' | 'inspectorate' | 'ofsted_listed'> }) {
  if (school.inspectorate === 'ISI') {
    return (
      <Section id="ofsted">
        <h2 className={styles.sectionTitle}>
          Inspection
          <a href={ISI_REPORTS} target="_blank" rel="noopener noreferrer" className={styles.ofstedReportLink}
             data-umami-event="external_link_clicked" data-umami-event-target="isi">
            ISI reports ↗
          </a>
        </h2>
        <p className={styles.ofstedDisclaimer}>Inspected by the Independent Schools Inspectorate (ISI)</p>
      </Section>
    );
  }
  if (school.ofsted_listed) {
    return (
      <Section id="ofsted">
        <h2 className={styles.sectionTitle}>
          Ofsted Rating
          <a href={`${OFSTED_REPORTS}${school.urn}`} target="_blank" rel="noopener noreferrer"
             className={styles.ofstedReportLink} data-umami-event="external_link_clicked" data-umami-event-target="ofsted">
            Ofsted reports ↗
          </a>
        </h2>
        <div className={styles.ofstedHeader}>
          <span className={styles.ofstedGrade}>Not yet inspected</span>
        </div>
      </Section>
    );
  }
  return null;
}

components/school/OfstedSection.tsx: import FE_FRAMEWORK from @/lib/ofstedStatus; directly after the closing </h2> add

      {ofsted.framework === FE_FRAMEWORK && (
        <p className={styles.ofstedDisclaimer}>{FE_FRAMEWORK}</p>
      )}

PrimarySchoolSections.tsx and SecondarySchoolSections.tsx: import hasReportCard from @/lib/ofstedStatus and InspectionNotice from ./InspectionNotice; set const isReportCard = !!ofsted && hasReportCard(ofsted);; change {ofsted && ( <OfstedSection … /> )} to {ofsted ? ( <OfstedSection … /> ) : <InspectionNotice school={schoolInfo} />}.

lib/compareLogic.ts: import hasReportCard from ./ofstedStatus; in ofstedDisplay replace the report-card condition with if (hasReportCard(ofsted)) {; add

/**
 * The compare cell when a school has no Ofsted record (audit H4): ISI per GIAS,
 * Ofsted's own "not yet inspected", or plainly that we hold nothing.
 */
export function noInspectionLabel(school: Pick<School, 'inspectorate' | 'ofsted_listed'>): string {
  if (school.inspectorate === 'ISI') return 'Inspected by ISI';
  if (school.ofsted_listed) return 'Not yet inspected';
  return 'No inspection in our dataset';
}

(import School from ./types if not already imported).

components/compare/CompareOfsted.tsx: give ResultCell a school prop (school: School) and return <span className={s.small}>{noInspectionLabel(school)}</span> for display.kind === 'none'; pass school={data[String(school.urn)]?.school_info ?? school} where it is rendered; in the judgement-detail cell replace No inspection in our dataset with {noInspectionLabel(data[String(school.urn)]?.school_info ?? school)}; change the Ofsted page fallback URL to https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/${school.urn}.

components/compare/CompareAtAGlance.tsx: replace No inspection in our dataset with {noInspectionLabel(data[String(school.urn)]?.school_info ?? school)}.

• Step 4: Run the tests, the suite and the type check

Run: the Step 2 command → pass. Run: npm run typecheck && npm test → all pass.

• Step 5: Commit

git add nextjs-app/components/school/InspectionNotice.tsx nextjs-app/components/school/OfstedSection.tsx nextjs-app/components/school/PrimarySchoolSections.tsx nextjs-app/components/school/SecondarySchoolSections.tsx nextjs-app/lib/compareLogic.ts nextjs-app/components/compare/CompareOfsted.tsx nextjs-app/components/compare/CompareAtAGlance.tsx nextjs-app/__tests__/components/InspectionNotice.test.tsx nextjs-app/__tests__/components/OfstedSection.test.tsx nextjs-app/__tests__/lib/compareLogic.test.ts nextjs-app/__tests__/components/CompareOfsted.test.tsx
git commit -m "fix(school, compare): ISI, not-yet-inspected and further-education states (H4)"

Task 13: E2E journeys

Files:

• Modify: e2e/tests/journeys.spec.ts (after the Ofsted journeys, test('a grade confirmed at an ungraded visit says so and is dated by it', …))

• Step 1: Add the journeys

test('a further education college shows its Ofsted result, not "Not yet inspected"', async ({ page }) => {
  // Queen Mary's College (145175) is inspected under Ofsted's further education
  // framework, which the site did not load: its badge said "Not yet inspected"
  // (audit H4). Its result may change; the label must never be absent or wrong.
  const res = await page.request.get('/api/schools/145175');
  expect(res.ok()).toBeTruthy();
  await page.goto('/school/145175');
  const ofsted = page.locator('#ofsted');
  await expect(ofsted).toBeVisible();
  await expect(ofsted.getByText('Further education and skills')).toBeVisible();
  await expect(ofsted).not.toContainText('Not yet inspected');
});

test('an ISI-inspected school says so instead of "Not yet inspected"', async ({ page }) => {
  const res = await page.request.get('/api/schools?school_type=independent&phase=secondary&page_size=100');
  expect(res.ok()).toBeTruthy();
  const school = ((await res.json()).schools ?? []).find(
    (s: { inspectorate?: string | null; ofsted_date?: string | null; ofsted_rc_date?: string | null }) =>
      s.inspectorate === 'ISI' && !s.ofsted_date && !s.ofsted_rc_date);
  test.skip(!school, 'no ISI school without an Ofsted record here');
  await page.goto(`/school/${school.urn}`);
  await expect(page.getByText('Inspected by the Independent Schools Inspectorate (ISI)')).toBeVisible();
  await expect(page.locator('#ofsted')).not.toContainText('Not yet inspected');
});

test('"Not yet inspected" appears only for a school Ofsted lists without an inspection', async ({ page }) => {
  const res = await page.request.get('/api/schools?phase=primary&page_size=500');
  expect(res.ok()).toBeTruthy();
  type Row = { urn: number; ofsted_listed?: boolean | null; ofsted_date?: string | null; ofsted_rc_date?: string | null; ofsted_grade?: number | null };
  const schools: Row[] = (await res.json()).schools ?? [];
  const unlisted = schools.find(s => !s.ofsted_listed && !s.ofsted_date && !s.ofsted_rc_date && !s.ofsted_grade);
  const listed = schools.find(s => s.ofsted_listed && !s.ofsted_date && !s.ofsted_rc_date && !s.ofsted_grade);
  test.skip(!listed, 'no listed, uninspected primary on this page');
  await page.goto(`/school/${listed!.urn}`);
  await expect(page.locator('#ofsted').getByText('Not yet inspected')).toBeVisible();
  if (unlisted) {
    await page.goto(`/school/${unlisted.urn}`);
    await expect(page.getByText('Not yet inspected')).toHaveCount(0);
  }
});

• Step 2: Check the file compiles and lists the tests

Run: cd e2e && npx playwright test --list tests/journeys.spec.ts -g "further education college|ISI-inspected|Ofsted lists without" → the three tests listed.

• Step 3: Run them against staging to see them fail

Only once PR 1 has merged and the daily and monthly Ofsted DAGs have run on staging: cd e2e && BASE_URL=https://stx.schoolcompare.co.uk npx playwright test tests/journeys.spec.ts -g "further education college|ISI-inspected|Ofsted lists without" --reporter=line --retries=0 Expected: all three FAIL for the right reason (staging still runs the old site: no FE label, no ISI section, the old badge rule), or skip where the old API lacks the fields; record which. If the DAGs have not run, record that and rely on the post-merge gate.

• Step 4: Commit

git add e2e/tests/journeys.spec.ts
git commit -m "test(e2e): further education, ISI and not-yet-inspected states (H4)"

Task 14: Verify and open PR 2

• Step 1: cd nextjs-app && npm run typecheck && npm test; from the repo root uv run --no-project --with-requirements requirements.txt python -c "from backend.app import app; print('backend imports OK')" and uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests pipeline/tests scripts/ci/tests -q → all pass.
• Step 2: git status --short → only .DS_Store. git push -u origin fix/ofsted-inspectorates-site.
• Step 3: Open the PR through the Gitea API (as Task 9): head fix/ofsted-inspectorates-site, title fix(site): ISI, Ofsted-listed and further-education inspection states (H4, part 2 of 2). Body: what changed, including the Ofsted page link fix (provider/21 404s for every non-primary school); Task 13 Step 3's result; merge only after the daily and monthly Ofsted DAGs have run on staging with PR 1; promote only after they have run on production (before that, the fallback query gives no badge for the ISI and listed groups); ends with the Claude Code line.
• Step 4: Report the PR URL. Do not merge.

---

After both PRs are live in production (Tudor promotes)

Validate against GIAS and Ofsted's school and FE MI: every "Not yet inspected" belongs to a school Ofsted's data lists without an inspection; every "Inspected by ISI" matches GIAS; every FE grade, date and report card matches the FE files; schools with no data show no badge; the compare page's Ofsted link resolves for a secondary and a special school. Then mark H4 resolved in the audit report artifact.