1963 lines
101 KiB
Markdown
1963 lines
101 KiB
Markdown
# Inspection Status Beyond Ofsted's School Data Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Stop labelling schools "Not yet inspected" unless Ofsted's own data lists them uninspected, and show ISI and Ofsted further-education results where the data exists (audit H4).
|
||
|
||
**Architecture:** PR 1 adds two GIAS fields, two Ofsted further-education (FE) streams, an FE status model folded into `int_ofsted_latest` / `fact_ofsted_latest` (framework "Further education and skills"), an `int_ofsted_listed` model, and `dim_school.inspectorate` / `dim_school.ofsted_listed`. PR 2, after the daily and monthly Ofsted DAGs have run on PR 1, publishes the two fields, changes the badge rule, adds the ISI / "Not yet inspected" school-page states and the FE label, updates compare, and fixes the Ofsted page link.
|
||
|
||
**Tech Stack:** Singer SDK taps (Python, pandas, requests), meltanolabs target-postgres, dbt-postgres ~1.10 (YAML unit tests, singular tests), FastAPI + pandas + SQLAlchemy, Next.js / React / TypeScript, Jest + Testing Library, Playwright.
|
||
|
||
**Spec:** `docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md` (read its "Amendments" section: UKPRN comes from a new `stg_gias_ukprn` view; report cards are detected by `rc_inspection_date`; `ofsted_page_url` is fixed).
|
||
|
||
## Global Constraints
|
||
|
||
- Badge order (search rows and map card): report card → "Report Card · year"; grade in force → grade word · year; inspected, no grade → "Inspected · year"; GIAS inspectorate `ISI` → "Inspected by ISI" (grey, css class `ofstedPending`); `ofsted_listed` → "Not yet inspected" (grey, `ofstedPending`); otherwise no badge.
|
||
- FE framework value, exactly: `Further education and skills`.
|
||
- FE rule: a report card on or after the previous-framework inspection (or with none) wins and leaves no grade in force; otherwise a previous-framework grade 1–4 is the current grade, dated by that inspection's first day, basis `graded`; latest visit is the later of that inspection (`graded`) and any short inspection (`short`); grade `-` with no report card gives no FE row.
|
||
- A school-framework record always wins over an FE record for the same URN.
|
||
- FE providers match a URN by `Provider URN` first, then by `Provider UKPRN` = GIAS UKPRN.
|
||
- The Ofsted models must never `ref` `stg_gias_establishments` (the daily build's `stg_gias_establishments+` would select them); they read UKPRN from `stg_gias_ukprn`, a view on the raw GIAS source.
|
||
- Copy: Ofsted's and GIAS's own terms ("Short inspection", "Not yet inspected", "Inspected by the Independent Schools Inspectorate (ISI)", "Further education and skills"); no derived sentences.
|
||
- ISI link: `https://www.isi.net/reports/`. Ofsted page link: `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}` (redirects to the right category; `provider/21/{urn}` 404s for every non-primary school).
|
||
- PR 2 must not merge before the daily DAG (GIAS fields) and `school_data_monthly_ofsted` have run on staging with PR 1, nor be promoted before they have run on production. Promotion is Tudor's; never trigger it.
|
||
- Never push to `main`. Commits end with the Co-Authored-By trailer the executing session's harness specifies; PR bodies end with the Claude Code line. Do not start a local app server.
|
||
|
||
## Review Focus
|
||
|
||
1. A school whose GIAS inspectorate is ISI **and** that Ofsted also lists (dual registration) must show "Inspected by ISI" only when there is no Ofsted inspection record; an Ofsted record always wins. Pinned in Task 11 (`an Ofsted record wins over ISI`).
|
||
2. An FE provider whose report card predates its previous-framework inspection (bad data) must not produce a row with both a grade and a report card (that breaks `assert_ofsted_current_grade_consistent`). Pinned in Task 5 (`int_ofsted_fe_report_card_older_than_grade_is_ignored`).
|
||
3. A UKPRN shared by an FE provider and a URN that Ofsted also lists under its own URN must take the URN match. Pinned in Task 5 (`int_ofsted_fe_prefers_the_urn_match`).
|
||
4. The FE "as at" file missing from the newest academic-year page (a new page with no file yet) must fall back to the previous page, not emit nothing. Pinned in Task 3 (`test_providers_fall_back_to_the_previous_page`).
|
||
5. A school-page or compare render for a school with `ofsted_listed` null (marts built before PR 1) must behave as "not listed": no "Not yet inspected". Pinned in Task 11 (null case) and Task 12.
|
||
|
||
## File Map
|
||
|
||
**PR 1 — branch `fix/ofsted-inspectorates-pipeline`** (exists; spec `5f7528b`)
|
||
|
||
- Modify `pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py` (two fields).
|
||
- Create `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py` (page/file selection, CSV reading, column maps — pandas-only).
|
||
- Modify `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (two streams).
|
||
- Create `pipeline/tests/test_gias_declared_fields.py`, `pipeline/tests/test_ofsted_fe_mi.py`.
|
||
- Modify `pipeline/transform/models/staging/stg_gias_establishments.sql`, `_stg_sources.yml`.
|
||
- Create `pipeline/transform/models/staging/stg_gias_ukprn.sql`, `stg_ofsted_fe_providers.sql`, `stg_ofsted_fe_inspections.sql`, `stg_ofsted_fe.yml`.
|
||
- Create `pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`, `int_ofsted_fe_latest.yml`, `int_ofsted_listed.sql`, `int_ofsted_listed.yml`.
|
||
- Modify `pipeline/transform/models/intermediate/int_ofsted_latest.sql`, `int_ofsted_latest.yml`.
|
||
- Modify `pipeline/transform/models/marts/dim_school.sql`, `dim_school.yml`, `_marts_schema.yml`.
|
||
- Create `pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql`.
|
||
- Modify `pipeline/dags/school_data_pipeline.py` (daily and monthly selectors), `pipeline/tests/test_dag_selectors.py` (`OPTIONAL_PARENTS`).
|
||
|
||
**PR 2 — branch `fix/ofsted-inspectorates-site`** (from `origin/main` after PR 1 merges; touches no PR 1 file)
|
||
|
||
- Modify `backend/schemas.py` (`SCHOOL_COLUMNS`), `backend/data_loader.py` (main query, optional columns), `backend/app.py` (detail and compare `school_info`), `backend/ofsted_codes.py` (`ofsted_page_url`).
|
||
- Create `backend/tests/test_inspection_status_payload.py`; modify `backend/tests/test_ofsted_codes.py`, `backend/tests/test_supplementary_enrichment.py`.
|
||
- Modify `nextjs-app/lib/types.ts`, `nextjs-app/lib/ofstedStatus.ts`, `nextjs-app/lib/utils.ts`, `nextjs-app/lib/compareLogic.ts`.
|
||
- Modify `nextjs-app/components/SchoolRow.tsx`, `SecondarySchoolRow.tsx`, `LeafletMapInner.tsx`, `school/OfstedSection.tsx`, `school/PrimarySchoolSections.tsx`, `school/SecondarySchoolSections.tsx`, `compare/CompareOfsted.tsx`, `compare/CompareAtAGlance.tsx`.
|
||
- Create `nextjs-app/components/school/InspectionNotice.tsx`.
|
||
- Modify tests: `nextjs-app/__tests__/lib/utils.test.ts`, `__tests__/lib/ofstedStatus.test.ts`, `__tests__/lib/compareLogic.test.ts`, `__tests__/components/OfstedSection.test.tsx`, `__tests__/components/CompareOfsted.test.tsx`, `__tests__/components/SchoolRow.test.tsx`; create `__tests__/components/InspectionNotice.test.tsx`.
|
||
- Modify `e2e/tests/journeys.spec.ts`.
|
||
|
||
**Commands used throughout.** `$SCRATCH` is the session scratchpad directory. Python tests run in CI's environment:
|
||
|
||
```bash
|
||
uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest <paths> -q
|
||
```
|
||
|
||
Local tap commands use `--with-editable` (a plain `--with ./path` reuses a cached build of the unchanged version and ignores edits). dbt runs through `$SCRATCH/dbt_local/dbt.sh` (Task 1).
|
||
|
||
---
|
||
|
||
## PR 1 — pipeline
|
||
|
||
### Task 1: Local dbt harness
|
||
|
||
`$SCRATCH/dbt_local/pgdata` survives from earlier work; its helper scripts do not.
|
||
|
||
**Files:** none in the repo.
|
||
|
||
- [ ] **Step 1: Recreate the helpers**
|
||
|
||
```bash
|
||
mkdir -p "$SCRATCH/dbt_local" && cd "$SCRATCH/dbt_local"
|
||
cat > start_pg.py <<'EOF'
|
||
import pgserver, sys
|
||
srv = pgserver.get_server(sys.argv[1], cleanup_mode=None)
|
||
print(srv.get_uri())
|
||
if "school_compare" not in srv.psql("select datname from pg_database;"):
|
||
srv.psql("create database school_compare;")
|
||
EOF
|
||
uv run --no-project --with pgserver python start_pg.py "$SCRATCH/dbt_local/pgdata"
|
||
```
|
||
|
||
Expected: a URI ending `?host=<socket dir>`. Then, with that socket dir:
|
||
|
||
```bash
|
||
cat > "$SCRATCH/dbt_local/dbt.sh" <<EOF
|
||
#!/bin/bash
|
||
# Run dbt against the throwaway pgserver database. Usage: dbt.sh <dbt args...>
|
||
export PG_HOST=<socket dir> PG_PORT=5432 PG_USER=postgres PG_PASSWORD= PG_DATABASE=school_compare
|
||
D=$SCRATCH/dbt_local
|
||
cd /Users/tudor/projects/school_compare/pipeline/transform
|
||
uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main "\$@" --profiles-dir . --target dev --target-path \$D/target --log-path \$D/logs > \$D/last.log 2>&1
|
||
status=\$?
|
||
sed 's/\x1b\[[0-9;]*m//g' \$D/last.log | grep -E "PASS|FAIL|ERROR|WARN |OK created|Done\.|Error|Compilation|Invalid"
|
||
exit \$status
|
||
EOF
|
||
cat > "$SCRATCH/dbt_local/sql.sh" <<'EOF'
|
||
#!/bin/bash
|
||
# Run SQL from stdin against the local school_compare database; print the output.
|
||
D=$(cd "$(dirname "$0")" && pwd)
|
||
SQL=$(cat) D="$D" uv run --no-project --with pgserver python -c '
|
||
import os, pgserver
|
||
srv = pgserver.get_server(os.environ["D"] + "/pgdata", cleanup_mode=None)
|
||
print(srv.psql("\\c school_compare\n" + os.environ["SQL"]))'
|
||
EOF
|
||
cat > "$SCRATCH/dbt_local/raw_tables.py" <<'EOF'
|
||
"""Create raw.<stream> with a text column per declared field (what target-postgres loads).
|
||
Usage: raw_tables.py <module>:<TapClass> <stream> [<stream> ...]"""
|
||
import importlib, os, sys, pgserver
|
||
module, cls = sys.argv[1].split(":")
|
||
tap = getattr(importlib.import_module(module), cls)(config={})
|
||
wanted = set(sys.argv[2:])
|
||
stmts = ["create schema if not exists raw;"]
|
||
for stream in tap.streams.values():
|
||
if stream.name in wanted:
|
||
cols = ", ".join(f'"{c}" text' for c in stream.schema["properties"])
|
||
stmts.append(f'drop table if exists raw.{stream.name} cascade; create table raw.{stream.name} ({cols});')
|
||
print("raw." + stream.name)
|
||
srv = pgserver.get_server(os.environ["SCRATCH"] + "/dbt_local/pgdata", cleanup_mode=None)
|
||
srv.psql("\\c school_compare\n" + "\n".join(stmts))
|
||
EOF
|
||
chmod +x "$SCRATCH/dbt_local/dbt.sh" "$SCRATCH/dbt_local/sql.sh"
|
||
```
|
||
|
||
- [ ] **Step 2: Raw tables for GIAS and the school MI**
|
||
|
||
```bash
|
||
cd /Users/tudor/projects/school_compare
|
||
SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-gias --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_gias.tap:TapUKGIAS gias_establishments gias_links
|
||
SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_ofsted.tap:TapUKOfsted ofsted_inspections
|
||
```
|
||
|
||
Expected: `raw.gias_establishments`, `raw.gias_links`, `raw.ofsted_inspections`.
|
||
|
||
- [ ] **Step 3: Baseline build**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_ofsted_inspections int_ofsted_latest fact_ofsted_latest dim_school`
|
||
Expected: exit 0; the 10 `int_ofsted_latest` and 2 `dim_school` unit tests PASS on empty tables. If `pgserver` cannot start, stop and report: dbt tests are then proven only by the staging DAG runs, and the PR description must say so.
|
||
|
||
### Task 2: GIAS inspectorate and UKPRN
|
||
|
||
**Files:**
|
||
- Modify: `pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py` (after `th.Property("StatutoryHighAge", th.StringType),`)
|
||
- Modify: `pipeline/transform/models/staging/stg_gias_establishments.sql`
|
||
- Create: `pipeline/transform/models/staging/stg_gias_ukprn.sql`
|
||
- Modify: `pipeline/dags/school_data_pipeline.py` (daily `dbt_build` selection)
|
||
- Test: `pipeline/tests/test_gias_declared_fields.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: raw columns `"UKPRN"`, `"InspectorateName (name)"`; `stg_gias_establishments.inspectorate_name text`; `stg_gias_ukprn(urn integer, ukprn text)`.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
`pipeline/tests/test_gias_declared_fields.py`:
|
||
|
||
```python
|
||
"""Every field the GIAS establishments stream declares must be a column of the
|
||
extract GIAS publishes, or it silently loads as null. H4 adds UKPRN (matches
|
||
Ofsted's further-education providers) and InspectorateName (name) ("ISI" for
|
||
about 1,340 independent schools). HEADER is GIAS's own, from
|
||
edubasealldata20261013.csv.
|
||
"""
|
||
import re
|
||
from pathlib import Path
|
||
|
||
TAP = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-gias'
|
||
/ 'tap_uk_gias' / 'tap.py')
|
||
|
||
HEADER = (
|
||
'URN', 'LA (code)', 'LA (name)', 'EstablishmentNumber', 'EstablishmentName',
|
||
'TypeOfEstablishment (code)', 'TypeOfEstablishment (name)', 'EstablishmentTypeGroup (code)',
|
||
'EstablishmentTypeGroup (name)', 'EstablishmentStatus (code)', 'EstablishmentStatus (name)',
|
||
'ReasonEstablishmentOpened (code)', 'ReasonEstablishmentOpened (name)', 'OpenDate',
|
||
'ReasonEstablishmentClosed (code)', 'ReasonEstablishmentClosed (name)', 'CloseDate',
|
||
'PhaseOfEducation (code)', 'PhaseOfEducation (name)', 'StatutoryLowAge', 'StatutoryHighAge',
|
||
'Boarders (code)', 'Boarders (name)', 'NurseryProvision (name)', 'OfficialSixthForm (code)',
|
||
'OfficialSixthForm (name)', 'Gender (code)', 'Gender (name)', 'ReligiousCharacter (code)',
|
||
'ReligiousCharacter (name)', 'ReligiousEthos (name)', 'Diocese (code)', 'Diocese (name)',
|
||
'AdmissionsPolicy (code)', 'AdmissionsPolicy (name)', 'SchoolCapacity', 'SpecialClasses (code)',
|
||
'SpecialClasses (name)', 'CensusDate', 'NumberOfPupils', 'NumberOfBoys', 'NumberOfGirls',
|
||
'PercentageFSM', 'TrustSchoolFlag (code)', 'TrustSchoolFlag (name)', 'Trusts (code)',
|
||
'Trusts (name)', 'SchoolSponsorFlag (name)', 'SchoolSponsors (name)', 'FederationFlag (name)',
|
||
'Federations (code)', 'Federations (name)', 'UKPRN', 'FEHEIdentifier',
|
||
'FurtherEducationType (name)', 'LastChangedDate', 'Street', 'Locality', 'Address3', 'Town',
|
||
'County (name)', 'Postcode', 'SchoolWebsite', 'TelephoneNum', 'HeadTitle (name)',
|
||
'HeadFirstName', 'HeadLastName', 'HeadPreferredJobTitle', 'BSOInspectorateName (name)',
|
||
'InspectorateReport', 'DateOfLastInspectionVisit', 'NextInspectionVisit', 'TeenMoth (name)',
|
||
'TeenMothPlaces', 'CCF (name)', 'SENPRU (name)', 'EBD (name)', 'PlacesPRU', 'FTProv (name)',
|
||
'EdByOther (name)', 'Section41Approved (name)', 'SEN1 (name)', 'SEN2 (name)', 'SEN3 (name)',
|
||
'SEN4 (name)', 'SEN5 (name)', 'SEN6 (name)', 'SEN7 (name)', 'SEN8 (name)', 'SEN9 (name)',
|
||
'SEN10 (name)', 'SEN11 (name)', 'SEN12 (name)', 'SEN13 (name)',
|
||
'TypeOfResourcedProvision (name)', 'ResourcedProvisionOnRoll', 'ResourcedProvisionCapacity',
|
||
'SenUnitOnRoll', 'SenUnitCapacity', 'GOR (code)', 'GOR (name)',
|
||
'DistrictAdministrative (code)', 'DistrictAdministrative (name)', 'AdministrativeWard (code)',
|
||
'AdministrativeWard (name)', 'ParliamentaryConstituency (code)',
|
||
'ParliamentaryConstituency (name)', 'UrbanRural (code)', 'UrbanRural (name)',
|
||
'GSSLACode (name)', 'Easting', 'Northing', 'MSOA (name)', 'LSOA (name)',
|
||
'InspectorateName (name)', 'SENStat', 'SENNoStat', 'BoardingEstablishment (name)', 'PropsName',
|
||
'PreviousLA (code)', 'PreviousLA (name)', 'PreviousEstablishmentNumber', 'Country (name)',
|
||
'UPRN', 'SiteName', 'QABName (code)', 'QABName (name)', 'EstablishmentAccredited (code)',
|
||
'EstablishmentAccredited (name)', 'QABReport', 'CHNumber', 'MSOA (code)', 'LSOA (code)', 'FSM',
|
||
'AccreditationExpiryDate',
|
||
)
|
||
|
||
|
||
def declared():
|
||
source = TAP.read_text()
|
||
block = source[source.index('class GIASEstablishmentsStream'):source.index('class GIASLinksStream')]
|
||
return re.findall(r'th\.Property\("([^"]+)"', block)
|
||
|
||
|
||
def test_every_declared_field_is_a_gias_column():
|
||
assert set(declared()) - set(HEADER) == set()
|
||
|
||
|
||
def test_the_inspection_fields_are_declared():
|
||
assert {'UKPRN', 'InspectorateName (name)'} <= set(declared())
|
||
```
|
||
|
||
- [ ] **Step 2: Run it to see it fail**
|
||
|
||
Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_gias_declared_fields.py -q`
|
||
Expected: `test_the_inspection_fields_are_declared` FAILS; `test_every_declared_field_is_a_gias_column` passes (a guard for existing fields).
|
||
|
||
- [ ] **Step 3: Declare the fields, stage them, add the UKPRN view**
|
||
|
||
In `tap_uk_gias/tap.py`, after `th.Property("StatutoryHighAge", th.StringType),`:
|
||
|
||
```python
|
||
# Matches Ofsted's further-education providers, which some FE
|
||
# records name by UKPRN only (audit H4).
|
||
th.Property("UKPRN", th.StringType),
|
||
# "ISI" for independent schools the Independent Schools Inspectorate
|
||
# inspects; "Ofsted" or "Not applicable" otherwise (audit H4).
|
||
th.Property("InspectorateName (name)", th.StringType),
|
||
```
|
||
|
||
In `stg_gias_establishments.sql`, after the line `"NurseryProvision (name)" as nursery_provision,`:
|
||
|
||
```sql
|
||
nullif(trim("InspectorateName (name)"), '') as inspectorate_name,
|
||
```
|
||
|
||
`pipeline/transform/models/staging/stg_gias_ukprn.sql`:
|
||
|
||
```sql
|
||
-- URN → UKPRN from the raw GIAS extract, for matching Ofsted's further
|
||
-- education providers (int_ofsted_fe_latest, int_ofsted_listed; audit H4).
|
||
-- A model on the raw source rather than a column of stg_gias_establishments:
|
||
-- a ref to that model would put the Ofsted models into the daily build's
|
||
-- stg_gias_establishments+ selection (pipeline/tests/test_dag_selectors.py).
|
||
|
||
select
|
||
cast("URN" as integer) as urn,
|
||
nullif(trim("UKPRN"), '') as ukprn
|
||
from {{ source('raw', 'gias_establishments') }}
|
||
where "URN" ~ '^[0-9]+$'
|
||
```
|
||
|
||
In `pipeline/dags/school_data_pipeline.py`, in the daily build, change `--select stg_gias_establishments+ stg_gias_links+ gias_code_names+` to `--select stg_gias_establishments+ stg_gias_links+ gias_code_names+ stg_gias_ukprn`.
|
||
|
||
- [ ] **Step 4: Run the tests and build**
|
||
|
||
Run: the Step 2 pytest command → `2 passed`.
|
||
Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_dag_selectors.py -q` → all pass.
|
||
Re-create the GIAS raw tables (Task 1 Step 2, first command) → the table now has `"UKPRN"` and `"InspectorateName (name)"`.
|
||
Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_gias_ukprn dim_school` → exit 0.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add pipeline/tests/test_gias_declared_fields.py pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py pipeline/transform/models/staging/stg_gias_establishments.sql pipeline/transform/models/staging/stg_gias_ukprn.sql pipeline/dags/school_data_pipeline.py
|
||
git commit -m "feat(gias): load UKPRN and the inspectorate GIAS names (H4)"
|
||
```
|
||
|
||
### Task 3: Ofsted further-education extractor
|
||
|
||
**Files:**
|
||
- Create: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py`
|
||
- Modify: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (imports, two stream classes, `discover_streams`)
|
||
- Test: `pipeline/tests/test_ofsted_fe_mi.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `fe_pages(results: list[dict]) -> list[str]`; `renewed_framework_pages(results: list[dict]) -> list[str]`; `latest_as_at_csv(html: str) -> str | None`; `latest_period_csv(html: str) -> str | None`; `read_mi_csv(content: bytes) -> pd.DataFrame`; `records(df, columns: dict[str, str], key: str) -> Iterator[dict]`; `providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None`; `PROVIDER_COLUMNS`, `INSPECTION_COLUMNS`. Streams `ofsted_fe_providers` (key `provider_urn`; fields `provider_urn, provider_ukprn, provider_name, provider_type, renewed_eif_full_inspection, graded_inspection_date, overall_effectiveness, short_inspection_date`) and `ofsted_fe_inspections` (key `inspection_number`; fields `inspection_number, provider_urn, provider_ukprn, first_day_of_inspection, date_published, safeguarding_standards`), all strings.
|
||
|
||
- [ ] **Step 1: Write the failing tests**
|
||
|
||
`pipeline/tests/test_ofsted_fe_mi.py`:
|
||
|
||
```python
|
||
"""Ofsted publishes further-education inspection outcomes on a gov.uk page per
|
||
academic year, separate from the school MI the tap already reads (audit H4).
|
||
The links below are the real ones from the 2025/26 page; the headers are
|
||
Ofsted's own.
|
||
"""
|
||
import importlib.util
|
||
from pathlib import Path
|
||
|
||
import pandas as pd
|
||
import pytest
|
||
|
||
MODULE = (Path(__file__).resolve().parents[1] / 'plugins' / 'extractors' / 'tap-uk-ofsted'
|
||
/ 'tap_uk_ofsted' / 'fe_mi.py')
|
||
|
||
A = 'https://assets.publishing.service.gov.uk/media/'
|
||
PAGE_2025_26 = f'''
|
||
<a href="{A}6989b16dc106cc5501b6e4c3/Management_information_-_further_education_and_skills_-_as_at_31_January_2026.csv">x</a>
|
||
<a href="{A}6989b18cd3f57710b50a9b92/Management_information_-_further_education_and_skills_-_10_November_2025_to_31_January_2026.csv">x</a>
|
||
<a href="{A}69aec8d6d620c14fa183eeb6/Management_information_-_further_education_and_skills_-__10_November_2025_to_28_February_2026.csv">x</a>
|
||
<a href="{A}69aec8fcc78869bf8eb8a54a/Management_information_-_further_education_and_skills_-_as_at_28_February_2026.csv">x</a>
|
||
<a href="{A}69aef091bde9c3f213c89a1e/Management_information_-_further_education_and_skills_-_as_at_31_January_2026.ods">x</a>
|
||
<a href="{A}6a9fec1e92e72b8ac437eef3/Management_information_-_further_education_and_skills_-_as_at_31_August_2026.csv">x</a>
|
||
<a href="{A}6a9fed6b5f6e942efe37eed4/Management_information_-_new_provider_monitoring_inspections_-_10_November_2025_to_31_August_2026.csv">x</a>
|
||
<a href="{A}6a9fed815f6e942efe37eed5/Management_information_-_monitoring_inspections_-_10_November_2025_to_31_August_2026.csv">x</a>
|
||
<a href="{A}6a9feda65f6e942efe37eed6/Management_information_-_further_education_and_skills_-_10_November_2025_to_31_August_2026.csv">x</a>
|
||
'''
|
||
|
||
SEARCH_RESULTS = [
|
||
{'title': 'Further education and skills inspections and outcomes: management information from September 2024 to August 2025',
|
||
'link': '/government/statistical-data-sets/fe-2024-25'},
|
||
{'title': 'Further education and skills inspections and outcomes: management information from November 2025 to August 2026',
|
||
'link': '/government/statistical-data-sets/fe-2025-26'},
|
||
{'title': 'Further education and skills inspections and outcomes: management information from September 2023 to August 2024',
|
||
'link': '/government/statistical-data-sets/fe-2023-24'},
|
||
{'title': 'Monthly management information: Ofsted\'s school inspections outcomes',
|
||
'link': '/government/statistical-data-sets/schools'},
|
||
]
|
||
|
||
PROVIDER_HEADER = (
|
||
'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group',
|
||
'Local authority', 'Region', 'Ofsted region',
|
||
'Has the provider received a full inspection under the renewed EIF?',
|
||
'Has the provider received a new provider monitoring inspection?',
|
||
'Date of latest short inspection before renewed EIF (if relevant)',
|
||
'Inspection number of latest full inspection before renewed EIF',
|
||
'First day of inspection of latest inspection before renewed EIF',
|
||
'Latest overall effectiveness before renewed EIF',
|
||
'Latest quality of education before renewed EIF',
|
||
'Latest behaviour and attitudes before renewed EIF',
|
||
'Latest personal development before renewed EIF',
|
||
'Latest leadership and management before renewed EIF',
|
||
'Latest contribution to meeting skills needs before renewed EIF',
|
||
'Latest safeguarding before renewed EIF', '',
|
||
)
|
||
|
||
INSPECTION_HEADER = (
|
||
'Provider URN', 'Provider UKPRN', 'Provider name', 'Provider type', 'Provider group',
|
||
'Local authority', 'Region', 'Ofsted region', 'Inspection number', 'First day of inspection',
|
||
'Date published', 'Safeguarding standards', 'Inclusion', 'Leadership and governance',
|
||
'Contribution to meeting skills needs',
|
||
'Education programmes for young people - curriculum, teaching and training',
|
||
'Education programmes for young people - achievement',
|
||
'Education programmes for young people - participation and development',
|
||
'Adult learning programmes - curriculum, teaching and training',
|
||
'Adult learning programmes - achievement',
|
||
'Adult learning programmes - participation and development',
|
||
'Apprenticeships - curriculum, teaching and training', 'Apprenticeships - achievement',
|
||
'Apprenticeships - participation and development',
|
||
'Provision for learners and apprentices with high needs - curriculum, teaching and training',
|
||
'Provision for learners and apprentices with high needs - achievement',
|
||
'Provision for learners and apprentices with high needs - participation and development', '',
|
||
)
|
||
|
||
|
||
@pytest.fixture
|
||
def fe():
|
||
spec = importlib.util.spec_from_file_location('fe_mi', MODULE)
|
||
module = importlib.util.module_from_spec(spec)
|
||
spec.loader.exec_module(module)
|
||
return module
|
||
|
||
|
||
def test_the_newest_as_at_file_is_chosen(fe):
|
||
assert fe.latest_as_at_csv(PAGE_2025_26).endswith('_as_at_31_August_2026.csv')
|
||
|
||
|
||
def test_the_latest_cumulative_inspections_file_is_chosen_not_a_monitoring_one(fe):
|
||
assert fe.latest_period_csv(PAGE_2025_26) == (
|
||
A + '6a9feda65f6e942efe37eed6/Management_information_-_further_education_and_skills_'
|
||
'-_10_November_2025_to_31_August_2026.csv')
|
||
|
||
|
||
def test_a_doubled_underscore_file_name_still_parses(fe):
|
||
page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'August' not in line)
|
||
assert fe.latest_period_csv(page).endswith('__10_November_2025_to_28_February_2026.csv')
|
||
|
||
|
||
def test_monitoring_files_alone_give_nothing(fe):
|
||
page = '\n'.join(line for line in PAGE_2025_26.splitlines() if 'monitoring' in line)
|
||
assert fe.latest_period_csv(page) is None
|
||
assert fe.latest_as_at_csv(page) is None
|
||
|
||
|
||
def test_pages_are_the_fe_series_newest_first(fe):
|
||
assert fe.fe_pages(SEARCH_RESULTS) == [
|
||
'/government/statistical-data-sets/fe-2025-26',
|
||
'/government/statistical-data-sets/fe-2024-25',
|
||
'/government/statistical-data-sets/fe-2023-24',
|
||
]
|
||
|
||
|
||
def test_report_cards_are_read_only_from_renewed_framework_pages(fe):
|
||
# Earlier pages hold previous-framework inspection lists, not report cards.
|
||
assert fe.renewed_framework_pages(SEARCH_RESULTS) == ['/government/statistical-data-sets/fe-2025-26']
|
||
|
||
|
||
def test_providers_fall_back_to_the_previous_page(fe):
|
||
pages = {'/new': '<p>No files yet</p>', '/old': PAGE_2025_26}
|
||
assert fe.providers_csv_url(['/new', '/old'], pages.__getitem__).endswith('_as_at_31_August_2026.csv')
|
||
|
||
|
||
@pytest.mark.parametrize('columns, header', [('PROVIDER_COLUMNS', PROVIDER_HEADER),
|
||
('INSPECTION_COLUMNS', INSPECTION_HEADER)])
|
||
def test_every_mapped_column_is_in_ofsteds_file(fe, columns, header):
|
||
assert set(getattr(fe, columns)) - set(header) == set()
|
||
|
||
|
||
def test_csv_reading_and_records(fe):
|
||
content = ('' + ','.join(f'"{c}"' for c in PROVIDER_HEADER[:14]) + '\n'
|
||
+ '145175,10005429,Queen Mary\'s College,Sixth form college,Sixth form colleges,Hampshire,'
|
||
'South East,South East,No,No,-,10227183,06/12/2022,2\n'
|
||
+ ',,Blank provider,,,,,,,,,,,\n').encode('utf-8')
|
||
rows = list(fe.records(fe.read_mi_csv(content), fe.PROVIDER_COLUMNS, 'provider_urn'))
|
||
assert rows == [{
|
||
'provider_urn': '145175', 'provider_ukprn': '10005429', 'provider_name': "Queen Mary's College",
|
||
'provider_type': 'Sixth form college', 'renewed_eif_full_inspection': 'No',
|
||
'graded_inspection_date': '06/12/2022', 'overall_effectiveness': '2', 'short_inspection_date': '-',
|
||
}]
|
||
```
|
||
|
||
- [ ] **Step 2: Run them to see them fail**
|
||
|
||
Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests/test_ofsted_fe_mi.py -q`
|
||
Expected: every test errors with `FileNotFoundError` for `fe_mi.py`.
|
||
|
||
- [ ] **Step 3: Write the module**
|
||
|
||
`pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py`:
|
||
|
||
```python
|
||
"""Ofsted's further education and skills management information (audit H4).
|
||
|
||
Ofsted publishes FE inspection outcomes on one gov.uk page per academic year
|
||
("Further education and skills inspections and outcomes: management
|
||
information from November 2025 to August 2026"). Each page carries "as at"
|
||
files (one row per provider, with its latest inspection under the previous
|
||
framework) and cumulative files of renewed-framework inspections, i.e. report
|
||
cards ("…_10_November_2025_to_31_January_2026.csv", "…_to_28_February_2026.csv",
|
||
…). Monitoring-visit files are a different series.
|
||
|
||
Free of the Singer SDK so CI's pytest, which installs only the backend's
|
||
requirements, can load it.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import io
|
||
import re
|
||
from datetime import date
|
||
from typing import Callable, Iterator
|
||
|
||
import pandas as pd
|
||
|
||
SEARCH_URL = "https://www.gov.uk/api/search.json"
|
||
SEARCH_QUERY = "further education and skills inspections and outcomes management information"
|
||
PAGE_TITLE_PREFIX = "Further education and skills inspections and outcomes: management information from "
|
||
# Renewed-framework (report card) inspections began on 10 November 2025.
|
||
RENEWED_FRAMEWORK_START = date(2025, 11, 1)
|
||
|
||
_MONTHS = {name: number for number, name in enumerate(
|
||
["january", "february", "march", "april", "may", "june", "july",
|
||
"august", "september", "october", "november", "december"], start=1)}
|
||
_TITLE_START = re.compile(r"from ([A-Za-z]+) (\d{4})")
|
||
_CSV_LINK = re.compile(r'https://assets\.publishing\.service\.gov\.uk/[^"\s<>]+\.csv')
|
||
_AS_AT = re.compile(r"further_education_and_skills_-_+as_at_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$")
|
||
_PERIOD = re.compile(
|
||
r"further_education_and_skills_-_+(\d{1,2})_([A-Za-z]+)_(\d{4})_to_(\d{1,2})_([A-Za-z]+)_(\d{4})\.csv$")
|
||
|
||
# Ofsted column → Singer field.
|
||
PROVIDER_COLUMNS = {
|
||
"Provider URN": "provider_urn",
|
||
"Provider UKPRN": "provider_ukprn",
|
||
"Provider name": "provider_name",
|
||
"Provider type": "provider_type",
|
||
"Has the provider received a full inspection under the renewed EIF?": "renewed_eif_full_inspection",
|
||
"First day of inspection of latest inspection before renewed EIF": "graded_inspection_date",
|
||
"Latest overall effectiveness before renewed EIF": "overall_effectiveness",
|
||
"Date of latest short inspection before renewed EIF (if relevant)": "short_inspection_date",
|
||
}
|
||
INSPECTION_COLUMNS = {
|
||
"Provider URN": "provider_urn",
|
||
"Provider UKPRN": "provider_ukprn",
|
||
"Inspection number": "inspection_number",
|
||
"First day of inspection": "first_day_of_inspection",
|
||
"Date published": "date_published",
|
||
"Safeguarding standards": "safeguarding_standards",
|
||
}
|
||
|
||
|
||
def _date(day: str, month: str, year: str) -> date | None:
|
||
number = _MONTHS.get(month.lower())
|
||
return date(int(year), number, int(day)) if number else None
|
||
|
||
|
||
def _page_start(title: str) -> date | None:
|
||
match = _TITLE_START.search(title)
|
||
number = _MONTHS.get(match.group(1).lower()) if match else None
|
||
return date(int(match.group(2)), number, 1) if number else None
|
||
|
||
|
||
def _dated_pages(results: list[dict]) -> list[tuple[date, str]]:
|
||
pages = []
|
||
for result in results:
|
||
title = result.get("title", "")
|
||
start = _page_start(title) if title.startswith(PAGE_TITLE_PREFIX) else None
|
||
if start:
|
||
pages.append((start, result["link"]))
|
||
return sorted(pages, reverse=True)
|
||
|
||
|
||
def fe_pages(results: list[dict]) -> list[str]:
|
||
"""gov.uk paths of the FE MI pages, newest academic year first."""
|
||
return [link for _, link in _dated_pages(results)]
|
||
|
||
|
||
def renewed_framework_pages(results: list[dict]) -> list[str]:
|
||
"""The pages that can hold report cards: from November 2025 onwards."""
|
||
return [link for start, link in _dated_pages(results) if start >= RENEWED_FRAMEWORK_START]
|
||
|
||
|
||
def _latest(html: str, pattern: re.Pattern, date_groups: slice) -> str | None:
|
||
best = None
|
||
for link in set(_CSV_LINK.findall(html)):
|
||
match = pattern.search(link)
|
||
when = _date(*match.groups()[date_groups]) if match else None
|
||
if when and (best is None or when > best[0]):
|
||
best = (when, link)
|
||
return best[1] if best else None
|
||
|
||
|
||
def latest_as_at_csv(html: str) -> str | None:
|
||
"""The newest "as at" CSV on a page."""
|
||
return _latest(html, _AS_AT, slice(0, 3))
|
||
|
||
|
||
def latest_period_csv(html: str) -> str | None:
|
||
"""The renewed-framework inspections CSV with the latest end date on a page."""
|
||
return _latest(html, _PERIOD, slice(3, 6))
|
||
|
||
|
||
def providers_csv_url(pages: list[str], fetch: Callable[[str], str]) -> str | None:
|
||
"""The newest "as at" CSV, from the newest page that has one."""
|
||
for page in pages:
|
||
url = latest_as_at_csv(fetch(page))
|
||
if url:
|
||
return url
|
||
return None
|
||
|
||
|
||
def read_mi_csv(content: bytes) -> pd.DataFrame:
|
||
"""Ofsted's MI CSVs: UTF-8 with a BOM, or Windows-1252. The header is the
|
||
line naming "Provider URN"."""
|
||
try:
|
||
text = content.decode("utf-8-sig")
|
||
except UnicodeDecodeError:
|
||
text = content.decode("cp1252")
|
||
lines = text.splitlines()
|
||
start = next((i for i, line in enumerate(lines[:20]) if "Provider URN" in line), 0)
|
||
return pd.read_csv(io.StringIO("\n".join(lines[start:])), dtype=str, keep_default_na=False)
|
||
|
||
|
||
def records(df: pd.DataFrame, columns: dict[str, str], key: str) -> Iterator[dict]:
|
||
"""Singer records: the mapped columns, stripped. Rows without `key` are skipped."""
|
||
present = {source: field for source, field in columns.items() if source in df.columns}
|
||
for _, row in df.iterrows():
|
||
record = {field: str(row[source]).strip() for source, field in present.items()}
|
||
if record.get(key):
|
||
yield record
|
||
```
|
||
|
||
- [ ] **Step 4: Add the two streams**
|
||
|
||
In `tap_uk_ofsted/tap.py`, after `from singer_sdk import typing as th`:
|
||
|
||
```python
|
||
|
||
from tap_uk_ofsted.fe_mi import (
|
||
INSPECTION_COLUMNS,
|
||
PROVIDER_COLUMNS,
|
||
SEARCH_QUERY,
|
||
SEARCH_URL,
|
||
fe_pages,
|
||
latest_period_csv,
|
||
providers_csv_url,
|
||
read_mi_csv,
|
||
records,
|
||
renewed_framework_pages,
|
||
)
|
||
```
|
||
|
||
Before `class TapUKOfsted(Tap):`:
|
||
|
||
```python
|
||
def _fe_search_results() -> list[dict]:
|
||
"""gov.uk search results for Ofsted's FE MI pages (one per academic year)."""
|
||
resp = requests.get(SEARCH_URL, params={"q": SEARCH_QUERY, "count": 50}, timeout=60)
|
||
resp.raise_for_status()
|
||
return resp.json().get("results", [])
|
||
|
||
|
||
def _gov_uk_page(path: str) -> str:
|
||
url = path if path.startswith("http") else "https://www.gov.uk" + path
|
||
resp = requests.get(url, timeout=60)
|
||
resp.raise_for_status()
|
||
return resp.text
|
||
|
||
|
||
class OfstedFEProvidersStream(Stream):
|
||
"""Ofsted's further education and skills providers: the newest "as at"
|
||
file, one row per provider with its latest inspection under the previous
|
||
framework (audit H4). Page and file selection: fe_mi.py."""
|
||
|
||
name = "ofsted_fe_providers"
|
||
primary_keys = ["provider_urn"]
|
||
replication_key = None
|
||
|
||
schema = th.PropertiesList(
|
||
th.Property("provider_urn", th.StringType, required=True),
|
||
*[th.Property(field, th.StringType) for field in PROVIDER_COLUMNS.values() if field != "provider_urn"],
|
||
).to_dict()
|
||
|
||
def get_records(self, context):
|
||
url = providers_csv_url(fe_pages(_fe_search_results()), _gov_uk_page)
|
||
if not url:
|
||
self.logger.error("No Ofsted FE 'as at' file found")
|
||
return
|
||
self.logger.info("Downloading Ofsted FE providers MI: %s", url)
|
||
resp = requests.get(url, timeout=120)
|
||
resp.raise_for_status()
|
||
yield from records(read_mi_csv(resp.content), PROVIDER_COLUMNS, "provider_urn")
|
||
|
||
|
||
class OfstedFEInspectionsStream(Stream):
|
||
"""Ofsted's renewed-framework FE inspections (report cards), from the
|
||
latest cumulative file on every page since November 2025, so a report
|
||
card survives the next academic year's page (audit H4)."""
|
||
|
||
name = "ofsted_fe_inspections"
|
||
primary_keys = ["inspection_number"]
|
||
replication_key = None
|
||
|
||
schema = th.PropertiesList(
|
||
th.Property("inspection_number", th.StringType, required=True),
|
||
*[th.Property(field, th.StringType) for field in INSPECTION_COLUMNS.values() if field != "inspection_number"],
|
||
).to_dict()
|
||
|
||
def get_records(self, context):
|
||
for page in renewed_framework_pages(_fe_search_results()):
|
||
url = latest_period_csv(_gov_uk_page(page))
|
||
if not url:
|
||
self.logger.warning("No renewed-framework FE inspections file on %s", page)
|
||
continue
|
||
self.logger.info("Downloading Ofsted FE inspections MI: %s", url)
|
||
resp = requests.get(url, timeout=120)
|
||
resp.raise_for_status()
|
||
yield from records(read_mi_csv(resp.content), INSPECTION_COLUMNS, "inspection_number")
|
||
```
|
||
|
||
Change `discover_streams` to `return [OfstedInspectionsStream(self), OfstedFEProvidersStream(self), OfstedFEInspectionsStream(self)]`.
|
||
|
||
- [ ] **Step 5: Run the tests, then both streams against gov.uk**
|
||
|
||
Run: the Step 2 pytest command → `11 passed`.
|
||
|
||
```bash
|
||
uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted python -c "
|
||
from tap_uk_ofsted.tap import TapUKOfsted
|
||
tap = TapUKOfsted(config={})
|
||
p = list(tap.streams['ofsted_fe_providers'].get_records(None))
|
||
i = list(tap.streams['ofsted_fe_inspections'].get_records(None))
|
||
print(len(p), [r for r in p if r['provider_urn'] in ('145175', '143526', '130645')])
|
||
print(len(i), [r for r in i if r['provider_urn'] == '143526'])"
|
||
```
|
||
|
||
Expected: about 1,979 providers (Ofsted's count as at 31 August 2026; a newer file may differ slightly) with Queen Mary's College graded `2` on `06/12/2022`, The Autism Project – CareTrade `2` on `16/11/2021`, Exeter and North Devon Colleges Group `-`; about 1,472 inspections with 143526 inspected `12/05/2026`, safeguarding `Met`.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/fe_mi.py pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py pipeline/tests/test_ofsted_fe_mi.py
|
||
git commit -m "feat(ofsted): extract Ofsted's further education and skills MI (H4)"
|
||
```
|
||
|
||
### Task 4: FE staging models
|
||
|
||
**Files:**
|
||
- Modify: `pipeline/transform/models/staging/_stg_sources.yml`
|
||
- Create: `pipeline/transform/models/staging/stg_ofsted_fe_providers.sql`, `stg_ofsted_fe_inspections.sql`, `stg_ofsted_fe.yml`
|
||
|
||
**Interfaces:**
|
||
- Consumes: raw `ofsted_fe_providers`, `ofsted_fe_inspections` (Task 3 fields).
|
||
- Produces: `stg_ofsted_fe_providers(provider_urn integer, provider_ukprn text, provider_name text, provider_type text, renewed_eif_full_inspection boolean, graded_inspection_date date, overall_effectiveness integer, short_inspection_date date)`; `stg_ofsted_fe_inspections(inspection_number text, provider_urn integer, provider_ukprn text, rc_inspection_date date, published_date date, rc_safeguarding_met boolean)`.
|
||
|
||
- [ ] **Step 1: Declare the sources, create the raw tables, write the unit tests**
|
||
|
||
In `_stg_sources.yml`, after the `ofsted_inspections` entry:
|
||
|
||
```yaml
|
||
- name: ofsted_fe_providers
|
||
description: Ofsted further education and skills MI, newest "as at" file — one row per provider with its latest inspection under the previous framework
|
||
|
||
- name: ofsted_fe_inspections
|
||
description: Ofsted further education and skills MI — renewed-framework inspections (report cards) since 10 November 2025, one row per inspection
|
||
```
|
||
|
||
Create the raw tables: `SCRATCH="$SCRATCH" uv run --no-project --with-editable ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver python "$SCRATCH/dbt_local/raw_tables.py" tap_uk_ofsted.tap:TapUKOfsted ofsted_fe_providers ofsted_fe_inspections` (from the repo root) → two `raw.…` lines.
|
||
|
||
`pipeline/transform/models/staging/stg_ofsted_fe.yml`:
|
||
|
||
```yaml
|
||
version: 2
|
||
|
||
unit_tests:
|
||
- name: stg_ofsted_fe_providers_parses_grades_and_dates
|
||
description: Ofsted writes "-" for no grade or date; grades are 1-4.
|
||
model: stg_ofsted_fe_providers
|
||
given:
|
||
- input: source('raw', 'ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: '145175', provider_ukprn: '10005429', provider_name: "Queen Mary's College", renewed_eif_full_inspection: 'No', graded_inspection_date: '06/12/2022', overall_effectiveness: '2', short_inspection_date: '-'}
|
||
- {provider_urn: '130645', provider_ukprn: '-', provider_name: 'Exeter and North Devon Colleges Group', renewed_eif_full_inspection: 'No', graded_inspection_date: '-', overall_effectiveness: '-', short_inspection_date: '-'}
|
||
- {provider_urn: '', provider_ukprn: '10000001', provider_name: 'No URN'}
|
||
expect:
|
||
rows:
|
||
- {provider_urn: 145175, provider_ukprn: '10005429', renewed_eif_full_inspection: false, graded_inspection_date: '2022-12-06', overall_effectiveness: 2, short_inspection_date: null}
|
||
- {provider_urn: 130645, provider_ukprn: null, renewed_eif_full_inspection: false, graded_inspection_date: null, overall_effectiveness: null, short_inspection_date: null}
|
||
|
||
- name: stg_ofsted_fe_inspections_parses_dates_and_safeguarding
|
||
model: stg_ofsted_fe_inspections
|
||
given:
|
||
- input: source('raw', 'ofsted_fe_inspections')
|
||
rows:
|
||
- {inspection_number: '10380001', provider_urn: '143526', provider_ukprn: '10000002', first_day_of_inspection: '12/05/2026', date_published: '16/06/2026', safeguarding_standards: 'Met'}
|
||
- {inspection_number: '10380002', provider_urn: '150042', provider_ukprn: '10000003', first_day_of_inspection: '20/01/2026', date_published: '24/02/2026', safeguarding_standards: 'Not met'}
|
||
- {inspection_number: '10380003', provider_urn: '1', provider_ukprn: '', first_day_of_inspection: '', date_published: '', safeguarding_standards: ''}
|
||
expect:
|
||
rows:
|
||
- {inspection_number: '10380001', provider_urn: 143526, rc_inspection_date: '2026-05-12', published_date: '2026-06-16', rc_safeguarding_met: true}
|
||
- {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', published_date: '2026-02-24', rc_safeguarding_met: false}
|
||
```
|
||
|
||
- [ ] **Step 2: Run them to see them fail**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "stg_ofsted_fe_providers stg_ofsted_fe_inspections,test_type:unit"`
|
||
Expected: non-zero exit; parsing fails because the two models do not exist.
|
||
|
||
- [ ] **Step 3: Write the models**
|
||
|
||
`pipeline/transform/models/staging/stg_ofsted_fe_providers.sql`:
|
||
|
||
```sql
|
||
-- Staging model: Ofsted further education and skills providers — the "as at"
|
||
-- MI file, one row per provider with its latest inspection under the previous
|
||
-- framework (audit H4). Ofsted writes '-' for no grade or date.
|
||
|
||
select
|
||
cast(trim(provider_urn) as integer) as provider_urn,
|
||
nullif(nullif(trim(provider_ukprn), ''), '-') as provider_ukprn,
|
||
provider_name,
|
||
provider_type,
|
||
lower(trim(renewed_eif_full_inspection)) = 'yes' as renewed_eif_full_inspection,
|
||
case when trim(graded_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$'
|
||
then to_date(trim(graded_inspection_date), 'DD/MM/YYYY') end as graded_inspection_date,
|
||
case when trim(overall_effectiveness) in ('1', '2', '3', '4')
|
||
then trim(overall_effectiveness)::integer end as overall_effectiveness,
|
||
case when trim(short_inspection_date) ~ '^\d{2}/\d{2}/\d{4}$'
|
||
then to_date(trim(short_inspection_date), 'DD/MM/YYYY') end as short_inspection_date
|
||
from {{ source('raw', 'ofsted_fe_providers') }}
|
||
where trim(provider_urn) ~ '^[0-9]+$'
|
||
```
|
||
|
||
`pipeline/transform/models/staging/stg_ofsted_fe_inspections.sql`:
|
||
|
||
```sql
|
||
-- Staging model: Ofsted's renewed-framework FE inspections (report cards),
|
||
-- one row per inspection (audit H4). Only the headline is kept: the date and
|
||
-- the safeguarding result.
|
||
|
||
select
|
||
trim(inspection_number) as inspection_number,
|
||
cast(trim(provider_urn) as integer) as provider_urn,
|
||
nullif(nullif(trim(provider_ukprn), ''), '-') as provider_ukprn,
|
||
to_date(trim(first_day_of_inspection), 'DD/MM/YYYY') as rc_inspection_date,
|
||
case when trim(date_published) ~ '^\d{2}/\d{2}/\d{4}$'
|
||
then to_date(trim(date_published), 'DD/MM/YYYY') end as published_date,
|
||
case lower(trim(safeguarding_standards))
|
||
when 'met' then true
|
||
when 'not met' then false
|
||
end as rc_safeguarding_met
|
||
from {{ source('raw', 'ofsted_fe_inspections') }}
|
||
where trim(provider_urn) ~ '^[0-9]+$'
|
||
and trim(first_day_of_inspection) ~ '^\d{2}/\d{2}/\d{4}$'
|
||
```
|
||
|
||
- [ ] **Step 4: Run the unit tests and build**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_ofsted_fe_providers stg_ofsted_fe_inspections`
|
||
Expected: exit 0; both unit tests PASS; both views created.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add pipeline/transform/models/staging/_stg_sources.yml pipeline/transform/models/staging/stg_ofsted_fe_providers.sql pipeline/transform/models/staging/stg_ofsted_fe_inspections.sql pipeline/transform/models/staging/stg_ofsted_fe.yml
|
||
git commit -m "feat(dbt): stage Ofsted's further education and skills MI (H4)"
|
||
```
|
||
|
||
### Task 5: `int_ofsted_fe_latest`
|
||
|
||
**Files:**
|
||
- Create: `pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`, `int_ofsted_fe_latest.yml`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `stg_gias_ukprn`, `stg_ofsted_fe_providers`, `stg_ofsted_fe_inspections`.
|
||
- Produces: `int_ofsted_fe_latest` with exactly `int_ofsted_latest`'s columns, in its order and types: `urn, latest_visit_date, latest_visit_kind, latest_visit_outcome, current_grade, current_grade_date, current_grade_basis, graded_inspection_date, ungraded_inspection_date, inspection_date, inspection_type, framework, overall_effectiveness, quality_of_education, behaviour_attitudes, personal_development, leadership_management, early_years_provision, sixth_form_provision, ungraded_outcome, ungraded_grade, rc_safeguarding_met, rc_inclusion, rc_curriculum_teaching, rc_achievement, rc_attendance_behaviour, rc_personal_development, rc_leadership_governance, rc_early_years, rc_sixth_form, rc_inspection_date, report_url`. `latest_visit_kind` ∈ `report_card`, `graded`, `short`.
|
||
|
||
- [ ] **Step 1: Write the unit tests**
|
||
|
||
`pipeline/transform/models/intermediate/int_ofsted_fe_latest.yml`:
|
||
|
||
```yaml
|
||
version: 2
|
||
|
||
unit_tests:
|
||
- name: int_ofsted_fe_grade_only
|
||
description: Queen Mary's College (145175), Good on 6 December 2022, no report card.
|
||
model: int_ofsted_fe_latest
|
||
given:
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 145175, ukprn: '10005429'}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 145175, provider_ukprn: '10005429', graded_inspection_date: '2022-12-06', overall_effectiveness: 2}
|
||
- input: ref('stg_ofsted_fe_inspections')
|
||
rows: []
|
||
expect:
|
||
rows:
|
||
- {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, rc_inspection_date: null, framework: 'Further education and skills'}
|
||
|
||
- name: int_ofsted_fe_short_inspection_is_a_visit_not_a_grade
|
||
model: int_ofsted_fe_latest
|
||
given:
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 1, ukprn: null}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 1, graded_inspection_date: '2019-03-05', overall_effectiveness: 2, short_inspection_date: '2023-02-01'}
|
||
- input: ref('stg_ofsted_fe_inspections')
|
||
rows: []
|
||
expect:
|
||
rows:
|
||
- {urn: 1, latest_visit_date: '2023-02-01', latest_visit_kind: short, current_grade: 2, current_grade_date: '2019-03-05', current_grade_basis: graded}
|
||
|
||
- name: int_ofsted_fe_newer_report_card_wins
|
||
description: The Autism Project - CareTrade (143526). Good 2021; report card 12 May 2026, safeguarding met.
|
||
model: int_ofsted_fe_latest
|
||
given:
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 143526, ukprn: '10000002'}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 143526, provider_ukprn: '10000002', graded_inspection_date: '2021-11-16', overall_effectiveness: 2}
|
||
- input: ref('stg_ofsted_fe_inspections')
|
||
rows:
|
||
- {inspection_number: '10380001', provider_urn: 143526, provider_ukprn: '10000002', rc_inspection_date: '2026-05-12', rc_safeguarding_met: true}
|
||
expect:
|
||
rows:
|
||
- {urn: 143526, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, current_grade: null, current_grade_date: null, current_grade_basis: null, rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, overall_effectiveness: 2}
|
||
|
||
- name: int_ofsted_fe_report_card_only
|
||
model: int_ofsted_fe_latest
|
||
given:
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 150042, ukprn: null}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 150042, graded_inspection_date: null, overall_effectiveness: null}
|
||
- input: ref('stg_ofsted_fe_inspections')
|
||
rows:
|
||
- {inspection_number: '10380002', provider_urn: 150042, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false}
|
||
expect:
|
||
rows:
|
||
- {urn: 150042, latest_visit_date: '2026-01-20', latest_visit_kind: report_card, current_grade: null, rc_inspection_date: '2026-01-20', rc_safeguarding_met: false}
|
||
|
||
- name: int_ofsted_fe_listed_never_inspected_has_no_row
|
||
model: int_ofsted_fe_latest
|
||
given:
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 130645, ukprn: null}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 130645, graded_inspection_date: null, overall_effectiveness: null}
|
||
- input: ref('stg_ofsted_fe_inspections')
|
||
rows: []
|
||
expect:
|
||
rows: []
|
||
|
||
- name: int_ofsted_fe_matches_by_ukprn_when_the_urn_differs
|
||
model: int_ofsted_fe_latest
|
||
given:
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 5, ukprn: '10099999'}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 9999999, provider_ukprn: '10099999', graded_inspection_date: '2024-01-30', overall_effectiveness: 3}
|
||
- input: ref('stg_ofsted_fe_inspections')
|
||
rows: []
|
||
expect:
|
||
rows:
|
||
- {urn: 5, current_grade: 3, current_grade_date: '2024-01-30'}
|
||
|
||
- name: int_ofsted_fe_prefers_the_urn_match
|
||
model: int_ofsted_fe_latest
|
||
given:
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 6, ukprn: '10088888'}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 6, provider_ukprn: '10077777', graded_inspection_date: '2023-05-01', overall_effectiveness: 1}
|
||
- {provider_urn: 8888888, provider_ukprn: '10088888', graded_inspection_date: '2024-05-01', overall_effectiveness: 4}
|
||
- input: ref('stg_ofsted_fe_inspections')
|
||
rows: []
|
||
expect:
|
||
rows:
|
||
- {urn: 6, current_grade: 1, current_grade_date: '2023-05-01'}
|
||
|
||
- name: int_ofsted_fe_report_card_older_than_grade_is_ignored
|
||
description: Bad data guard. A report card dated before the graded inspection cannot be the latest, and a row must never carry both a grade and a report card.
|
||
model: int_ofsted_fe_latest
|
||
given:
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 7, ukprn: null}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 7, graded_inspection_date: '2026-03-01', overall_effectiveness: 2}
|
||
- input: ref('stg_ofsted_fe_inspections')
|
||
rows:
|
||
- {inspection_number: '1', provider_urn: 7, rc_inspection_date: '2026-01-10', rc_safeguarding_met: true}
|
||
expect:
|
||
rows:
|
||
- {urn: 7, latest_visit_kind: graded, current_grade: 2, rc_inspection_date: null, rc_safeguarding_met: null}
|
||
```
|
||
|
||
- [ ] **Step 2: Run them to see them fail**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_fe_latest,test_type:unit"`
|
||
Expected: non-zero exit; parsing fails (model missing).
|
||
|
||
- [ ] **Step 3: Write the model**
|
||
|
||
`pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql`:
|
||
|
||
```sql
|
||
-- Intermediate model: Ofsted further education and skills status per URN
|
||
-- (audit H4), in int_ofsted_latest's columns so it can be unioned there.
|
||
-- A provider matches a school by Provider URN, else by UKPRN (stg_gias_ukprn).
|
||
-- Rule, in the C1 status rule's terms
|
||
-- (docs/superpowers/specs/2026-10-06-inspection-status-beyond-ofsted-schools-design.md):
|
||
-- a report card on or after the previous-framework inspection wins and
|
||
-- leaves no grade in force; otherwise a previous-framework grade 1-4 is the
|
||
-- current grade, dated by that inspection; the latest visit is the later of
|
||
-- it and any short inspection (whose outcome the MI does not give).
|
||
|
||
with gias as (
|
||
select urn, ukprn from {{ ref('stg_gias_ukprn') }}
|
||
),
|
||
|
||
providers as (
|
||
select distinct on (g.urn)
|
||
g.urn,
|
||
p.graded_inspection_date,
|
||
p.overall_effectiveness,
|
||
p.short_inspection_date
|
||
from gias g
|
||
join {{ ref('stg_ofsted_fe_providers') }} p
|
||
on p.provider_urn = g.urn
|
||
or (g.ukprn is not null and p.provider_ukprn = g.ukprn)
|
||
order by g.urn, (p.provider_urn = g.urn) desc, p.graded_inspection_date desc nulls last
|
||
),
|
||
|
||
report_cards as (
|
||
select distinct on (g.urn)
|
||
g.urn,
|
||
c.rc_inspection_date,
|
||
c.rc_safeguarding_met
|
||
from gias g
|
||
join {{ ref('stg_ofsted_fe_inspections') }} c
|
||
on c.provider_urn = g.urn
|
||
or (g.ukprn is not null and c.provider_ukprn = g.ukprn)
|
||
order by g.urn, (c.provider_urn = g.urn) desc, c.rc_inspection_date desc, c.inspection_number desc
|
||
),
|
||
|
||
combined as (
|
||
select
|
||
coalesce(p.urn, c.urn) as urn,
|
||
p.graded_inspection_date,
|
||
p.overall_effectiveness,
|
||
p.short_inspection_date,
|
||
c.rc_inspection_date as card_date,
|
||
c.rc_safeguarding_met as card_safeguarding_met
|
||
from providers p
|
||
full outer join report_cards c on c.urn = p.urn
|
||
),
|
||
|
||
decided as (
|
||
select
|
||
*,
|
||
card_date is not null
|
||
and card_date >= coalesce(greatest(graded_inspection_date, short_inspection_date), card_date)
|
||
as report_card_latest
|
||
from combined
|
||
)
|
||
|
||
select
|
||
urn,
|
||
case when report_card_latest then card_date
|
||
else greatest(graded_inspection_date, short_inspection_date) end as latest_visit_date,
|
||
case when report_card_latest then 'report_card'
|
||
when short_inspection_date > graded_inspection_date then 'short'
|
||
else 'graded' end as latest_visit_kind,
|
||
null::text as latest_visit_outcome,
|
||
case when not report_card_latest then overall_effectiveness end as current_grade,
|
||
case when not report_card_latest then graded_inspection_date end as current_grade_date,
|
||
case when not report_card_latest then 'graded' end as current_grade_basis,
|
||
graded_inspection_date,
|
||
null::date as ungraded_inspection_date,
|
||
coalesce(graded_inspection_date, card_date) as inspection_date,
|
||
null::text as inspection_type,
|
||
'Further education and skills'::text as framework,
|
||
overall_effectiveness,
|
||
null::integer as quality_of_education,
|
||
null::integer as behaviour_attitudes,
|
||
null::integer as personal_development,
|
||
null::integer as leadership_management,
|
||
null::integer as early_years_provision,
|
||
null::integer as sixth_form_provision,
|
||
null::text as ungraded_outcome,
|
||
null::integer as ungraded_grade,
|
||
case when report_card_latest then card_safeguarding_met end as rc_safeguarding_met,
|
||
null::integer as rc_inclusion,
|
||
null::integer as rc_curriculum_teaching,
|
||
null::integer as rc_achievement,
|
||
null::integer as rc_attendance_behaviour,
|
||
null::integer as rc_personal_development,
|
||
null::integer as rc_leadership_governance,
|
||
null::integer as rc_early_years,
|
||
null::integer as rc_sixth_form,
|
||
case when report_card_latest then card_date end as rc_inspection_date,
|
||
null::text as report_url
|
||
from decided
|
||
where report_card_latest or overall_effectiveness between 1 and 4
|
||
```
|
||
|
||
- [ ] **Step 4: Run the unit tests and build**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_ukprn stg_ofsted_fe_providers stg_ofsted_fe_inspections int_ofsted_fe_latest`
|
||
Expected: exit 0; all 8 `int_ofsted_fe_latest` unit tests PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add pipeline/transform/models/intermediate/int_ofsted_fe_latest.sql pipeline/transform/models/intermediate/int_ofsted_fe_latest.yml
|
||
git commit -m "feat(dbt): further education status per URN under the C1 rule (H4)"
|
||
```
|
||
|
||
### Task 6: Fold FE into `int_ofsted_latest` / `fact_ofsted_latest`
|
||
|
||
**Files:**
|
||
- Modify: `pipeline/transform/models/intermediate/int_ofsted_latest.sql`, `int_ofsted_latest.yml`
|
||
- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (`fact_ofsted_latest.latest_visit_kind` accepted values)
|
||
- Create: `pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `int_ofsted_fe_latest` (Task 5; same columns and order as `int_ofsted_latest`).
|
||
- Produces: `int_ofsted_latest` / `fact_ofsted_latest` rows with `framework = 'Further education and skills'` for URNs that have no school-framework record; `latest_visit_kind` may be `short`.
|
||
|
||
- [ ] **Step 1: Write the failing unit tests, and give the existing ones the new input**
|
||
|
||
In `int_ofsted_latest.yml`, under `given:` of **each** of the 10 existing tests, after its `stg_ofsted_inspections` input, add:
|
||
|
||
```yaml
|
||
- input: ref('int_ofsted_fe_latest')
|
||
rows: []
|
||
```
|
||
|
||
Append:
|
||
|
||
```yaml
|
||
- name: fe_status_fills_a_school_with_no_school_record
|
||
description: Queen Mary's College (145175) has no school-framework record; its FE status passes through (audit H4).
|
||
model: int_ofsted_latest
|
||
given:
|
||
- input: ref('stg_ofsted_inspections')
|
||
rows: []
|
||
- input: ref('int_ofsted_fe_latest')
|
||
rows:
|
||
- {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'}
|
||
expect:
|
||
rows:
|
||
- {urn: 145175, latest_visit_date: '2022-12-06', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2022-12-06', current_grade_basis: graded, framework: 'Further education and skills'}
|
||
|
||
- name: a_school_record_wins_over_an_fe_record
|
||
model: int_ofsted_latest
|
||
given:
|
||
- input: ref('stg_ofsted_inspections')
|
||
rows:
|
||
- {urn: 1, graded_inspection_date: '2019-06-01', overall_effectiveness: 2, framework: 'Schools - S5'}
|
||
- input: ref('int_ofsted_fe_latest')
|
||
rows:
|
||
- {urn: 1, latest_visit_date: '2026-05-12', latest_visit_kind: report_card, rc_inspection_date: '2026-05-12', framework: 'Further education and skills'}
|
||
expect:
|
||
rows:
|
||
- {urn: 1, latest_visit_date: '2019-06-01', latest_visit_kind: graded, current_grade: 2, framework: 'Schools - S5'}
|
||
```
|
||
|
||
- [ ] **Step 2: Run them to see them fail**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_latest,test_type:unit"`
|
||
Expected: non-zero exit — `int_ofsted_latest` does not reference `int_ofsted_fe_latest`, so the given input is rejected (or, if dbt accepts unused inputs, `fe_status_fills_a_school_with_no_school_record` FAILS with no rows).
|
||
|
||
- [ ] **Step 3: Union the FE rows**
|
||
|
||
In `int_ofsted_latest.sql`, add to the header comment:
|
||
|
||
```sql
|
||
-- Schools with no school-framework record take their further education and
|
||
-- skills status from int_ofsted_fe_latest (same columns, same order; audit H4).
|
||
```
|
||
|
||
Replace the final `select … from graded` with the same select wrapped in a CTE, followed by the union:
|
||
|
||
```sql
|
||
school_status as (
|
||
select
|
||
urn,
|
||
-- (… the existing select list, unchanged, through report_url …)
|
||
report_url
|
||
from graded
|
||
),
|
||
|
||
fe_status as (
|
||
-- Same columns in the same order as school_status.
|
||
select f.*
|
||
from {{ ref('int_ofsted_fe_latest') }} f
|
||
where not exists (select 1 from school_status s where s.urn = f.urn)
|
||
)
|
||
|
||
select * from school_status
|
||
union all
|
||
select * from fe_status
|
||
```
|
||
|
||
(The `school_status` select list is the existing final select list verbatim; only `select` becomes `school_status as (select` and a `)` closes it. Remember the comma after the `graded` CTE's closing parenthesis.)
|
||
|
||
In `_marts_schema.yml`, `fact_ofsted_latest` → `latest_visit_kind` → `accepted_values` → `values: ['report_card', 'graded', 'ungraded', 'short']`.
|
||
|
||
`pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql`:
|
||
|
||
```sql
|
||
-- A school-framework record always wins over a further education one for the
|
||
-- same URN (audit H4): no FE row may stand where Ofsted's school MI has one.
|
||
|
||
select f.urn
|
||
from {{ ref('fact_ofsted_latest') }} f
|
||
join {{ ref('stg_ofsted_inspections') }} s on s.urn = f.urn
|
||
where f.framework = 'Further education and skills'
|
||
```
|
||
|
||
- [ ] **Step 4: Run the unit tests and build**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select int_ofsted_fe_latest int_ofsted_latest fact_ofsted_latest assert_ofsted_fe_never_overrides_school assert_ofsted_current_grade_consistent`
|
||
Expected: exit 0; 12 `int_ofsted_latest` unit tests PASS; schema and data tests PASS on empty tables.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add pipeline/transform/models/intermediate/int_ofsted_latest.sql pipeline/transform/models/intermediate/int_ofsted_latest.yml pipeline/transform/models/marts/_marts_schema.yml pipeline/transform/tests/assert_ofsted_fe_never_overrides_school.sql
|
||
git commit -m "feat(dbt): further education status where a school has no school record (H4)"
|
||
```
|
||
|
||
### Task 7: `int_ofsted_listed`, `dim_school` columns, DAG selectors
|
||
|
||
**Files:**
|
||
- Create: `pipeline/transform/models/intermediate/int_ofsted_listed.sql`, `int_ofsted_listed.yml`
|
||
- Modify: `pipeline/transform/models/marts/dim_school.sql`, `dim_school.yml`
|
||
- Modify: `pipeline/dags/school_data_pipeline.py` (monthly Ofsted selection), `pipeline/tests/test_dag_selectors.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: raw `ofsted_inspections`, `stg_gias_ukprn`, `stg_ofsted_fe_providers`, `stg_gias_establishments.inspectorate_name`.
|
||
- Produces: `int_ofsted_listed(urn integer)`; `dim_school.inspectorate text`, `dim_school.ofsted_listed boolean` (null when `int_ofsted_listed` does not exist yet).
|
||
|
||
- [ ] **Step 1: Write the unit tests**
|
||
|
||
`pipeline/transform/models/intermediate/int_ofsted_listed.yml`:
|
||
|
||
```yaml
|
||
version: 2
|
||
|
||
unit_tests:
|
||
- name: int_ofsted_listed_includes_uninspected_schools_and_fe_providers
|
||
description: >
|
||
Abbas and Templecombe (150031) is in Ofsted's school MI with no inspection;
|
||
Exeter and North Devon Colleges Group (130645) is in the FE MI with none;
|
||
a UKPRN-only FE match counts too. A school in neither is absent (audit H4).
|
||
model: int_ofsted_listed
|
||
given:
|
||
- input: source('raw', 'ofsted_inspections')
|
||
rows:
|
||
- {urn: '150031', inspection_date: 'NULL'}
|
||
- {urn: '102408', inspection_date: '17/06/2025'}
|
||
- input: ref('stg_gias_ukprn')
|
||
rows:
|
||
- {urn: 130645, ukprn: null}
|
||
- {urn: 5, ukprn: '10099999'}
|
||
- {urn: 133058, ukprn: null}
|
||
- input: ref('stg_ofsted_fe_providers')
|
||
rows:
|
||
- {provider_urn: 130645, provider_ukprn: null}
|
||
- {provider_urn: 9999999, provider_ukprn: '10099999'}
|
||
expect:
|
||
rows:
|
||
- {urn: 150031}
|
||
- {urn: 102408}
|
||
- {urn: 130645}
|
||
- {urn: 5}
|
||
```
|
||
|
||
In `dim_school.yml`, under `given:` of both existing tests add:
|
||
|
||
```yaml
|
||
- input: ref('int_ofsted_listed')
|
||
rows: []
|
||
```
|
||
|
||
and append:
|
||
|
||
```yaml
|
||
- name: dim_school_carries_inspectorate_and_ofsted_listing
|
||
description: >
|
||
Audit H4. GIAS names ISI for Abbey College Cambridge; Ofsted lists Abbas
|
||
and Templecombe without an inspection; a third school is in neither.
|
||
model: dim_school
|
||
given:
|
||
- input: ref('stg_gias_establishments')
|
||
rows:
|
||
- {urn: 136083, school_name: 'Abbey College Cambridge', status_code: 1, school_type_code: 11, local_authority_code: 873, phase_code: 4, inspectorate_name: 'ISI'}
|
||
- {urn: 150031, school_name: 'Abbas and Templecombe Church School', status_code: 1, school_type_code: 28, local_authority_code: 933, phase_code: 2, inspectorate_name: 'Not applicable'}
|
||
- {urn: 1, school_name: 'No data school', status_code: 1, school_type_code: 1, local_authority_code: 312, phase_code: 2, inspectorate_name: null}
|
||
- input: ref('int_ofsted_latest')
|
||
rows: []
|
||
- input: ref('int_ofsted_listed')
|
||
rows:
|
||
- {urn: 150031}
|
||
expect:
|
||
rows:
|
||
- {urn: 136083, inspectorate: 'ISI', ofsted_listed: false}
|
||
- {urn: 150031, inspectorate: 'Not applicable', ofsted_listed: true}
|
||
- {urn: 1, inspectorate: null, ofsted_listed: false}
|
||
```
|
||
|
||
- [ ] **Step 2: Run them to see them fail**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh test --no-partial-parse --select "int_ofsted_listed dim_school,test_type:unit"`
|
||
Expected: non-zero exit (model `int_ofsted_listed` missing; `dim_school` has no such columns).
|
||
|
||
- [ ] **Step 3: Write `int_ofsted_listed` and extend `dim_school`**
|
||
|
||
`pipeline/transform/models/intermediate/int_ofsted_listed.sql`:
|
||
|
||
```sql
|
||
-- Intermediate model: every URN Ofsted's own data lists, inspected or not
|
||
-- (audit H4). Only these may be called "Not yet inspected". Reads the raw
|
||
-- school MI because stg_ofsted_inspections drops rows with no inspection date.
|
||
|
||
with school_mi as (
|
||
select distinct cast(urn as integer) as urn
|
||
from {{ source('raw', 'ofsted_inspections') }}
|
||
where urn is not null
|
||
),
|
||
|
||
fe as (
|
||
select distinct g.urn
|
||
from {{ ref('stg_gias_ukprn') }} g
|
||
join {{ ref('stg_ofsted_fe_providers') }} p
|
||
on p.provider_urn = g.urn
|
||
or (g.ukprn is not null and p.provider_ukprn = g.ukprn)
|
||
)
|
||
|
||
select urn from school_mi
|
||
union
|
||
select urn from fe
|
||
```
|
||
|
||
In `dim_school.sql`:
|
||
|
||
1. Below the existing `-- depends_on: {{ ref('int_ofsted_latest') }}` line add `-- depends_on: {{ ref('int_ofsted_listed') }}` and extend the comment above it: `int_ofsted_latest and int_ofsted_listed are only ref()'d inside conditional blocks below (they may not exist before the Ofsted pipeline first runs), so declare the dependencies explicitly for dbt's DAG.`
|
||
2. After the `{% set ofsted_relation = … %}` block add:
|
||
|
||
```sql
|
||
{% set listed_relation = adapter.get_relation(
|
||
database=target.database,
|
||
schema='intermediate',
|
||
identifier='int_ofsted_listed'
|
||
) %}
|
||
```
|
||
|
||
3. After `s.admissions_policy_code,` add:
|
||
|
||
```sql
|
||
-- The inspectorate GIAS names ("ISI" for most independent schools) and
|
||
-- whether Ofsted's own data lists the school, inspected or not (audit H4).
|
||
s.inspectorate_name as inspectorate,
|
||
{% if listed_relation is not none %}
|
||
l.urn is not null as ofsted_listed,
|
||
{% else %}
|
||
null::boolean as ofsted_listed,
|
||
{% endif %}
|
||
```
|
||
|
||
4. After the `{% if ofsted_relation is not none %} left join … {% endif %}` block add:
|
||
|
||
```sql
|
||
{% if listed_relation is not none %}
|
||
left join {{ ref('int_ofsted_listed') }} l on s.urn = l.urn
|
||
{% endif %}
|
||
```
|
||
|
||
In `pipeline/dags/school_data_pipeline.py`, in the monthly Ofsted build change `--select stg_ofsted_inspections+ int_ofsted_latest+` to `--select stg_ofsted_inspections+ stg_ofsted_fe_providers+ stg_ofsted_fe_inspections+ int_ofsted_latest+`.
|
||
|
||
In `pipeline/tests/test_dag_selectors.py`:
|
||
|
||
```python
|
||
# dim_school reads int_ofsted_latest and int_ofsted_listed only when the
|
||
# relations exist (adapter.get_relation), so a missing table is not a failure.
|
||
OPTIONAL_PARENTS = {'int_ofsted_latest', 'int_ofsted_listed'}
|
||
```
|
||
|
||
- [ ] **Step 4: Run the tests and build**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select int_ofsted_listed dim_school`
|
||
Expected: exit 0; the `int_ofsted_listed` unit test and 3 `dim_school` unit tests PASS.
|
||
Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest pipeline/tests -q` → all pass.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add pipeline/transform/models/intermediate/int_ofsted_listed.sql pipeline/transform/models/intermediate/int_ofsted_listed.yml pipeline/transform/models/marts/dim_school.sql pipeline/transform/models/marts/dim_school.yml pipeline/dags/school_data_pipeline.py pipeline/tests/test_dag_selectors.py
|
||
git commit -m "feat(dbt): dim_school.inspectorate and ofsted_listed (H4)"
|
||
```
|
||
|
||
### Task 8: Load the real data end to end and verify PR 1
|
||
|
||
**Files:** none in the repo.
|
||
|
||
- [ ] **Step 1: Load GIAS and Ofsted through the changed taps**
|
||
|
||
```bash
|
||
mkdir -p "$SCRATCH/e2e_h4" && cd "$SCRATCH/e2e_h4"
|
||
echo "drop table if exists raw.gias_establishments, raw.gias_links, raw.ofsted_inspections, raw.ofsted_fe_providers, raw.ofsted_fe_inspections cascade;" | "$SCRATCH/dbt_local/sql.sh"
|
||
SOCK=$(grep -o 'PG_HOST=[^ ]*' "$SCRATCH/dbt_local/dbt.sh" | cut -d= -f2)
|
||
echo "{\"sqlalchemy_url\": \"postgresql+psycopg://postgres@/school_compare?host=$SOCK\", \"default_target_schema\": \"raw\"}" > target.json
|
||
echo '{}' > tap.json
|
||
cat > run_load.sh <<'EOF'
|
||
#!/bin/bash
|
||
R=/Users/tudor/projects/school_compare/pipeline/plugins/extractors
|
||
for tap in gias ofsted; do
|
||
uv run --no-project --with-editable "$R/tap-uk-$tap" tap-uk-$tap --config tap.json 2> tap-$tap.log \
|
||
| uv run --no-project --with meltanolabs-target-postgres target-postgres --config target.json > target-$tap.log 2>&1
|
||
echo "$tap exit ${PIPESTATUS[*]}"
|
||
done
|
||
EOF
|
||
bash run_load.sh
|
||
```
|
||
|
||
Expected: `gias exit 0 0` and `ofsted exit 0 0`.
|
||
|
||
- [ ] **Step 2: Build and test everything this PR touches**
|
||
|
||
Run: `$SCRATCH/dbt_local/dbt.sh build --no-partial-parse --select stg_gias_establishments stg_gias_ukprn stg_ofsted_inspections stg_ofsted_fe_providers stg_ofsted_fe_inspections int_ofsted_fe_latest int_ofsted_latest int_ofsted_listed fact_ofsted_latest dim_school assert_ofsted_fe_never_overrides_school assert_ofsted_current_grade_consistent`
|
||
Expected: exit 0; all unit, schema and data tests PASS.
|
||
|
||
- [ ] **Step 3: Check the values**
|
||
|
||
Through `"$SCRATCH/dbt_local/sql.sh" <<'SQL' … SQL`:
|
||
|
||
```sql
|
||
select urn, latest_visit_date, latest_visit_kind, current_grade, current_grade_date, rc_inspection_date, rc_safeguarding_met, framework
|
||
from marts.fact_ofsted_latest where urn in (145175, 143526, 130457) order by urn;
|
||
select count(*) from marts.fact_ofsted_latest where framework = 'Further education and skills';
|
||
select count(*) filter (where inspectorate = 'ISI') as isi, count(*) filter (where ofsted_listed) as listed from marts.dim_school;
|
||
select d.urn, d.ofsted_listed, f.urn is not null as has_record from marts.dim_school d left join marts.fact_ofsted_latest f using (urn) where d.urn in (150031, 136083);
|
||
select count(*) as fe_rows_for_listed_schools from marts.fact_ofsted_latest f join marts.dim_school d using (urn) where f.framework = 'Further education and skills';
|
||
```
|
||
|
||
Expected: 145175 graded 2 on 2022-12-06; 143526 report card 2026-05-12 with safeguarding true; 130457 graded 2 on 2024-01-30; about 490 FE rows for listed schools (443 + 45 + 2 as at 31 August 2026); about 1,341 ISI schools; 150031 listed with no record (unless Ofsted has inspected it since); 136083 not listed, no record. Record any drift from these numbers and its cause (newer MI files) in the PR description.
|
||
|
||
- [ ] **Step 4: Run CI's Python suite**
|
||
|
||
Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests pipeline/tests scripts/ci/tests -q` → all pass.
|
||
|
||
### Task 9: Open PR 1
|
||
|
||
- [ ] **Step 1:** `git status --short` → only `.DS_Store`.
|
||
- [ ] **Step 2:** `git push -u origin fix/ofsted-inspectorates-pipeline`.
|
||
- [ ] **Step 3:** Open the PR through the Gitea API with basic auth from `git credential fill` (host `privaterepo.sitaru.org`; never print the secret): `POST /api/v1/repos/tudor/school_compare/pulls`, head `fix/ofsted-inspectorates-pipeline`, base `main`, title `fix(pipeline): inspectorate, Ofsted listing and further-education status (H4, part 1 of 2)`. Body: what changed; Task 8's numbers; **After merge:** run the daily DAG (GIAS fields) and then `school_data_monthly_ofsted` on staging — the monthly build reads `raw.gias_establishments."UKPRN"`, which exists only after a daily run with this PR; then merge PR 2; production the same order before PR 2 is promoted; ends with the Claude Code line.
|
||
- [ ] **Step 4:** Report the PR URL. Do not merge.
|
||
|
||
---
|
||
|
||
## PR 2 — site
|
||
|
||
After PR 1 merges: `git fetch origin && git switch -c fix/ofsted-inspectorates-site origin/main`.
|
||
|
||
### Task 10: Backend payloads, optional columns, Ofsted page link
|
||
|
||
**Files:**
|
||
- Modify: `backend/schemas.py` (`SCHOOL_COLUMNS`), `backend/data_loader.py` (`_MAIN_QUERY`, `_EXTRA_OPTIONAL_COLUMNS`, the fallback loop), `backend/app.py` (detail and compare `school_info`), `backend/ofsted_codes.py` (`ofsted_page_url`)
|
||
- Test: create `backend/tests/test_inspection_status_payload.py`; modify `backend/tests/test_ofsted_codes.py:45`, `backend/tests/test_supplementary_enrichment.py:36`
|
||
|
||
**Interfaces:**
|
||
- Produces: list rows, detail `school_info` and compare `school_info` carry `inspectorate: str | null` and `ofsted_listed: bool | null`; `ofsted_page_url(urn)` returns `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}`.
|
||
|
||
- [ ] **Step 1: Write the failing tests**
|
||
|
||
`backend/tests/test_inspection_status_payload.py`:
|
||
|
||
```python
|
||
"""Audit H4: every payload that feeds an Ofsted badge carries the inspectorate
|
||
GIAS names and whether Ofsted's own data lists the school, so the site never
|
||
says "Not yet inspected" on the strength of missing data."""
|
||
|
||
import pandas as pd
|
||
import pytest
|
||
from fastapi.testclient import TestClient
|
||
|
||
|
||
def _df():
|
||
base = dict(year=202425, local_authority='Cambridgeshire', local_authority_code=873,
|
||
postcode='CB1 1AA', latitude=52.2, longitude=0.12, address='1 High St',
|
||
status='Open', town='Cambridge')
|
||
return pd.DataFrame([
|
||
dict(base, urn=136083, school_name='Abbey College Cambridge', phase='Secondary',
|
||
school_type='Other independent school', inspectorate='ISI', ofsted_listed=False),
|
||
dict(base, urn=150031, school_name='Abbas and Templecombe Church School', phase='Primary',
|
||
school_type='Academy converter', inspectorate='Not applicable', ofsted_listed=True),
|
||
])
|
||
|
||
|
||
@pytest.fixture()
|
||
def client(monkeypatch):
|
||
from backend import app as app_module
|
||
from backend import database as database_module
|
||
|
||
def _no_db():
|
||
raise RuntimeError('no database in this test')
|
||
|
||
monkeypatch.setattr(app_module, 'load_latest_school_data', _df)
|
||
monkeypatch.setattr(app_module, 'load_school_data', _df)
|
||
monkeypatch.setattr(app_module, 'get_supplementary_data', lambda db, urn: {})
|
||
monkeypatch.setattr(app_module, '_place_registry', None)
|
||
monkeypatch.setattr(database_module, 'SessionLocal', _no_db)
|
||
return TestClient(app_module.app, raise_server_exceptions=False)
|
||
|
||
|
||
def test_the_list_carries_inspectorate_and_listing(client):
|
||
resp = client.get('/api/schools?page_size=50')
|
||
assert resp.status_code == 200, resp.text
|
||
by_urn = {s['urn']: s for s in resp.json()['schools']}
|
||
assert by_urn[136083]['inspectorate'] == 'ISI'
|
||
assert by_urn[150031]['ofsted_listed'] is True
|
||
|
||
|
||
def test_the_school_page_carries_them(client):
|
||
info = client.get('/api/schools/136083').json()['school_info']
|
||
assert info['inspectorate'] == 'ISI'
|
||
assert info['ofsted_listed'] is False
|
||
|
||
|
||
def test_compare_carries_them(client):
|
||
body = client.get('/api/compare?urns=136083,150031').json()
|
||
assert body['comparison']['150031']['school_info']['ofsted_listed'] is True
|
||
assert body['comparison']['136083']['school_info']['inspectorate'] == 'ISI'
|
||
|
||
|
||
def test_a_mart_built_before_the_columns_still_loads():
|
||
# Missing columns make the main query retry with NULLs (data_loader).
|
||
from backend import data_loader
|
||
assert {'inspectorate', 'ofsted_listed'} <= set(data_loader._EXTRA_OPTIONAL_COLUMNS)
|
||
fallback = str(data_loader._MAIN_QUERY_NO_EXTRA_COLS)
|
||
assert 'NULL AS inspectorate,' in fallback and 'NULL AS ofsted_listed,' in fallback
|
||
```
|
||
|
||
(`/api/compare` returns `{"comparison": {<urn>: {"school_info": …}}}`.) Change `backend/tests/test_ofsted_codes.py:45` and `backend/tests/test_supplementary_enrichment.py:36` to expect `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/<urn>` (keep each test's URN).
|
||
|
||
- [ ] **Step 2: Run them to see them fail**
|
||
|
||
Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests/test_inspection_status_payload.py backend/tests/test_ofsted_codes.py backend/tests/test_supplementary_enrichment.py -q`
|
||
Expected: the four new tests fail (`KeyError: 'inspectorate'`; the optional-columns assertion); the two link tests fail on the old `provider/21` URL.
|
||
|
||
- [ ] **Step 3: Implement**
|
||
|
||
- `backend/schemas.py`, `SCHOOL_COLUMNS`: after `"ofsted_rc_date",` add `"inspectorate",` and `"ofsted_listed",`.
|
||
- `backend/data_loader.py`, `_MAIN_QUERY`: after ` s.nursery_provision,` add ` s.inspectorate,` and ` s.ofsted_listed,`. Add `"inspectorate",` and `"ofsted_listed",` to `_EXTRA_OPTIONAL_COLUMNS`, and `("inspectorate", "s"),` and `("ofsted_listed", "s"),` to the tuple of `(column, alias)` pairs the fallback loop replaces. Extend the comment above `_EXTRA_OPTIONAL_COLUMNS`: `… parliamentary constituency, and the inspectorate / Ofsted-listing columns added for audit H4) …`.
|
||
- `backend/app.py`, detail `school_info` (after `"nursery_provision": latest.get("nursery_provision"),`):
|
||
|
||
```python
|
||
# What the Ofsted badge may say when there is no inspection record
|
||
# (audit H4): the inspectorate GIAS names, and whether Ofsted's own
|
||
# data lists the school at all.
|
||
"inspectorate": latest.get("inspectorate"),
|
||
"ofsted_listed": latest.get("ofsted_listed"),
|
||
```
|
||
|
||
- `backend/app.py`, compare `school_info` (after `"has_sixth_form": convert_to_native(latest.get("has_sixth_form")),`):
|
||
|
||
```python
|
||
# Audit H4: what compare may say when there is no inspection record.
|
||
"inspectorate": convert_to_native(latest.get("inspectorate")),
|
||
"ofsted_listed": convert_to_native(latest.get("ofsted_listed")),
|
||
```
|
||
|
||
- `backend/ofsted_codes.py`:
|
||
|
||
```python
|
||
def ofsted_page_url(urn: int) -> str:
|
||
"""The school's page on ofsted.gov.uk (all its reports live there — we
|
||
never deep-link an individual report). Ofsted's provider pages sit under a
|
||
category (21 primary, 23 secondary, 25 special, 40 FE colleges, …); this
|
||
address redirects to the right one, where provider/21 404s for every
|
||
non-primary school."""
|
||
return f"https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/{urn}"
|
||
```
|
||
|
||
- [ ] **Step 4: Run the tests and the backend suite**
|
||
|
||
Run: the Step 2 command → all pass. Run: `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests -q` → all pass.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add backend/schemas.py backend/data_loader.py backend/app.py backend/ofsted_codes.py backend/tests/test_inspection_status_payload.py backend/tests/test_ofsted_codes.py backend/tests/test_supplementary_enrichment.py
|
||
git commit -m "fix(api): inspectorate and Ofsted listing in payloads; Ofsted links that resolve (H4)"
|
||
```
|
||
|
||
### Task 11: The badge rule and its callers
|
||
|
||
**Files:**
|
||
- Modify: `nextjs-app/lib/types.ts` (`School`, `OfstedLatestVisit`), `nextjs-app/lib/ofstedStatus.ts`, `nextjs-app/lib/utils.ts` (`buildOfstedListBadge`)
|
||
- Modify: `nextjs-app/components/SchoolRow.tsx`, `SecondarySchoolRow.tsx`, `LeafletMapInner.tsx`
|
||
- Test: `nextjs-app/__tests__/lib/utils.test.ts`, `__tests__/lib/ofstedStatus.test.ts`, `__tests__/components/SchoolRow.test.tsx`
|
||
|
||
**Interfaces:**
|
||
- Produces: `School.inspectorate?: string | null`, `School.ofsted_listed?: boolean | null`; `OfstedLatestVisit.kind` adds `'short'`; `FE_FRAMEWORK = 'Further education and skills'`; `hasReportCard(ofsted: Pick<OfstedInspection, 'rc_inspection_date' | 'report_card'>): boolean`; `buildOfstedListBadge(school): OfstedListBadge | null`.
|
||
|
||
- [ ] **Step 1: Write the failing tests**
|
||
|
||
In `__tests__/lib/utils.test.ts`, inside `describe('buildOfstedListBadge', …)`: replace the test `returns pending badge when no grade and no inspection on record` with the block below, and in every other test in that describe use `buildOfstedListBadge(…)!` (the function may now return null):
|
||
|
||
```ts
|
||
it('says "Not yet inspected" only when Ofsted lists the school uninspected (audit H4)', () => {
|
||
const badge = buildOfstedListBadge({ ofsted_grade: null, ofsted_date: null, ofsted_listed: true });
|
||
expect(badge).toEqual({ label: 'Not yet inspected', cssClass: 'ofstedPending' });
|
||
});
|
||
|
||
it('names ISI for a school GIAS says ISI inspects', () => {
|
||
expect(buildOfstedListBadge({ ofsted_date: null, inspectorate: 'ISI', ofsted_listed: false }))
|
||
.toEqual({ label: 'Inspected by ISI', cssClass: 'ofstedPending' });
|
||
});
|
||
|
||
it('an Ofsted record wins over ISI', () => {
|
||
expect(buildOfstedListBadge({ ofsted_grade: 2, ofsted_grade_date: '2022-03-01', inspectorate: 'ISI' })!.label)
|
||
.toBe('Good · 2022');
|
||
});
|
||
|
||
it('gives no badge without any inspection data', () => {
|
||
expect(buildOfstedListBadge({ ofsted_grade: null, ofsted_date: null, ofsted_framework: null })).toBeNull();
|
||
expect(buildOfstedListBadge({ ofsted_listed: null, inspectorate: 'Not applicable' })).toBeNull();
|
||
});
|
||
```
|
||
|
||
In `__tests__/lib/ofstedStatus.test.ts` add:
|
||
|
||
```ts
|
||
it('names a short inspection as Ofsted does', () => {
|
||
expect(latestVisitLine({ date: '2023-02-01', kind: 'short', outcome: null })).toBe('Short inspection, 1 February 2023');
|
||
});
|
||
|
||
it('treats a report card with only a safeguarding result as a report card (FE)', () => {
|
||
expect(hasReportCard({ rc_inspection_date: '2026-05-12', report_card: {} })).toBe(true);
|
||
expect(hasReportCard({ rc_inspection_date: null, report_card: {} })).toBe(false);
|
||
});
|
||
```
|
||
|
||
(change the file's first line to `import { gradeSourceLine, hasReportCard, latestVisitLine, showLatestVisitLine } from '@/lib/ofstedStatus';`).
|
||
|
||
In `__tests__/components/SchoolRow.test.tsx` add (the file's school fixture is `base`):
|
||
|
||
```tsx
|
||
it('renders no Ofsted badge for a school with no inspection data (audit H4)', () => {
|
||
const { container } = render(<SchoolRow school={{ ...base, ofsted_grade: null, ofsted_date: null, ofsted_listed: false, inspectorate: null }} />);
|
||
expect(container.textContent).not.toMatch(/Not yet inspected|Inspected/);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run them to see them fail**
|
||
|
||
Run: `cd nextjs-app && npx jest __tests__/lib/utils.test.ts __tests__/lib/ofstedStatus.test.ts __tests__/components/SchoolRow.test.tsx`
|
||
Expected: the new tests fail (no ISI rule; `hasReportCard` missing; "Not yet inspected" for no data).
|
||
|
||
- [ ] **Step 3: Implement**
|
||
|
||
`lib/types.ts`: in `School`, after `ofsted_framework?: string | null;` add
|
||
|
||
```ts
|
||
/** The inspectorate GIAS names ("ISI", "Ofsted", "Not applicable"). Audit H4. */
|
||
inspectorate?: string | null;
|
||
/** Whether Ofsted's own data lists the school, inspected or not. Audit H4. */
|
||
ofsted_listed?: boolean | null;
|
||
```
|
||
|
||
and in `OfstedLatestVisit` change `kind` to `'report_card' | 'graded' | 'ungraded' | 'short';`.
|
||
|
||
`lib/ofstedStatus.ts`: add `short: 'Short inspection',` to `VISIT_KIND`, and
|
||
|
||
```ts
|
||
/** fact_ofsted_latest.framework for Ofsted's further education and skills records (audit H4). */
|
||
export const FE_FRAMEWORK = 'Further education and skills';
|
||
|
||
/**
|
||
* A report card is in force. Checked by its date as well as its area labels:
|
||
* an FE report card carries a safeguarding result and no school areas.
|
||
*/
|
||
export function hasReportCard(
|
||
ofsted: Pick<OfstedInspection, 'rc_inspection_date' | 'report_card'>,
|
||
): boolean {
|
||
return !!ofsted.rc_inspection_date || Object.keys(ofsted.report_card ?? {}).length > 0;
|
||
}
|
||
```
|
||
|
||
`lib/utils.ts`, `buildOfstedListBadge`: add `inspectorate?: string | null; ofsted_listed?: boolean | null;` to the parameter type, change the return type to `OfstedListBadge | null`, replace the last bullet of the doc comment with
|
||
|
||
```ts
|
||
* - No inspection on record (audit H4): "Inspected by ISI" when GIAS names
|
||
* ISI; "Not yet inspected" only when Ofsted's own data lists the school;
|
||
* otherwise no badge, never a claim the data cannot support.
|
||
```
|
||
|
||
and replace `return { label: 'Not yet inspected', cssClass: 'ofstedPending' };` with
|
||
|
||
```ts
|
||
if (school.inspectorate === 'ISI') return { label: 'Inspected by ISI', cssClass: 'ofstedPending' };
|
||
if (school.ofsted_listed) return { label: 'Not yet inspected', cssClass: 'ofstedPending' };
|
||
return null;
|
||
```
|
||
|
||
`components/SchoolRow.tsx` and `components/SecondarySchoolRow.tsx`: wrap the badge span in `{ofstedBadge && ( … )}`.
|
||
|
||
`components/LeafletMapInner.tsx`, `cardHtml`: replace the tags line with
|
||
|
||
```ts
|
||
<div class="sc-card-tags">${badge ? `<span class="sc-ofsted sc-${badge.cssClass}">${escapeHtml(badge.label)}</span>` : ''}${type}</div>
|
||
```
|
||
|
||
- [ ] **Step 4: Run the tests, the suite and the type check**
|
||
|
||
Run: `npx jest __tests__/lib/utils.test.ts __tests__/lib/ofstedStatus.test.ts __tests__/components/SchoolRow.test.tsx` → pass. Run: `npm run typecheck && npm test` → all pass (fix any other caller the type check names by handling `null`, never by asserting it away).
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add nextjs-app/lib/types.ts nextjs-app/lib/ofstedStatus.ts nextjs-app/lib/utils.ts nextjs-app/components/SchoolRow.tsx nextjs-app/components/SecondarySchoolRow.tsx nextjs-app/components/LeafletMapInner.tsx nextjs-app/__tests__/lib/utils.test.ts nextjs-app/__tests__/lib/ofstedStatus.test.ts nextjs-app/__tests__/components/SchoolRow.test.tsx
|
||
git commit -m "fix(search): no \"Not yet inspected\" without Ofsted's word; ISI badge (H4)"
|
||
```
|
||
|
||
### Task 12: School page and compare
|
||
|
||
**Files:**
|
||
- Create: `nextjs-app/components/school/InspectionNotice.tsx`; test `nextjs-app/__tests__/components/InspectionNotice.test.tsx`
|
||
- Modify: `nextjs-app/components/school/OfstedSection.tsx`, `PrimarySchoolSections.tsx`, `SecondarySchoolSections.tsx`, `nextjs-app/lib/compareLogic.ts`, `nextjs-app/components/compare/CompareOfsted.tsx`, `CompareAtAGlance.tsx`
|
||
- Test: `nextjs-app/__tests__/components/OfstedSection.test.tsx`, `__tests__/components/CompareOfsted.test.tsx`, `__tests__/lib/compareLogic.test.ts`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `FE_FRAMEWORK`, `hasReportCard`, `School.inspectorate`, `School.ofsted_listed` (Task 11).
|
||
- Produces: `InspectionNotice({ school }: { school: Pick<School, 'urn' | 'inspectorate' | 'ofsted_listed'> })` — renders the ISI or "Not yet inspected" section, or nothing; `noInspectionLabel(school): string` in `lib/compareLogic.ts`.
|
||
|
||
- [ ] **Step 1: Write the failing tests**
|
||
|
||
`__tests__/components/InspectionNotice.test.tsx`:
|
||
|
||
```tsx
|
||
import { render, screen } from '@testing-library/react';
|
||
import { InspectionNotice } from '@/components/school/InspectionNotice';
|
||
|
||
it('names ISI and links its reports for an ISI school (audit H4)', () => {
|
||
render(<InspectionNotice school={{ urn: 136083, inspectorate: 'ISI', ofsted_listed: false }} />);
|
||
expect(screen.getByText('Inspected by the Independent Schools Inspectorate (ISI)')).toBeInTheDocument();
|
||
expect(screen.getByRole('link', { name: /ISI reports/ })).toHaveAttribute('href', 'https://www.isi.net/reports/');
|
||
});
|
||
|
||
it('says "Not yet inspected" for a school Ofsted lists without an inspection', () => {
|
||
render(<InspectionNotice school={{ urn: 150031, inspectorate: 'Not applicable', ofsted_listed: true }} />);
|
||
expect(screen.getByText('Not yet inspected')).toBeInTheDocument();
|
||
expect(screen.getByRole('link', { name: /Ofsted reports/ })).toHaveAttribute(
|
||
'href', 'https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/150031');
|
||
});
|
||
|
||
it('renders nothing without inspection data, including an unknown listing', () => {
|
||
const { container } = render(<InspectionNotice school={{ urn: 1, inspectorate: null, ofsted_listed: null }} />);
|
||
expect(container).toBeEmptyDOMElement();
|
||
});
|
||
```
|
||
|
||
In `__tests__/components/OfstedSection.test.tsx` add (uses the file's `renderSection`):
|
||
|
||
```tsx
|
||
it('labels a further education record and prints its short inspection (audit H4)', () => {
|
||
renderSection({
|
||
framework: 'Further education and skills',
|
||
current_grade: { grade: 2, date: '2019-03-05', basis: 'graded' },
|
||
latest_visit: { date: '2023-02-01', kind: 'short', outcome: null },
|
||
});
|
||
expect(screen.getByText('Further education and skills')).toBeInTheDocument();
|
||
expect(screen.getByText('Good')).toBeInTheDocument();
|
||
expect(screen.getByText('Latest visit: Short inspection, 1 February 2023')).toBeInTheDocument();
|
||
});
|
||
```
|
||
|
||
In `__tests__/lib/compareLogic.test.ts` add:
|
||
|
||
```ts
|
||
it('shows an FE report card with only a safeguarding result as a report card (audit H4)', () => {
|
||
const display = ofstedDisplay(ofsted({ rc_inspection_date: '2026-05-12', rc_safeguarding_met: true, report_card: {} }));
|
||
expect(display.kind).toBe('report_card');
|
||
});
|
||
|
||
it('words the no-record compare cell from the evidence', () => {
|
||
expect(noInspectionLabel({ inspectorate: 'ISI' })).toBe('Inspected by ISI');
|
||
expect(noInspectionLabel({ ofsted_listed: true })).toBe('Not yet inspected');
|
||
expect(noInspectionLabel({ ofsted_listed: null })).toBe('No inspection in our dataset');
|
||
});
|
||
```
|
||
|
||
(`ofsted(partial)` is the file's existing fixture helper; add `noInspectionLabel` to its import from `@/lib/compareLogic`.)
|
||
|
||
In `__tests__/components/CompareOfsted.test.tsx`, inside `describe('CompareOfsted', …)`:
|
||
|
||
```tsx
|
||
it('words a school without an Ofsted record from the evidence (audit H4)', () => {
|
||
const isi = { ...school(4, 'ISI School'), inspectorate: 'ISI' } as School;
|
||
const listed = { ...school(5, 'New Academy'), ofsted_listed: true } as School;
|
||
render(<CompareOfsted schools={[isi, listed]} data={{
|
||
'4': { school_info: isi, yearly_data: [] } as unknown as ComparisonData,
|
||
'5': { school_info: listed, yearly_data: [] } as unknown as ComparisonData,
|
||
}} />);
|
||
expect(screen.getAllByText('Inspected by ISI').length).toBeGreaterThan(0);
|
||
expect(screen.getAllByText('Not yet inspected').length).toBeGreaterThan(0);
|
||
expect(screen.queryByText(/No inspection outcome in our dataset/)).toBeNull();
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run them to see them fail**
|
||
|
||
Run: `npx jest __tests__/components/InspectionNotice.test.tsx __tests__/components/OfstedSection.test.tsx __tests__/lib/compareLogic.test.ts __tests__/components/CompareOfsted.test.tsx`
|
||
Expected: the new tests fail (component and helper missing; no FE label; FE report card read as `no_overall_grade`).
|
||
|
||
- [ ] **Step 3: Implement**
|
||
|
||
`components/school/InspectionNotice.tsx`:
|
||
|
||
```tsx
|
||
/**
|
||
* InspectionNotice — the school page's inspection section when there is no
|
||
* Ofsted inspection record (audit H4): the inspectorate GIAS names (ISI), or
|
||
* Ofsted's own "not yet inspected" when its data lists the school. Nothing
|
||
* otherwise: no claim the data cannot support. Server component.
|
||
*/
|
||
|
||
import type { School } from '@/lib/types';
|
||
import { Section, sectionStyles as styles } from './sectionShared';
|
||
|
||
const OFSTED_REPORTS = 'https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/';
|
||
const ISI_REPORTS = 'https://www.isi.net/reports/';
|
||
|
||
export function InspectionNotice({ school }: { school: Pick<School, 'urn' | 'inspectorate' | 'ofsted_listed'> }) {
|
||
if (school.inspectorate === 'ISI') {
|
||
return (
|
||
<Section id="ofsted">
|
||
<h2 className={styles.sectionTitle}>
|
||
Inspection
|
||
<a href={ISI_REPORTS} target="_blank" rel="noopener noreferrer" className={styles.ofstedReportLink}
|
||
data-umami-event="external_link_clicked" data-umami-event-target="isi">
|
||
ISI reports ↗
|
||
</a>
|
||
</h2>
|
||
<p className={styles.ofstedDisclaimer}>Inspected by the Independent Schools Inspectorate (ISI)</p>
|
||
</Section>
|
||
);
|
||
}
|
||
if (school.ofsted_listed) {
|
||
return (
|
||
<Section id="ofsted">
|
||
<h2 className={styles.sectionTitle}>
|
||
Ofsted Rating
|
||
<a href={`${OFSTED_REPORTS}${school.urn}`} target="_blank" rel="noopener noreferrer"
|
||
className={styles.ofstedReportLink} data-umami-event="external_link_clicked" data-umami-event-target="ofsted">
|
||
Ofsted reports ↗
|
||
</a>
|
||
</h2>
|
||
<div className={styles.ofstedHeader}>
|
||
<span className={styles.ofstedGrade}>Not yet inspected</span>
|
||
</div>
|
||
</Section>
|
||
);
|
||
}
|
||
return null;
|
||
}
|
||
```
|
||
|
||
`components/school/OfstedSection.tsx`: import `FE_FRAMEWORK` from `@/lib/ofstedStatus`; directly after the closing `</h2>` add
|
||
|
||
```tsx
|
||
{ofsted.framework === FE_FRAMEWORK && (
|
||
<p className={styles.ofstedDisclaimer}>{FE_FRAMEWORK}</p>
|
||
)}
|
||
```
|
||
|
||
`PrimarySchoolSections.tsx` and `SecondarySchoolSections.tsx`: import `hasReportCard` from `@/lib/ofstedStatus` and `InspectionNotice` from `./InspectionNotice`; set `const isReportCard = !!ofsted && hasReportCard(ofsted);`; change `{ofsted && ( <OfstedSection … /> )}` to `{ofsted ? ( <OfstedSection … /> ) : <InspectionNotice school={schoolInfo} />}`.
|
||
|
||
`lib/compareLogic.ts`: import `hasReportCard` from `./ofstedStatus`; in `ofstedDisplay` replace the report-card condition with `if (hasReportCard(ofsted)) {`; add
|
||
|
||
```ts
|
||
/**
|
||
* The compare cell when a school has no Ofsted record (audit H4): ISI per GIAS,
|
||
* Ofsted's own "not yet inspected", or plainly that we hold nothing.
|
||
*/
|
||
export function noInspectionLabel(school: Pick<School, 'inspectorate' | 'ofsted_listed'>): string {
|
||
if (school.inspectorate === 'ISI') return 'Inspected by ISI';
|
||
if (school.ofsted_listed) return 'Not yet inspected';
|
||
return 'No inspection in our dataset';
|
||
}
|
||
```
|
||
|
||
(import `School` from `./types` if not already imported).
|
||
|
||
`components/compare/CompareOfsted.tsx`: give `ResultCell` a `school` prop (`school: School`) and return `<span className={s.small}>{noInspectionLabel(school)}</span>` for `display.kind === 'none'`; pass `school={data[String(school.urn)]?.school_info ?? school}` where it is rendered; in the judgement-detail cell replace `No inspection in our dataset` with `{noInspectionLabel(data[String(school.urn)]?.school_info ?? school)}`; change the `Ofsted page` fallback URL to `https://reports.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/${school.urn}`.
|
||
|
||
`components/compare/CompareAtAGlance.tsx`: replace `No inspection in our dataset` with `{noInspectionLabel(data[String(school.urn)]?.school_info ?? school)}`.
|
||
|
||
- [ ] **Step 4: Run the tests, the suite and the type check**
|
||
|
||
Run: the Step 2 command → pass. Run: `npm run typecheck && npm test` → all pass.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add nextjs-app/components/school/InspectionNotice.tsx nextjs-app/components/school/OfstedSection.tsx nextjs-app/components/school/PrimarySchoolSections.tsx nextjs-app/components/school/SecondarySchoolSections.tsx nextjs-app/lib/compareLogic.ts nextjs-app/components/compare/CompareOfsted.tsx nextjs-app/components/compare/CompareAtAGlance.tsx nextjs-app/__tests__/components/InspectionNotice.test.tsx nextjs-app/__tests__/components/OfstedSection.test.tsx nextjs-app/__tests__/lib/compareLogic.test.ts nextjs-app/__tests__/components/CompareOfsted.test.tsx
|
||
git commit -m "fix(school, compare): ISI, not-yet-inspected and further-education states (H4)"
|
||
```
|
||
|
||
### Task 13: E2E journeys
|
||
|
||
**Files:**
|
||
- Modify: `e2e/tests/journeys.spec.ts` (after the Ofsted journeys, `test('a grade confirmed at an ungraded visit says so and is dated by it', …)`)
|
||
|
||
- [ ] **Step 1: Add the journeys**
|
||
|
||
```ts
|
||
test('a further education college shows its Ofsted result, not "Not yet inspected"', async ({ page }) => {
|
||
// Queen Mary's College (145175) is inspected under Ofsted's further education
|
||
// framework, which the site did not load: its badge said "Not yet inspected"
|
||
// (audit H4). Its result may change; the label must never be absent or wrong.
|
||
const res = await page.request.get('/api/schools/145175');
|
||
expect(res.ok()).toBeTruthy();
|
||
await page.goto('/school/145175');
|
||
const ofsted = page.locator('#ofsted');
|
||
await expect(ofsted).toBeVisible();
|
||
await expect(ofsted.getByText('Further education and skills')).toBeVisible();
|
||
await expect(ofsted).not.toContainText('Not yet inspected');
|
||
});
|
||
|
||
test('an ISI-inspected school says so instead of "Not yet inspected"', async ({ page }) => {
|
||
const res = await page.request.get('/api/schools?school_type=independent&phase=secondary&page_size=100');
|
||
expect(res.ok()).toBeTruthy();
|
||
const school = ((await res.json()).schools ?? []).find(
|
||
(s: { inspectorate?: string | null; ofsted_date?: string | null; ofsted_rc_date?: string | null }) =>
|
||
s.inspectorate === 'ISI' && !s.ofsted_date && !s.ofsted_rc_date);
|
||
test.skip(!school, 'no ISI school without an Ofsted record here');
|
||
await page.goto(`/school/${school.urn}`);
|
||
await expect(page.getByText('Inspected by the Independent Schools Inspectorate (ISI)')).toBeVisible();
|
||
await expect(page.locator('#ofsted')).not.toContainText('Not yet inspected');
|
||
});
|
||
|
||
test('"Not yet inspected" appears only for a school Ofsted lists without an inspection', async ({ page }) => {
|
||
const res = await page.request.get('/api/schools?phase=primary&page_size=500');
|
||
expect(res.ok()).toBeTruthy();
|
||
type Row = { urn: number; ofsted_listed?: boolean | null; ofsted_date?: string | null; ofsted_rc_date?: string | null; ofsted_grade?: number | null };
|
||
const schools: Row[] = (await res.json()).schools ?? [];
|
||
const unlisted = schools.find(s => !s.ofsted_listed && !s.ofsted_date && !s.ofsted_rc_date && !s.ofsted_grade);
|
||
const listed = schools.find(s => s.ofsted_listed && !s.ofsted_date && !s.ofsted_rc_date && !s.ofsted_grade);
|
||
test.skip(!listed, 'no listed, uninspected primary on this page');
|
||
await page.goto(`/school/${listed!.urn}`);
|
||
await expect(page.locator('#ofsted').getByText('Not yet inspected')).toBeVisible();
|
||
if (unlisted) {
|
||
await page.goto(`/school/${unlisted.urn}`);
|
||
await expect(page.getByText('Not yet inspected')).toHaveCount(0);
|
||
}
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Check the file compiles and lists the tests**
|
||
|
||
Run: `cd e2e && npx playwright test --list tests/journeys.spec.ts -g "further education college|ISI-inspected|Ofsted lists without"` → the three tests listed.
|
||
|
||
- [ ] **Step 3: Run them against staging to see them fail**
|
||
|
||
Only once PR 1 has merged and the daily and monthly Ofsted DAGs have run on staging: `cd e2e && BASE_URL=https://stx.schoolcompare.co.uk npx playwright test tests/journeys.spec.ts -g "further education college|ISI-inspected|Ofsted lists without" --reporter=line --retries=0`
|
||
Expected: all three FAIL for the right reason (staging still runs the old site: no FE label, no ISI section, the old badge rule), or skip where the old API lacks the fields; record which. If the DAGs have not run, record that and rely on the post-merge gate.
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add e2e/tests/journeys.spec.ts
|
||
git commit -m "test(e2e): further education, ISI and not-yet-inspected states (H4)"
|
||
```
|
||
|
||
### Task 14: Verify and open PR 2
|
||
|
||
- [ ] **Step 1:** `cd nextjs-app && npm run typecheck && npm test`; from the repo root `uv run --no-project --with-requirements requirements.txt python -c "from backend.app import app; print('backend imports OK')"` and `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests pipeline/tests scripts/ci/tests -q` → all pass.
|
||
- [ ] **Step 2:** `git status --short` → only `.DS_Store`. `git push -u origin fix/ofsted-inspectorates-site`.
|
||
- [ ] **Step 3:** Open the PR through the Gitea API (as Task 9): head `fix/ofsted-inspectorates-site`, title `fix(site): ISI, Ofsted-listed and further-education inspection states (H4, part 2 of 2)`. Body: what changed, including the Ofsted page link fix (`provider/21` 404s for every non-primary school); Task 13 Step 3's result; **merge only after the daily and monthly Ofsted DAGs have run on staging with PR 1; promote only after they have run on production** (before that, the fallback query gives no badge for the ISI and listed groups); ends with the Claude Code line.
|
||
- [ ] **Step 4:** Report the PR URL. Do not merge.
|
||
|
||
---
|
||
|
||
## After both PRs are live in production (Tudor promotes)
|
||
|
||
Validate against GIAS and Ofsted's school and FE MI: every "Not yet inspected" belongs to a school Ofsted's data lists without an inspection; every "Inspected by ISI" matches GIAS; every FE grade, date and report card matches the FE files; schools with no data show no badge; the compare page's Ofsted link resolves for a secondary and a special school. Then mark H4 resolved in the audit report artifact.
|