From 12c52244ee9711a884177cc02ec70da3b1d62279 Mon Sep 17 00:00:00 2001 From: Tudor Date: Mon, 5 Oct 2026 16:00:35 +0100 Subject: [PATCH 1/5] docs(spec): one current Ofsted status per school MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design for audit findings C1 and M1. A grade carried forward from an older ungraded visit is shown under a newer inspection's date ("Good · 2025" for Rabbsfarm, whose 2025 inspection gave no grade), and "Inspected" dates show the last graded inspection rather than the latest visit. The rule moves into int_ofsted_latest and a new fact_ofsted_latest mart, shipped as a pipeline PR and then a backend/UI PR. Co-Authored-By: Claude Opus 5.5 --- ...2026-10-05-ofsted-current-status-design.md | 255 ++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md diff --git a/docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md b/docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md new file mode 100644 index 0000000..abee389 --- /dev/null +++ b/docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md @@ -0,0 +1,255 @@ +# Ofsted Current Status — Design + +**Date:** 2026-10-05 +**Status:** approved design, not yet implemented +**Scope:** `pipeline/transform` Ofsted models, `backend/data_loader.py`, list and +detail API Ofsted fields, search and map badges, school-page Ofsted section, +compare Ofsted rows, Typesense rating, sitemap +**Fixes:** audit findings C1, M1 and (as a side effect) M2 and part of H3, +from the 3 Oct 2026 accuracy audit + +## Goal + +Never show an Ofsted grade under a date it was not awarded or confirmed on, and +always date "Inspected" by the school's latest visit. + +## The problem + +Ofsted's management information gives each school at most three inspections: +the latest graded inspection (date G, an overall grade or "Not judged", area +grades), the latest ungraded inspection (date U, an outcome sentence) and the +latest report card (date RC). + +The site derives a grade and a date from these with two independent rules: + +- grade = the graded inspection's overall grade, or else the grade parsed from + the ungraded outcome ("School remains Good" → 2); +- date = G, or else U. + +The two rules can pick different inspections. Every inspection from +September 2024 to November 2025 was graded with "Not judged" overall, so the +grade falls back to an older ungraded visit while the date stays the new one. + +- **C1.** Rabbsfarm Primary School (102408) shows "Good · 2025". Ofsted's + 17 June 2025 inspection gave no overall grade and rated quality of education, + behaviour and leadership Requires Improvement. The "Good" comes from an + ungraded visit on 6 February 2020. Site-wide, 932 badges pair a + carried-forward grade with a newer inspection's year, and 147 of them say Good + or Outstanding while that inspection rated an area Requires Improvement or + Inadequate. Acre Wood Academy (151783) reads "Good · 2024" though the + October 2024 inspection rated all four areas Inadequate. +- **M1.** When a newer ungraded visit exists, the page shows the older graded + date. Washwood Heath Academy (139888) reads "Inspected 3 Mar 2020 · 4+ years + ago"; Ofsted visited on 21 May 2025. 667 schools. + +Ofsted's own provider page for Rabbsfarm leads with the 2025 area judgements and +"From September 2024, Ofsted no longer makes an overall effectiveness +judgement". It shows no overall grade. + +The rule is also implemented four times: `dim_school.ofsted_grade` (feeds +Typesense), the list SQL in `data_loader.py`, `_ofsted_block`, and two separate +"latest row" picks over `marts.fact_ofsted_inspection` by `inspection_date`, +which tie arbitrarily on duplicate monthly rows. + +## Non-goals + +- Predecessor inspections (audit M10): a grade Ofsted attributes to a previous + URN stays unlabelled. +- Post-16 and ISI-inspected schools (H4) and the "Not yet inspected" label. +- The compare page's broken Ofsted link (M4). +- Report-card display, which is unchanged. + +## The rule + +Computed once per URN in `int_ofsted_latest`. + +**Latest visit:** the newest of RC, G and U. + +- `latest_visit_date` +- `latest_visit_kind`: `report_card`, `graded` or `ungraded` +- `latest_visit_outcome`: the ungraded outcome text when the kind is `ungraded`, + otherwise null + +**Current grade:** the overall grade still in force, if any. + +| Situation | `current_grade` | `current_grade_date` | `current_grade_basis` | +|---|---|---|---| +| A report card exists | null | null | null | +| Latest is graded, overall 1–4 | that grade | G | `graded` | +| Latest is graded, "Not judged" | null | null | null | +| Latest is ungraded, outcome "School remains X…" (any qualifier) | X | U | `confirmed` | +| Latest is ungraded, any other outcome ("Standards maintained", "Improved significantly", "Some aspects not as strong") | the graded inspection's overall grade if it is 1–4, else null | G when a grade is kept | `graded` when a grade is kept | +| No inspection | null | null | null | + +Report cards began in November 2025, after the last legacy graded and ungraded +inspections, so when a school has a report card it is always the latest visit, +and no legacy grade stays in force beside it. Ties between G, U and RC on the +same date resolve in the order report card, graded, ungraded. + +Invariants: `current_grade` is null or 1–4; `current_grade_date <= +latest_visit_date`; `current_grade_basis` is null exactly when `current_grade` +is null. + +### Expected results (Ofsted MI as at 31 Aug 2026) + +| URN | School | Ofsted data | Latest visit | Current grade | +|---|---|---|---|---| +| 102408 | Rabbsfarm Primary School | G 17 Jun 2025 Not judged; U 6 Feb 2020 remains Good | graded, 17 Jun 2025 | none | +| 151783 | Acre Wood Academy | G 1 Oct 2024 Not judged; U 14 Mar 2023 remains Good (Concerns) | graded, 1 Oct 2024 | none | +| 139888 | Washwood Heath Academy | G 3 Mar 2020 Good; U 21 May 2025 Standards maintained | ungraded, 21 May 2025, "Standards maintained" | Good, 3 Mar 2020, graded | +| 104762 | Robins Lane Community Primary | G 7 Jan 2020 Good; U 18 Jul 2024 School remains Good | ungraded, 18 Jul 2024 | Good, 18 Jul 2024, confirmed | +| 100094 | Royal Free Hospital Children's School | G 9 Oct 2019 Outstanding; U 5 Feb 2025 Some aspects not as strong | ungraded, 5 Feb 2025 | Outstanding, 9 Oct 2019, graded | +| 136454 | Oakgrove School | U 13 Nov 2024 Standards maintained only | ungraded, 13 Nov 2024 | none | +| 137086 | Bishop Stopford School | G 1 Apr 2025 Not judged | graded, 1 Apr 2025 | none | +| 110048 | The Willink School | U 5 Oct 2023 remains Good; RC 6 May 2026 | report card, 6 May 2026 | none (report card shown) | +| 149612 | St Michael's Catholic School | RC 10 Feb 2026 only | report card, 10 Feb 2026 | none (report card shown) | + +## What each page shows + +**Search and map badge** (`buildOfstedListBadge`), first match wins: + +1. Report card: "Report Card · *RC year*" (unchanged) +2. Current grade: "*Grade* · *year of `current_grade_date`*" +3. Latest visit: "Inspected · *year of `latest_visit_date`*" +4. "Not yet inspected" (unchanged) + +**School page** (`OfstedSection`, both phases): + +- Title date: "Inspected *latest visit date*". +- Headline: the report card; or the current grade with a source line + ("Graded inspection, 6 July 2016" or "Confirmed at an ungraded inspection, + 14 March 2023"); or "No overall grade" with "Ofsted stopped giving overall + grades in September 2024". +- "Latest visit" line when the latest visit is not the grade's source, e.g. + "Ungraded inspection, 13 Nov 2024: Standards maintained". +- The area grid shows the graded inspection's judgements through + `ofstedLegacyAreas()`, dated by that inspection when it is not the latest + visit. The primary and secondary no-grade branches merge into one; the + secondary branch's four hard-coded areas (audit M2) go with it. + +**Compare:** `ofstedDisplay` returns `report_card`, `graded`, `confirmed`, +`no_overall_grade` or `none`. The "Latest Ofsted inspection", "Result" and +"Inspected" rows use the same fields as the school page. + +## Delivery + +Two pull requests. The mart columns exist before anything reads them, so +neither needs compatibility code. + +### PR 1: pipeline (additive) + +- `stg_ofsted_inspections`: keep `graded_inspection_date`, + `ungraded_inspection_date` and `rc_inspection_date` as separate typed + columns, with the report-card date's existing guard. Keep `inspection_date` + (graded, else ungraded) for the current backend. Keep a row when any of the + three dates is present, so report-card-only schools are no longer dropped + (part of H3: 123 schools). +- `int_ofsted_latest`: pick one row per URN by `latest_visit_date` descending, + then `rc_inspection_date`, `ungraded_inspection_date` and + `graded_inspection_date` descending (nulls last). A duplicate monthly row that + carries a newer report card therefore always wins. Add the five status + columns. +- New mart `marts.fact_ofsted_latest`: one row per URN from `int_ofsted_latest` + with every column the pages need (status, area grades, report-card grades, + ungraded outcome, report URL). It does not join `dim_school`, so only the + monthly Ofsted DAG builds it. +- `dim_school` is not changed in PR 1: the daily DAG does not rebuild + `int_ofsted_latest`, and reading a column that the monthly DAG has not yet + built would fail the daily run. +- Visible effect: report-card-only schools gain their report card, because the + backend's existing reads of `fact_ofsted_inspection` now see their rows. + Nothing else changes. + +### PR 2: backend and UI (after the Ofsted DAG has run on PR 1) + +- `data_loader.py`: the list query and the batch query read + `marts.fact_ofsted_latest` instead of picking the latest row of + `fact_ofsted_inspection`. `_ofsted_block` reads the status columns and loses + its fallback to `ungraded_grade`. +- List rows: `ofsted_grade` becomes `current_grade`; `ofsted_date` becomes + `latest_visit_date`; new `ofsted_grade_date`. `ofsted_rc_date` stays. +- `ofsted` block: `overall_effectiveness` and `inspection_date` are the graded + inspection's own result and date (they label the area grid); new + `current_grade` `{grade, date, basis}` (or null) and `latest_visit` + `{date, kind, outcome}`; `grade_source` is removed. Report-card fields are + unchanged. +- `dim_school.ofsted_grade` becomes `current_grade` (Typesense's rating follows + at the next sync); `ofsted_date` becomes `latest_visit_date`. +- Sitemap: `lastmod` from `latest_visit_date`; `_PUBLISHABLE_FIELDS` also counts + a latest visit, so schools that lose a carried grade keep their sitemap entry. +- Front end: `lib/types.ts`, `buildOfstedListBadge`, `OfstedSection`, + `PrimarySchoolSections`, `SecondarySchoolSections`, `compareLogic.ofstedDisplay`, + `CompareAtAGlance`, `CompareOfsted`. Place-page counts need no change. +- Delete `buildOfstedHeroChip` and `buildSchoolSummary` in their own commit: + nothing renders them and they encode the old rule. + +## Testing + +**PR 1** + +- dbt unit tests on `int_ofsted_latest`, one per table row above plus + "report-card only" and "duplicate rows, newer report card wins". +- Schema tests on `fact_ofsted_latest`: unique, not-null `urn`; accepted values + for `latest_visit_kind` and `current_grade_basis`; `current_grade` null or + 1–4; `current_grade_date <= latest_visit_date`. +- Run locally against a throwaway Postgres from `pgserver` (no Docker here). If + that fails, they still run in the Ofsted DAG's `dbt build`, which fails on any + broken case. +- `pipeline/tests/test_dag_selectors.py` (PR #181) keeps passing. + +**PR 2** + +- pytest: contract test for the list and `ofsted` fields (style of + `test_school_page_flag_fields.py`); `_ofsted_block` from a + `fact_ofsted_latest` row; sitemap publishability. +- Jest: a badge case per table row; `OfstedSection` for graded, confirmed, no + grade, the latest-visit line and a sixth-form area; `ofstedDisplay` kinds. + Rewrite tests that assert `carried_forward`. +- E2E (same PR): Rabbsfarm's search row says "Inspected · 2025" and its page + says "No overall grade" with quality of education Requires Improvement; a + confirmed school says "Confirmed at an ungraded inspection". The existing + report-card journey stays. + +## Rollout and verification + +1. Merge PR 1. On staging, run `school_data_monthly_ofsted`, then: + + ```sql + -- one row per school + select count(*) = count(distinct urn) from marts.fact_ofsted_latest; + -- the examples above + select urn, latest_visit_date, latest_visit_kind, latest_visit_outcome, + current_grade, current_grade_date, current_grade_basis + from marts.fact_ofsted_latest + where urn in (102408, 151783, 139888, 104762, 100094, 136454, 137086, 110048, 149612); + -- C1: a grade in force although the latest inspection gave none (expect 0) + select count(*) from marts.fact_ofsted_latest + where current_grade is not null and latest_visit_kind = 'graded' + and overall_effectiveness is null; + -- invariant (expect 0) + select count(*) from marts.fact_ofsted_latest where current_grade_date > latest_visit_date; + ``` + + Check through the API that St Michael's Catholic School (149612) shows its + report card. +2. Promote PR 1 to production; run the Ofsted DAG there; repeat the checks. +3. Merge PR 2. Let the daily DAG run (or trigger it) so `dim_school` and + Typesense pick up the change; run the E2E journeys; re-run the audit's C1, + M1 and M2 checks against staging: expect 0. +4. Promote PR 2; repeat the audit checks on production. + +## Expected visible change + +About 932 schools change from a grade badge dated by a no-grade inspection +("Good · 2025") to "Inspected · 2025". Counts of Good and Outstanding schools +on place pages fall by the same schools, and Typesense's rating changes for +them. Dates beside a grade can move earlier (to the inspection that awarded +it); "Inspected" dates move later (to the latest visit). + +## Risks + +- Ofsted changes its MI columns most months. Unknown grade text parses to null + (`safe_numeric`), which degrades to "Inspected · year", never to a wrong grade. +- PR 2 depends on the Ofsted DAG having run on the target environment after PR + 1. If PR 2 is promoted first, the backend reads a missing table: promote in + order. From 3be902e98f9b8aac06d04361212b42e721317258 Mon Sep 17 00:00:00 2001 From: Tudor Date: Mon, 5 Oct 2026 16:20:49 +0100 Subject: [PATCH 2/5] docs(plan): implementation plan for the current Ofsted status Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-10-05-ofsted-current-status.md | 1593 +++++++++++++++++ 1 file changed, 1593 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-05-ofsted-current-status.md diff --git a/docs/superpowers/plans/2026-10-05-ofsted-current-status.md b/docs/superpowers/plans/2026-10-05-ofsted-current-status.md new file mode 100644 index 0000000..d8944c8 --- /dev/null +++ b/docs/superpowers/plans/2026-10-05-ofsted-current-status.md @@ -0,0 +1,1593 @@ +# Ofsted Current Status Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Compute one current Ofsted status per school in dbt and make every page read it, so no grade is shown under a date it was not awarded or confirmed on (C1), and "Inspected" always means the latest visit (M1). + +**Architecture:** `stg_ofsted_inspections` stops merging the graded, ungraded and report-card dates; `int_ofsted_latest` applies the rule and a new `marts.fact_ofsted_latest` (one row per URN) exposes it. PR 1 ships the pipeline change only (additive). PR 2, after the Ofsted DAG has run, switches `dim_school`, the backend and the UI to the new columns. + +**Tech Stack:** dbt-postgres ~1.10 (unit tests in YAML), FastAPI + pandas + SQLAlchemy, Next.js 15 / React / TypeScript, Jest + Testing Library, Playwright. + +**Spec:** `docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md` + +## Global Constraints + +- The rule table and expected results in the spec are the contract; dbt unit tests encode them row for row. +- PR 1 must not change `dim_school` and must keep `fact_ofsted_inspection.inspection_date` not null. +- PR 2 must not be merged before `school_data_monthly_ofsted` has run on staging with PR 1, nor promoted before it has run on production. +- No compatibility code in the backend for marts built before PR 1 (agreed in brainstorming). +- Copy follows the site's factual rule: print what Ofsted published; no derived summaries. Dates in prose use long months ("14 March 2023"). +- Commits end with `Co-Authored-By: Claude Opus 5.5 `; PR bodies end with the Claude Code line. Never push to `main`. + +## Review Focus + +1. A school with only an ungraded visit and an outcome that names no grade ("Standards maintained") must show "Inspected · year" and the outcome, never "Not rated" or a grade. Pinned in Task 3 (unit test `ungraded_only_standards_maintained`) and Task 9 (Jest). +2. A graded inspection and an ungraded visit on the same day must resolve to the graded one (spec tie order). Pinned in Task 3 (`same_day_graded_wins`). +3. Area grades with the MI sentinel 9 must never become a current grade. Pinned in Task 3 (`overall_sentinel_is_not_a_grade`). +4. A list row with a current grade but no `ofsted_grade_date` (should not happen, but nulls do) must still render a badge without a year rather than crash. Pinned in Task 8 (Jest). +5. Report-card schools must keep exactly today's display. Pinned in Task 3 (`report_card_wins`), Task 8 and the existing E2E report-card journey. + +## File Map + +**PR 1 — branch `fix/ofsted-current-status-pipeline`** + +- Modify `pipeline/transform/models/staging/stg_ofsted_inspections.sql` — three separate dates, report-card-only rows kept, compatibility `inspection_date`. +- Create `pipeline/transform/models/staging/stg_ofsted_inspections.yml` — unit tests for the staging model. +- Modify `pipeline/transform/models/intermediate/int_ofsted_latest.sql` — the rule. +- Create `pipeline/transform/models/intermediate/int_ofsted_latest.yml` — unit tests, one per rule row. +- Create `pipeline/transform/models/marts/fact_ofsted_latest.sql` — the mart. +- Modify `pipeline/transform/models/marts/_marts_schema.yml` — schema tests for the mart. +- Create `pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql` — invariants. + +**PR 2 — branch `fix/ofsted-current-status-site` (from `main`)** + +- Modify `pipeline/transform/models/marts/dim_school.sql` — `ofsted_grade`/`ofsted_date` from the status columns. +- Modify `backend/models.py` — `FactOfstedLatest`. +- Modify `backend/data_loader.py` — list query join, batch query, `_ofsted_block`. +- Modify `backend/schemas.py` — `ofsted_grade_date` in `SCHOOL_COLUMNS`. +- Modify `backend/app.py` — `_PUBLISHABLE_FIELDS`. +- Tests: `backend/tests/test_supplementary_enrichment.py`, `test_supplementary_batch.py`, `test_compare_enrichment.py`, new `backend/tests/test_ofsted_status_payload.py`. +- Modify `nextjs-app/lib/types.ts`, `nextjs-app/lib/utils.ts` (`buildOfstedListBadge`; delete `buildOfstedHeroChip`, `buildSchoolSummary`), `nextjs-app/lib/compareLogic.ts` (`ofstedDisplay`). +- Create `nextjs-app/lib/ofstedStatus.ts` — the two sentences both pages print. +- Modify `nextjs-app/components/school/OfstedSection.tsx`, `PrimarySchoolSections.tsx`, `SecondarySchoolSections.tsx`, `components/compare/CompareAtAGlance.tsx`, `components/compare/CompareOfsted.tsx`. +- Tests: `nextjs-app/__tests__/lib/utils.test.ts`, `__tests__/lib/compareLogic.test.ts`, new `__tests__/lib/ofstedStatus.test.ts`, new `__tests__/components/OfstedSection.test.tsx`, `__tests__/components/CompareOfsted.test.tsx`, `__tests__/support/schoolFixtures.ts`, `__tests__/components/schoolDetail.characterization.test.tsx`. +- Modify `e2e/tests/journeys.spec.ts` — Rabbsfarm and a confirmed-grade journey. + +--- + +## PR 1 — pipeline + +### Task 1: Local dbt harness + +CI has no dbt. Unit tests run inside every `dbt build` on Airflow, but they must also pass locally before the PR. There is no Docker or Postgres on this machine; `pgserver` bundles Postgres binaries in a pip wheel. + +**Files:** none in the repo. Work in `$SCRATCH/dbt_local` (the session scratchpad). + +- [ ] **Step 1: Start a throwaway Postgres** + +```bash +mkdir -p "$SCRATCH/dbt_local" && cd "$SCRATCH/dbt_local" +cat > start_pg.py <<'EOF' +import pgserver, sys, time +srv = pgserver.get_server(sys.argv[1], cleanup_mode=None) +print(srv.get_uri()) +srv.psql("create database school_compare;") if "school_compare" not in srv.psql("select datname from pg_database;") else None +EOF +uv run --no-project --with pgserver python start_pg.py "$SCRATCH/dbt_local/pgdata" +``` + +Expected: a URI such as `postgresql://postgres:@/postgres?host=/…/pgdata`. Note the socket directory after `host=`. + +- [ ] **Step 2: Create the raw source table with the tap's columns** + +```bash +uv run --no-project --with pgserver python - <<'EOF' +import pgserver, os +srv = pgserver.get_server(os.environ["SCRATCH"] + "/dbt_local/pgdata", cleanup_mode=None) +cols = ["inspection_date","inspection_type","event_type_grouping","overall_effectiveness", + "quality_of_education","behaviour_and_attitudes","personal_development", + "effectiveness_of_leadership_and_management","early_years_provision","sixth_form_provision", + "ungraded_outcome","ungraded_inspection_date","rc_safeguarding_met","rc_inclusion", + "rc_curriculum_teaching","rc_achievement","rc_attendance_behaviour","rc_personal_development", + "rc_leadership_governance","rc_early_years","rc_sixth_form","rc_inspection_date","report_url"] +ddl = ", ".join(f'"{c}" text' for c in cols) +srv.psql(f'\\c school_compare\ncreate schema if not exists raw;\ndrop table if exists raw.ofsted_inspections;\ncreate table raw.ofsted_inspections (urn integer, {ddl});') +print("ok") +EOF +``` + +- [ ] **Step 3: Load the nine example schools from the 31 Aug 2026 MI** + +Map MI columns to raw columns with the tap's own resolver so the fixture is exactly what production loads: + +```bash +uv run --no-project --with ./pipeline/plugins/extractors/tap-uk-ofsted --with pgserver --with pandas python - <<'EOF' +import os, pandas as pd, pgserver +from tap_uk_ofsted.tap import OfstedInspectionsStream, TapUKOfsted +mi = pd.read_csv(os.environ["SCRATCH"] + "/audit/ofsted_mi_latest.csv", dtype=str, keep_default_na=False, encoding="cp1252") +stream = OfstedInspectionsStream(TapUKOfsted(config={})) +col_map = stream._resolve_columns(list(mi.columns)) +urns = {"102408","151783","139888","104762","100094","136454","137086","110048","149612"} +rows = mi[mi["URN"].isin(urns)] +srv = pgserver.get_server(os.environ["SCRATCH"] + "/dbt_local/pgdata", cleanup_mode=None) +def q(v): return "NULL" if v is None else "'" + str(v).replace("'", "''") + "'" +stmts = [] +for _, r in rows.iterrows(): + rec = {f: r[c] for f, c in col_map.items() if f != "urn"} + names = ", ".join(f'"{k}"' for k in rec) + stmts.append(f'insert into raw.ofsted_inspections (urn, {names}) values ({int(r["URN"])}, {", ".join(q(v) for v in rec.values())});') +srv.psql("\\c school_compare\n" + "\n".join(stmts)) +print(len(stmts), "rows") +EOF +``` + +Expected: `9 rows`. + +- [ ] **Step 4: Run dbt against it with today's models (baseline)** + +```bash +cd pipeline/transform +export PG_HOST= PG_PORT=5432 PG_USER=postgres PG_PASSWORD= PG_DATABASE=school_compare +uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main run --profiles-dir . --target dev \ + --target-path "$SCRATCH/dbt_local/target" --select stg_ofsted_inspections int_ofsted_latest fact_ofsted_inspection +``` + +Expected: 3 models OK. If `pgserver` cannot start on this machine, stop and report; the unit tests will then be proven only by the staging Ofsted DAG run, and the PR description must say so. + +### Task 2: `stg_ofsted_inspections` keeps the three dates apart + +**Files:** +- Modify: `pipeline/transform/models/staging/stg_ofsted_inspections.sql` +- Create: `pipeline/transform/models/staging/stg_ofsted_inspections.yml` + +**Interfaces:** +- Produces columns used by Task 3: `graded_inspection_date date`, `ungraded_inspection_date date`, `rc_inspection_date date` (null unless the row carries report-card grades), `inspection_date date` (= coalesce of graded, ungraded, report card; never null), plus every existing column unchanged. + +- [ ] **Step 1: Write the failing unit tests** + +`pipeline/transform/models/staging/stg_ofsted_inspections.yml`: + +```yaml +version: 2 + +unit_tests: + - name: stg_ofsted_keeps_the_three_dates_apart + description: The graded and ungraded dates must stay separate so the latest visit can be found. + model: stg_ofsted_inspections + given: + - input: source('raw', 'ofsted_inspections') + rows: + - {urn: 102408, inspection_date: '17/06/2025', ungraded_inspection_date: '06/02/2020', overall_effectiveness: 'Not judged', ungraded_outcome: 'School remains Good', rc_inspection_date: '17/06/2025', rc_inclusion: 'NULL'} + expect: + rows: + - {urn: 102408, graded_inspection_date: '2025-06-17', ungraded_inspection_date: '2020-02-06', rc_inspection_date: null, inspection_date: '2025-06-17', ungraded_grade: 2} + + - name: stg_ofsted_keeps_report_card_only_rows + description: A school whose only inspection is a report card used to be dropped (audit H3). + model: stg_ofsted_inspections + given: + - input: source('raw', 'ofsted_inspections') + rows: + - {urn: 149612, inspection_date: 'NULL', ungraded_inspection_date: 'NULL', rc_inspection_date: '10/02/2026', rc_inclusion: 'Expected standard', rc_safeguarding_met: 'Met'} + - {urn: 1, inspection_date: 'NULL', ungraded_inspection_date: 'NULL', rc_inspection_date: 'NULL'} + expect: + rows: + - {urn: 149612, graded_inspection_date: null, ungraded_inspection_date: null, rc_inspection_date: '2026-02-10', inspection_date: '2026-02-10', rc_inclusion: 3} +``` + +- [ ] **Step 2: Run them to see them fail** + +```bash +uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main test --profiles-dir . --target dev \ + --target-path "$SCRATCH/dbt_local/target" --select "stg_ofsted_inspections,test_type:unit" +``` + +Expected: FAIL (`graded_inspection_date` does not exist). + +- [ ] **Step 3: Rewrite the model** + +Replace the whole of `stg_ofsted_inspections.sql` with: + +```sql +-- Staging model: Ofsted inspection records +-- Handles both OEIF (pre-Nov 2025) and Report Card (post-Nov 2025) frameworks. +-- +-- Ofsted's MI carries up to three inspections per school: the latest graded +-- one, the latest ungraded one and the latest report card. Their dates stay +-- separate here so int_ofsted_latest can tell which came last and which one a +-- grade belongs to (docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md). + +with source as ( + select * from {{ source('raw', 'ofsted_inspections') }} +), + +renamed as ( + select + cast(urn as integer) as urn, + to_date(nullif(trim(inspection_date), 'NULL'), 'DD/MM/YYYY') as graded_inspection_date, + to_date(nullif(trim(ungraded_inspection_date), 'NULL'), 'DD/MM/YYYY') as ungraded_inspection_date, + inspection_type, + event_type_grouping as framework, + + -- OEIF grades (1-4 scale; 9 = not applicable) + {{ safe_numeric('overall_effectiveness') }}::integer as overall_effectiveness, + {{ safe_numeric('quality_of_education') }}::integer as quality_of_education, + {{ safe_numeric('behaviour_and_attitudes') }}::integer as behaviour_attitudes, + {{ safe_numeric('personal_development') }}::integer as personal_development, + {{ safe_numeric('effectiveness_of_leadership_and_management') }}::integer as leadership_management, + {{ safe_numeric('early_years_provision') }}::integer as early_years_provision, + {{ safe_numeric('sixth_form_provision') }}::integer as sixth_form_provision, + + -- Ungraded (Section 8) inspection outcome — free text, plus the grade + -- it confirms ("School remains Good" → 2); null for outcomes that + -- name no grade. + nullif(trim(ungraded_outcome), 'NULL') as ungraded_outcome, + {{ parse_ungraded_outcome('ungraded_outcome') }}::integer as ungraded_grade, + + -- Report Card fields (post-Nov 2025 framework), 5-point scale: + -- 1 Exceptional · 2 Strong standard · 3 Expected standard + -- · 4 Needs attention · 5 Urgent improvement + case lower(trim(nullif(rc_safeguarding_met, 'NULL'))) + when 'met' then true + when 'not met' then false + end as rc_safeguarding_met, + {{ parse_report_card_grade('rc_inclusion') }}::integer as rc_inclusion, + {{ parse_report_card_grade('rc_curriculum_teaching') }}::integer as rc_curriculum_teaching, + {{ parse_report_card_grade('rc_achievement') }}::integer as rc_achievement, + {{ parse_report_card_grade('rc_attendance_behaviour') }}::integer as rc_attendance_behaviour, + {{ parse_report_card_grade('rc_personal_development') }}::integer as rc_personal_development, + {{ parse_report_card_grade('rc_leadership_governance') }}::integer as rc_leadership_governance, + {{ parse_report_card_grade('rc_early_years') }}::integer as rc_early_years, + {{ parse_report_card_grade('rc_sixth_form') }}::integer as rc_sixth_form, + + -- Start date of the latest FULL inspection (the report-card + -- inspection in the renewed framework). Only kept when the row + -- carries report-card grades, because in legacy-format files this + -- column is the legacy inspection date. + to_date(nullif(trim(rc_inspection_date), 'NULL'), 'DD/MM/YYYY') as rc_inspection_date_raw, + + nullif(trim(report_url), 'NULL') as report_url + from source + where urn is not null +), + +dated as ( + select + *, + case + when rc_safeguarding_met is not null + or rc_inclusion is not null + or rc_curriculum_teaching is not null + or rc_achievement is not null + or rc_attendance_behaviour is not null + or rc_personal_development is not null + or rc_leadership_governance is not null + then rc_inspection_date_raw + end as rc_inspection_date + from renamed +) + +select + *, + -- For readers that predate the separate dates (fact_ofsted_inspection and + -- the backend until it reads fact_ofsted_latest). Never null below. + coalesce(graded_inspection_date, ungraded_inspection_date, rc_inspection_date) as inspection_date +from dated +where coalesce(graded_inspection_date, ungraded_inspection_date, rc_inspection_date) is not null +``` + +- [ ] **Step 4: Run the staging unit tests and the baseline models** + +```bash +uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main build --profiles-dir . --target dev \ + --target-path "$SCRATCH/dbt_local/target" --select stg_ofsted_inspections int_ofsted_latest fact_ofsted_inspection +``` + +Expected: unit tests PASS; `fact_ofsted_inspection`'s `not_null` on `inspection_date` PASS; 9 rows in `staging.stg_ofsted_inspections` (St Michael's 149612 now present). + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/staging/stg_ofsted_inspections.sql pipeline/transform/models/staging/stg_ofsted_inspections.yml +git commit -m "feat(pipeline): keep Ofsted's graded, ungraded and report-card dates apart" -m "int_ofsted_latest needs to know which inspection came last. Report-card-only schools are no longer dropped; inspection_date stays for current readers and now falls back to the report-card date." -m "Co-Authored-By: Claude Opus 5.5 " +``` + +### Task 3: `int_ofsted_latest` computes the current status + +**Files:** +- Modify: `pipeline/transform/models/intermediate/int_ofsted_latest.sql` +- Create: `pipeline/transform/models/intermediate/int_ofsted_latest.yml` + +**Interfaces:** +- Consumes: Task 2's columns. +- Produces (Task 4 and PR 2): `latest_visit_date date`, `latest_visit_kind text` (`report_card`|`graded`|`ungraded`), `latest_visit_outcome text`, `current_grade integer` (1–4 or null), `current_grade_date date`, `current_grade_basis text` (`graded`|`confirmed`), `graded_inspection_date`, `ungraded_inspection_date`, plus every column the old model selected. + +- [ ] **Step 1: Write the failing unit tests** (one per spec rule row and Review Focus item) + +`pipeline/transform/models/intermediate/int_ofsted_latest.yml`: + +```yaml +version: 2 + +unit_tests: + - name: graded_not_judged_has_no_grade + description: Rabbsfarm (102408). The 2025 inspection gave no overall grade, so the 2020 "remains Good" is not carried forward (audit C1). + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 102408, graded_inspection_date: '2025-06-17', ungraded_inspection_date: '2020-02-06', overall_effectiveness: null, ungraded_grade: 2, ungraded_outcome: 'School remains Good'} + expect: + rows: + - {urn: 102408, latest_visit_date: '2025-06-17', latest_visit_kind: graded, latest_visit_outcome: null, current_grade: null, current_grade_date: null, current_grade_basis: null} + + - name: graded_with_overall_grade + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 1, graded_inspection_date: '2019-06-01', overall_effectiveness: 2} + expect: + rows: + - {urn: 1, latest_visit_date: '2019-06-01', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2019-06-01', current_grade_basis: graded} + + - name: ungraded_remains_good_confirms_the_grade + description: Robins Lane (104762). Graded Good 2020, "School remains Good" July 2024 — Good, dated by the confirming visit. + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 104762, graded_inspection_date: '2020-01-07', ungraded_inspection_date: '2024-07-18', overall_effectiveness: 2, ungraded_grade: 2, ungraded_outcome: 'School remains Good'} + expect: + rows: + - {urn: 104762, latest_visit_date: '2024-07-18', latest_visit_kind: ungraded, latest_visit_outcome: 'School remains Good', current_grade: 2, current_grade_date: '2024-07-18', current_grade_basis: confirmed} + + - name: post_2024_ungraded_keeps_graded_grade_with_its_own_date + description: Washwood Heath (139888). Graded Good 2020, "Standards maintained" May 2025 — Good, dated 2020; latest visit May 2025 (audit M1). + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 139888, graded_inspection_date: '2020-03-03', ungraded_inspection_date: '2025-05-21', overall_effectiveness: 2, ungraded_grade: null, ungraded_outcome: 'Standards maintained'} + expect: + rows: + - {urn: 139888, latest_visit_date: '2025-05-21', latest_visit_kind: ungraded, latest_visit_outcome: 'Standards maintained', current_grade: 2, current_grade_date: '2020-03-03', current_grade_basis: graded} + + - name: ungraded_only_standards_maintained + description: Oakgrove (136454). Only an ungraded visit, outcome names no grade. + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 136454, ungraded_inspection_date: '2024-11-13', ungraded_grade: null, ungraded_outcome: 'Standards maintained'} + expect: + rows: + - {urn: 136454, latest_visit_date: '2024-11-13', latest_visit_kind: ungraded, latest_visit_outcome: 'Standards maintained', current_grade: null, current_grade_date: null, current_grade_basis: null} + + - name: report_card_wins + description: The Willink School (110048). A report card is the latest visit and no legacy grade stays in force. + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 110048, ungraded_inspection_date: '2023-10-05', ungraded_grade: 2, ungraded_outcome: 'School remains Good', rc_inspection_date: '2026-05-06', rc_inclusion: 3} + expect: + rows: + - {urn: 110048, latest_visit_date: '2026-05-06', latest_visit_kind: report_card, latest_visit_outcome: null, current_grade: null, current_grade_date: null, current_grade_basis: null} + + - name: duplicate_rows_newer_report_card_wins + description: Monthly loads can leave an older row beside a newer one for the same graded date; the row with the report card must win (audit H3). + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 138186, graded_inspection_date: '2023-06-13', overall_effectiveness: 3} + - {urn: 138186, graded_inspection_date: '2023-06-13', overall_effectiveness: 3, rc_inspection_date: '2026-06-02', rc_inclusion: 3} + expect: + rows: + - {urn: 138186, latest_visit_date: '2026-06-02', latest_visit_kind: report_card, current_grade: null} + + - name: same_day_graded_wins + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 2, graded_inspection_date: '2024-03-01', ungraded_inspection_date: '2024-03-01', overall_effectiveness: 1, ungraded_grade: 2, ungraded_outcome: 'School remains Good'} + expect: + rows: + - {urn: 2, latest_visit_kind: graded, current_grade: 1, current_grade_basis: graded} + + - name: overall_sentinel_is_not_a_grade + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 3, graded_inspection_date: '2018-05-01', overall_effectiveness: 9} + expect: + rows: + - {urn: 3, latest_visit_kind: graded, current_grade: null} +``` + +- [ ] **Step 2: Run them to see them fail** + +```bash +uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main test --profiles-dir . --target dev \ + --target-path "$SCRATCH/dbt_local/target" --select "int_ofsted_latest,test_type:unit" +``` + +Expected: FAIL (`latest_visit_date` does not exist). + +- [ ] **Step 3: Rewrite the model** + +Replace the whole of `int_ofsted_latest.sql` with: + +```sql +-- Intermediate model: the current Ofsted status per URN +-- One row per school: its latest visit (report card, graded or ungraded +-- inspection) and the overall grade still in force, if any. A grade is dated +-- by the inspection that awarded or confirmed it, never by a later visit. +-- Rule and examples: docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md + +with inspections as ( + select + *, + -- Report cards began in Nov 2025, after the last legacy inspections, + -- so a school's report card is always its latest visit. + coalesce( + rc_inspection_date, + greatest(graded_inspection_date, ungraded_inspection_date) + ) as latest_visit_date + from {{ ref('stg_ofsted_inspections') }} +), + +ranked as ( + select + *, + -- Monthly loads can leave several rows per school. The newest visit + -- wins; the tie-breaks keep the choice deterministic. + row_number() over ( + partition by urn + order by latest_visit_date desc, + rc_inspection_date desc nulls last, + graded_inspection_date desc nulls last, + ungraded_inspection_date desc nulls last + ) as rn + from inspections +), + +latest as ( + select + *, + case + when rc_inspection_date is not null then 'report_card' + when graded_inspection_date = latest_visit_date then 'graded' + else 'ungraded' + end as latest_visit_kind + from ranked + where rn = 1 +), + +graded as ( + select + *, + -- The grade still in force. A graded inspection's own overall grade + -- (1-4; "Not judged" and the sentinel 9 are no grade); else an + -- ungraded visit's "School remains X"; else, after an ungraded visit + -- that names no grade, the graded inspection's grade. + case + when latest_visit_kind = 'graded' and overall_effectiveness between 1 and 4 + then 'graded_latest' + when latest_visit_kind = 'ungraded' and ungraded_grade is not null + then 'confirmed' + when latest_visit_kind = 'ungraded' and overall_effectiveness between 1 and 4 + then 'graded_earlier' + end as grade_case + from latest +) + +select + urn, + latest_visit_date, + latest_visit_kind, + case when latest_visit_kind = 'ungraded' then ungraded_outcome end as latest_visit_outcome, + case grade_case + when 'confirmed' then ungraded_grade + when 'graded_latest' then overall_effectiveness + when 'graded_earlier' then overall_effectiveness + end as current_grade, + case grade_case + when 'confirmed' then ungraded_inspection_date + when 'graded_latest' then graded_inspection_date + when 'graded_earlier' then graded_inspection_date + end as current_grade_date, + case grade_case + when 'confirmed' then 'confirmed' + when 'graded_latest' then 'graded' + when 'graded_earlier' then 'graded' + end as current_grade_basis, + graded_inspection_date, + ungraded_inspection_date, + inspection_date, + inspection_type, + framework, + overall_effectiveness, + quality_of_education, + behaviour_attitudes, + personal_development, + leadership_management, + early_years_provision, + sixth_form_provision, + ungraded_outcome, + ungraded_grade, + rc_safeguarding_met, + rc_inclusion, + rc_curriculum_teaching, + rc_achievement, + rc_attendance_behaviour, + rc_personal_development, + rc_leadership_governance, + rc_early_years, + rc_sixth_form, + rc_inspection_date, + report_url +from graded +``` + +Note `dim_school` still reads `overall_effectiveness`, `ungraded_grade`, `inspection_date` and `framework` from this model; all are kept. + +- [ ] **Step 4: Run the unit tests** + +Same command as Step 2. Expected: 9 PASS. + +- [ ] **Step 5: Commit** — `feat(pipeline): work out each school's current Ofsted status once` + +### Task 4: `fact_ofsted_latest` mart, schema tests and invariants + +**Files:** +- Create: `pipeline/transform/models/marts/fact_ofsted_latest.sql` +- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (after `fact_ofsted_inspection`) +- Create: `pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql` + +**Interfaces:** Produces `marts.fact_ofsted_latest`, one row per URN, with every column `int_ofsted_latest` selects. PR 2's `FactOfstedLatest` ORM model mirrors it. + +- [ ] **Step 1: Write the schema tests and the invariant test** + +Append to `_marts_schema.yml` after the `fact_ofsted_inspection` entry: + +```yaml + - name: fact_ofsted_latest + description: > + Current Ofsted status, one row per URN: the latest visit and the overall + grade still in force. See int_ofsted_latest for the rule. + columns: + - name: urn + tests: [not_null, unique] + - name: latest_visit_date + tests: [not_null] + - name: latest_visit_kind + tests: + - not_null + - accepted_values: + values: ['report_card', 'graded', 'ungraded'] + - name: current_grade + tests: + - accepted_values: + values: [1, 2, 3, 4] + quote: false + - name: current_grade_basis + tests: + - accepted_values: + values: ['graded', 'confirmed'] +``` + +`tests/assert_ofsted_current_grade_consistent.sql`: + +```sql +-- A grade is dated by the inspection that awarded or confirmed it, never +-- after the latest visit, and has a date and a basis exactly when it exists. +-- A report card leaves no legacy grade in force. + +select urn +from {{ ref('fact_ofsted_latest') }} +where current_grade_date > latest_visit_date + or (current_grade is null) <> (current_grade_date is null) + or (current_grade is null) <> (current_grade_basis is null) + or (latest_visit_kind = 'report_card' and current_grade is not null) +``` + +- [ ] **Step 2: Run them to see them fail** — `dbt test --select fact_ofsted_latest assert_ofsted_current_grade_consistent` → fails: model missing. + +- [ ] **Step 3: Create the mart** + +`fact_ofsted_latest.sql`: + +```sql +-- Mart: current Ofsted status — one row per URN +-- The backend reads this instead of choosing the latest row of +-- fact_ofsted_inspection itself. The rule lives in int_ofsted_latest. + +select + urn, + latest_visit_date, + latest_visit_kind, + latest_visit_outcome, + current_grade, + current_grade_date, + current_grade_basis, + graded_inspection_date, + ungraded_inspection_date, + rc_inspection_date, + inspection_type, + framework, + overall_effectiveness, + quality_of_education, + behaviour_attitudes, + personal_development, + leadership_management, + early_years_provision, + sixth_form_provision, + ungraded_outcome, + ungraded_grade, + rc_safeguarding_met, + rc_inclusion, + rc_curriculum_teaching, + rc_achievement, + rc_attendance_behaviour, + rc_personal_development, + rc_leadership_governance, + rc_early_years, + rc_sixth_form, + report_url +from {{ ref('int_ofsted_latest') }} +``` + +- [ ] **Step 4: Build the Ofsted models end to end on the nine real schools** + +```bash +uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main build --profiles-dir . --target dev \ + --target-path "$SCRATCH/dbt_local/target" \ + --select stg_ofsted_inspections int_ofsted_latest fact_ofsted_inspection fact_ofsted_latest assert_ofsted_current_grade_consistent +``` + +Then run the spec's verification SQL against the local database and compare row for row with the spec's "Expected results" table: + +```sql +select urn, latest_visit_date, latest_visit_kind, latest_visit_outcome, + current_grade, current_grade_date, current_grade_basis +from marts.fact_ofsted_latest order by urn; +``` + +Expected (from the spec): 100094 ungraded 2025-02-05 "Some aspects not as strong" / 1, 2019-10-09, graded · 102408 graded 2025-06-17 / null · 104762 ungraded 2024-07-18 / 2, 2024-07-18, confirmed · 110048 report_card 2026-05-06 / null · 136454 ungraded 2024-11-13 / null · 137086 graded 2025-04-01 / null · 139888 ungraded 2025-05-21 / 2, 2020-03-03, graded · 149612 report_card 2026-02-10 / null · 151783 graded 2024-10-01 / null. + +- [ ] **Step 5: Check the selector rule from PR #181 still holds** + +`fact_ofsted_latest` refs only `int_ofsted_latest`, so the monthly Ofsted build (`int_ofsted_latest+`) builds it and the daily build does not. If PR #181 has merged, run `python -m pytest pipeline/tests/test_dag_selectors.py -q` on a branch that contains both; otherwise confirm with: + +```bash +uv run --no-project --with "dbt-postgres~=1.10.0" python -m dbt.cli.main ls --profiles-dir . --target dev --resource-type model \ + --select stg_gias_establishments+ stg_gias_links+ gias_code_names+ --exclude int_ks2_with_lineage+ int_ks4_with_lineage+ | grep -c fact_ofsted_latest +``` + +Expected: `0`. + +- [ ] **Step 6: Commit** — `feat(pipeline): add fact_ofsted_latest, one current Ofsted status per school` + +### Task 5: Open PR 1 + +- [ ] **Step 1:** `python -m pytest backend/tests pipeline/tests scripts/ci/tests -q` (CI's command, via `uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml`). Expected: all pass (no Python changed). +- [ ] **Step 2:** `git status --short` — only the intended files; push; open the PR via the Gitea API (basic auth from `git credential fill`). The body states: what changes, the one visible effect (report-card-only schools), the local unit-test and nine-school results, and the post-merge steps — run `school_data_monthly_ofsted` on staging, then the spec's verification SQL — and that PR 2 depends on that run. + +--- + +## PR 2 — backend and UI + +Branch `fix/ofsted-current-status-site` from `origin/main`. It does not need PR 1's code to build or test (tests use fixtures), but must not merge until the staging Ofsted DAG has run with PR 1. + +### Task 6: `dim_school` reads the status + +**Files:** Modify `pipeline/transform/models/marts/dim_school.sql:74-84`. + +- [ ] **Step 1: Change the Ofsted columns** + +```sql + -- Latest Ofsted (populated after monthly Ofsted pipeline runs). The grade + -- still in force and the latest visit, from int_ofsted_latest — never a + -- grade carried past a newer inspection. + {% if ofsted_relation is not none %} + o.current_grade as ofsted_grade, + o.latest_visit_date as ofsted_date, + o.framework as ofsted_framework + {% else %} +``` + +- [ ] **Step 2: Verify locally** — with the Task 1 harness plus a GIAS fixture is heavy; instead compile and check the SQL: `dbt compile --select dim_school` and confirm the compiled query selects `o.current_grade`. The behaviour is covered by Task 3's unit tests (same columns) and by the Task 13 staging check (Typesense rating for Rabbsfarm empty). +- [ ] **Step 3: Commit** — `fix(pipeline): dim_school's Ofsted grade is the one still in force` + +### Task 7: Backend reads `fact_ofsted_latest` + +**Files:** +- Modify: `backend/models.py` (add `FactOfstedLatest` after `FactOfstedInspection`) +- Modify: `backend/data_loader.py:21`, `:254-257`, `:336-352`, `:734-787`, `:1100-1115` +- Modify: `backend/schemas.py:576-579` +- Modify: `backend/app.py:101` +- Test: `backend/tests/test_supplementary_enrichment.py`, `backend/tests/test_supplementary_batch.py`, `backend/tests/test_compare_enrichment.py`, create `backend/tests/test_ofsted_status_payload.py` + +**Interfaces:** +- Consumes: `marts.fact_ofsted_latest` (Task 4 columns). +- Produces (Task 8 onward): list rows carry `ofsted_grade` (int|null), `ofsted_grade_date` (ISO date|null), `ofsted_date` (latest visit, ISO|null), `ofsted_framework`, `ofsted_rc_date`. The `ofsted` block carries `current_grade: {grade, date, basis} | None`, `latest_visit: {date, kind, outcome} | None`, `overall_effectiveness` (graded inspection's own, 1–4 or None), `inspection_date` (graded inspection's date), all existing area and report-card keys, `report_card`, `ofsted_page_url`; no `grade_source`. + +- [ ] **Step 1: Write the failing tests** + +Replace the row helper and the grade-source test in `test_supplementary_enrichment.py`: + +```python +def _row(**kw): + base = dict( + framework="RC", inspection_type=None, + graded_inspection_date=None, ungraded_inspection_date=None, rc_inspection_date=None, + latest_visit_date=None, latest_visit_kind=None, latest_visit_outcome=None, + current_grade=None, current_grade_date=None, current_grade_basis=None, + overall_effectiveness=None, quality_of_education=None, + behaviour_attitudes=None, personal_development=None, + leadership_management=None, early_years_provision=None, + sixth_form_provision=None, ungraded_outcome=None, ungraded_grade=None, + rc_safeguarding_met=None, rc_inclusion=None, rc_curriculum_teaching=None, + rc_achievement=None, rc_attendance_behaviour=None, + rc_personal_development=None, rc_leadership_governance=None, + rc_early_years=None, rc_sixth_form=None, report_url=None, + ) + base.update(kw) + return types.SimpleNamespace(**base) + + +def test_no_grade_is_carried_past_a_newer_inspection(): + # Rabbsfarm (102408): the 2025 inspection gave no overall grade. + block = _ofsted_block(_row( + graded_inspection_date=date(2025, 6, 17), ungraded_inspection_date=date(2020, 2, 6), + latest_visit_date=date(2025, 6, 17), latest_visit_kind="graded", + ungraded_grade=2, ungraded_outcome="School remains Good", quality_of_education=3, + ), urn=102408) + assert block["current_grade"] is None + assert block["overall_effectiveness"] is None + assert block["latest_visit"] == {"date": "2025-06-17", "kind": "graded", "outcome": None} + assert block["inspection_date"] == "2025-06-17" + assert "grade_source" not in block + + +def test_confirmed_grade_is_dated_by_the_confirming_visit(): + block = _ofsted_block(_row( + graded_inspection_date=date(2020, 1, 7), ungraded_inspection_date=date(2024, 7, 18), + latest_visit_date=date(2024, 7, 18), latest_visit_kind="ungraded", + latest_visit_outcome="School remains Good", overall_effectiveness=2, + current_grade=2, current_grade_date=date(2024, 7, 18), current_grade_basis="confirmed", + ), urn=104762) + assert block["current_grade"] == {"grade": 2, "date": "2024-07-18", "basis": "confirmed"} + assert block["overall_effectiveness"] == 2 + assert block["inspection_date"] == "2020-01-07" + + +def test_overall_sentinel_is_not_served_as_a_grade(): + assert _ofsted_block(_row(overall_effectiveness=9), urn=1)["overall_effectiveness"] is None +``` + +Delete `test_grade_source_graded_vs_carried_forward`. In `test_ofsted_block_carries_rc_inspection_date` and `test_ofsted_block_rc_inspection_date_none_when_absent`, replace `inspection_date=` with `graded_inspection_date=` and remove `ungraded_grade=2`. In `test_ofsted_block_keeps_existing_keys`, drop `"grade_source"` from the key list if present and add `"current_grade", "latest_visit"`. + +In `test_supplementary_batch.py`, rename the table key `"FactOfstedInspection"` to `"FactOfstedLatest"` in both the rows dict and the expected `session.queries` list, and rewrite `_ofsted_row` (one row per URN now; the mart already chose): + +```python +def _ofsted_row(urn, date, oe): + base = {f: None for f in ( + "framework", "inspection_type", "quality_of_education", "behaviour_attitudes", + "personal_development", "leadership_management", "early_years_provision", + "sixth_form_provision", "ungraded_outcome", "ungraded_grade", + "ungraded_inspection_date", "rc_inspection_date", "latest_visit_outcome", + "rc_safeguarding_met", "rc_inclusion", "rc_curriculum_teaching", "rc_achievement", + "rc_attendance_behaviour", "rc_personal_development", "rc_leadership_governance", + "rc_early_years", "rc_sixth_form", "report_url", + )} + when = types.SimpleNamespace(isoformat=lambda: date) + base.update(urn=urn, graded_inspection_date=when, latest_visit_date=when, + latest_visit_kind="graded", overall_effectiveness=oe, + current_grade=oe, current_grade_date=when, current_grade_basis="graded") + return types.SimpleNamespace(**base) +``` + +and give URN 1 a single row (`_ofsted_row(1, "2023-01-01", 2)`); the "keep the latest" comment becomes "one row per URN". + +In `test_compare_enrichment.py:15`, replace `"grade_source": "graded"` with `"current_grade": {"grade": 2, "date": "2023-01-01", "basis": "graded"}` and the assertion at `:89` with `assert school["ofsted"]["current_grade"]["basis"] == "graded"`. + +Create `backend/tests/test_ofsted_status_payload.py`: + +```python +"""The search badge reads ofsted_grade, ofsted_grade_date and ofsted_date from +list rows (nextjs-app/lib/utils.ts buildOfstedListBadge). A field the list +never sends would silently fall back to "Not yet inspected".""" + +import numpy as np +import pandas as pd +import pytest +from fastapi.testclient import TestClient + +from backend.schemas import SCHOOL_COLUMNS + + +def test_list_columns_include_the_status_fields(): + for field in ("ofsted_grade", "ofsted_grade_date", "ofsted_date", "ofsted_rc_date"): + assert field in SCHOOL_COLUMNS + + +def _df(): + return pd.DataFrame([{ + "urn": 102408, "school_name": "Rabbsfarm Primary School", "phase": "Primary", + "school_type": "Community school", "local_authority": "Hillingdon", + "address": "Gordon Road, Yiewsley, UB7 8AH", "postcode": "UB7 8AH", + "latitude": 51.51, "longitude": -0.47, "year": 202425, "rwm_expected_pct": 58.0, + "total_pupils": 60, "gias_total_pupils": 616, + "ofsted_grade": np.nan, "ofsted_grade_date": None, "ofsted_date": "2025-06-17", + "ofsted_framework": "Schools - S5", "ofsted_rc_date": None, + }]) + + +@pytest.fixture() +def client(monkeypatch): + from backend import app as app_module + monkeypatch.setattr(app_module, "load_school_data", _df) + monkeypatch.setattr(app_module, "load_latest_school_data", _df) + return TestClient(app_module.app) + + +def test_search_rows_carry_the_status_fields(client): + row = client.get("/api/schools?search=Rabbsfarm").json()["schools"][0] + assert row["ofsted_grade"] is None + assert row["ofsted_date"] == "2025-06-17" + assert "ofsted_grade_date" in row + + +def test_a_school_with_only_an_inspection_is_publishable(): + from backend.app import _has_publishable_data + assert _has_publishable_data({"rwm_expected_pct": None, "attainment_8_score": None, + "ofsted_grade": None, "ofsted_date": "2025-06-17"}) +``` + +If `/api/schools?search=` needs Typesense in tests, mirror the client fixture in `test_school_page_flag_fields.py` (it patches the same loaders and works without Typesense); adjust the query to whatever that file uses. + +- [ ] **Step 2: Run them to see them fail** + +`uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests/test_supplementary_enrichment.py backend/tests/test_supplementary_batch.py backend/tests/test_compare_enrichment.py backend/tests/test_ofsted_status_payload.py -q` → failures on `current_grade`, `FactOfstedLatest`, `ofsted_grade_date`. + +- [ ] **Step 3: Implement** + +`backend/models.py`, after `FactOfstedInspection`: + +```python +class FactOfstedLatest(Base): + """Current Ofsted status — one row per URN (pipeline: int_ofsted_latest). + + `current_grade` is the overall grade still in force, dated by the + inspection that awarded or confirmed it; `latest_visit_*` is the school's + most recent inspection of any kind. + """ + __tablename__ = "fact_ofsted_latest" + __table_args__ = MARTS + + urn = Column(Integer, primary_key=True) + latest_visit_date = Column(Date) + latest_visit_kind = Column(String(20)) + latest_visit_outcome = Column(String(100)) + current_grade = Column(Integer) + current_grade_date = Column(Date) + current_grade_basis = Column(String(20)) + graded_inspection_date = Column(Date) + ungraded_inspection_date = Column(Date) + rc_inspection_date = Column(Date) + inspection_type = Column(String(100)) + framework = Column(String(20)) + overall_effectiveness = Column(Integer) + quality_of_education = Column(Integer) + behaviour_attitudes = Column(Integer) + personal_development = Column(Integer) + leadership_management = Column(Integer) + early_years_provision = Column(Integer) + sixth_form_provision = Column(Integer) + ungraded_outcome = Column(String(100)) + ungraded_grade = Column(Integer) + rc_safeguarding_met = Column(Boolean) + rc_inclusion = Column(Integer) + rc_curriculum_teaching = Column(Integer) + rc_achievement = Column(Integer) + rc_attendance_behaviour = Column(Integer) + rc_personal_development = Column(Integer) + rc_leadership_governance = Column(Integer) + rc_early_years = Column(Integer) + rc_sixth_form = Column(Integer) + report_url = Column(Text) +``` + +`backend/data_loader.py`: +- import `FactOfstedLatest` alongside `FactOfstedInspection` (drop `FactOfstedInspection` from the import if nothing else uses it). +- In `_MAIN_QUERY` select list replace `foi.ofsted_grade, foi.ofsted_date, foi.ofsted_framework, foi.ofsted_rc_date,` with + +```sql + foi.current_grade AS ofsted_grade, + foi.current_grade_date AS ofsted_grade_date, + foi.latest_visit_date AS ofsted_date, + foi.framework AS ofsted_framework, + foi.rc_inspection_date AS ofsted_rc_date, +``` + +- Replace the `LEFT JOIN (SELECT DISTINCT ON (urn) … FROM marts.fact_ofsted_inspection ORDER BY …) foi ON s.urn = foi.urn` block with `LEFT JOIN marts.fact_ofsted_latest foi ON s.urn = foi.urn`. +- Replace `_ofsted_block` with: + +```python +def _iso(d): + return d.isoformat() if d else None + + +def _ofsted_block(o, urn: int) -> dict: + """Serialize a fact_ofsted_latest row for API responses. + + `current_grade` is the overall grade still in force, dated by the + inspection that awarded or confirmed it; `latest_visit` is the school's + most recent inspection of any kind. The rule lives in int_ofsted_latest + (docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md). + `overall_effectiveness` and `inspection_date` describe the graded + inspection itself and label its area judgements. `report_card` holds the + renewed-framework (Nov 2025) area judgements; safeguarding is a separate + boolean and never appears among the graded areas. + """ + block = { + "framework": o.framework, + "inspection_date": _iso(o.graded_inspection_date), + "rc_inspection_date": _iso(o.rc_inspection_date), + "inspection_type": o.inspection_type, + "overall_effectiveness": o.overall_effectiveness if o.overall_effectiveness in (1, 2, 3, 4) else None, + "current_grade": ( + {"grade": o.current_grade, "date": _iso(o.current_grade_date), "basis": o.current_grade_basis} + if o.current_grade is not None else None + ), + "latest_visit": ( + {"date": _iso(o.latest_visit_date), "kind": o.latest_visit_kind, "outcome": o.latest_visit_outcome} + if o.latest_visit_date else None + ), + "quality_of_education": o.quality_of_education, + "behaviour_attitudes": o.behaviour_attitudes, + "personal_development": o.personal_development, + "leadership_management": o.leadership_management, + "early_years_provision": o.early_years_provision, + "sixth_form_provision": o.sixth_form_provision, + "previous_overall": None, # Not available in new schema + "rc_safeguarding_met": o.rc_safeguarding_met, + "rc_inclusion": o.rc_inclusion, + "rc_curriculum_teaching": o.rc_curriculum_teaching, + "rc_achievement": o.rc_achievement, + "rc_attendance_behaviour": o.rc_attendance_behaviour, + "rc_personal_development": o.rc_personal_development, + "rc_leadership_governance": o.rc_leadership_governance, + "rc_early_years": o.rc_early_years, + "rc_sixth_form": o.rc_sixth_form, + "report_url": o.report_url, + "ofsted_page_url": ofsted_page_url(urn), + } + block["report_card"] = report_card_labels(block) + return block +``` + +- In `get_supplementary_data_batch`, replace the `_ofsted` body with: + +```python + # Ofsted — the mart already holds one current status per URN. + def _ofsted(): + rows = ( + db.query(FactOfstedLatest) + .filter(FactOfstedLatest.urn.in_(urns)) + .all() + ) + for o in rows: + result[o.urn]["ofsted"] = _ofsted_block(o, o.urn) +``` + +`backend/schemas.py`: add `"ofsted_grade_date",` after `"ofsted_grade",` in `SCHOOL_COLUMNS`. + +`backend/app.py:101`: `_PUBLISHABLE_FIELDS = ("rwm_expected_pct", "attainment_8_score", "ofsted_grade", "ofsted_date")` and add to the comment above it: "An inspection with no overall grade still gives a page something to state." + +Grep for any other reader of `grade_source` or `ungraded_carried_forward` in `backend/` and remove it. + +- [ ] **Step 4: Run the backend suite** + +`uv run --no-project --with-requirements requirements.txt --with pytest --with "httpx<0.28" --with pyyaml python -m pytest backend/tests -q` → all pass. + +- [ ] **Step 5: Commit** — `fix(backend): serve the current Ofsted status from fact_ofsted_latest` + +### Task 8: Types, badge and compare display + +**Files:** +- Modify: `nextjs-app/lib/types.ts:78-84`, `:91-128` +- Modify: `nextjs-app/lib/utils.ts` (`buildOfstedListBadge`) +- Modify: `nextjs-app/lib/compareLogic.ts:100-131` +- Create: `nextjs-app/lib/ofstedStatus.ts` +- Test: `nextjs-app/__tests__/lib/utils.test.ts`, `__tests__/lib/compareLogic.test.ts`, create `__tests__/lib/ofstedStatus.test.ts` + +**Interfaces:** +- Produces: `OfstedCurrentGrade`, `OfstedLatestVisit` types; `ofstedDisplay(ofsted)` returning `{kind:'none'} | {kind:'graded'|'confirmed'; grade; gradeLabel; gradeDate} | {kind:'no_overall_grade'} | {kind:'report_card'; summary}`; `gradeSourceLine(current)`, `latestVisitLine(visit)`, `showLatestVisitLine(ofsted)` from `lib/ofstedStatus.ts`. + +- [ ] **Step 1: Write the failing tests** + +`__tests__/lib/ofstedStatus.test.ts`: + +```ts +import { gradeSourceLine, latestVisitLine, showLatestVisitLine } from '@/lib/ofstedStatus'; +import type { OfstedInspection } from '@/lib/types'; + +const base = { report_card: {} } as unknown as OfstedInspection; + +describe('gradeSourceLine', () => { + it('names the graded inspection and its date', () => { + expect(gradeSourceLine({ grade: 2, date: '2016-07-06', basis: 'graded' })) + .toBe('Graded inspection, 6 July 2016'); + }); + it('names the ungraded visit that confirmed it', () => { + expect(gradeSourceLine({ grade: 2, date: '2023-03-14', basis: 'confirmed' })) + .toBe('Confirmed at an ungraded inspection, 14 March 2023'); + }); +}); + +describe('latestVisitLine', () => { + it('prints the ungraded outcome', () => { + expect(latestVisitLine({ date: '2024-11-13', kind: 'ungraded', outcome: 'Standards maintained' })) + .toBe('Ungraded inspection, 13 November 2024: Standards maintained'); + }); + it('prints a graded visit without an outcome', () => { + expect(latestVisitLine({ date: '2025-06-17', kind: 'graded', outcome: null })) + .toBe('Graded inspection, 17 June 2025'); + }); +}); + +describe('showLatestVisitLine', () => { + it('shows a later visit than the one the grade came from', () => { + expect(showLatestVisitLine({ ...base, + current_grade: { grade: 2, date: '2020-03-03', basis: 'graded' }, + latest_visit: { date: '2025-05-21', kind: 'ungraded', outcome: 'Standards maintained' } })).toBe(true); + }); + it('hides it when the visit is the grade’s own source', () => { + expect(showLatestVisitLine({ ...base, + current_grade: { grade: 2, date: '2024-07-18', basis: 'confirmed' }, + latest_visit: { date: '2024-07-18', kind: 'ungraded', outcome: 'School remains Good' } })).toBe(false); + }); + it('shows an ungraded outcome when there is no grade', () => { + expect(showLatestVisitLine({ ...base, current_grade: null, + latest_visit: { date: '2024-11-13', kind: 'ungraded', outcome: 'Standards maintained' } })).toBe(true); + }); + it('hides it for a graded visit with no grade (the title already dates it)', () => { + expect(showLatestVisitLine({ ...base, current_grade: null, + latest_visit: { date: '2025-06-17', kind: 'graded', outcome: null } })).toBe(false); + }); +}); +``` + +In `__tests__/lib/utils.test.ts`, replace the `buildOfstedListBadge` grade cases so the year comes from `ofsted_grade_date`, and add the C1 and Review Focus cases: + +```ts + it('dates a grade by the inspection that awarded it, not the latest visit', () => { + const badge = buildOfstedListBadge({ ofsted_grade: 2, ofsted_grade_date: '2020-03-03', ofsted_date: '2025-05-21' }); + expect(badge.label).toBe('Good · 2020'); + }); + + it('shows Inspected for a latest inspection that gave no grade (Rabbsfarm)', () => { + const badge = buildOfstedListBadge({ ofsted_grade: null, ofsted_grade_date: null, ofsted_date: '2025-06-17' }); + expect(badge.label).toBe('Inspected · 2025'); + expect(badge.cssClass).toBe('ofstedInspected'); + }); + + it('prints a grade without a year when its date is missing', () => { + expect(buildOfstedListBadge({ ofsted_grade: 2, ofsted_grade_date: null, ofsted_date: '2025-01-01' }).label).toBe('Good'); + }); +``` + +Update the existing "returns grade word + year…" cases to pass `ofsted_grade_date` instead of `ofsted_date`; delete the existing "returns grade word without year when date is missing" if it duplicates the new one. + +In `__tests__/lib/compareLogic.test.ts` replace the `ofstedDisplay` block with: + +```ts +describe('ofstedDisplay', () => { + it('returns none without an inspection', () => { + expect(ofstedDisplay(null).kind).toBe('none'); + expect(ofstedDisplay(ofsted({})).kind).toBe('none'); + }); + it('distinguishes a graded grade from a confirmed one', () => { + expect(ofstedDisplay(ofsted({ current_grade: { grade: 1, date: '2019-10-09', basis: 'graded' }, + latest_visit: { date: '2025-02-05', kind: 'ungraded', outcome: 'Some aspects not as strong' } }))) + .toMatchObject({ kind: 'graded', gradeLabel: 'Outstanding', gradeDate: '2019-10-09' }); + expect(ofstedDisplay(ofsted({ current_grade: { grade: 2, date: '2024-07-18', basis: 'confirmed' }, + latest_visit: { date: '2024-07-18', kind: 'ungraded', outcome: 'School remains Good' } }))) + .toMatchObject({ kind: 'confirmed', gradeLabel: 'Good' }); + }); + it('has no overall grade when the latest inspection gave none', () => { + expect(ofstedDisplay(ofsted({ current_grade: null, overall_effectiveness: null, + latest_visit: { date: '2025-06-17', kind: 'graded', outcome: null } })).kind).toBe('no_overall_grade'); + }); +}); +``` + +(Keep the file's existing `ofsted()` helper and the report-card cases.) + +- [ ] **Step 2: Run them to see them fail** — `cd nextjs-app && npx jest __tests__/lib -t "ofsted|Ofsted"` → fails (module and fields missing). + +- [ ] **Step 3: Implement** + +`lib/types.ts`: in `School`, after `ofsted_grade`, add + +```ts + /** Date the grade was awarded or confirmed (null without a grade). */ + ofsted_grade_date?: string | null; +``` + +and change the `ofsted_date` doc to `/** The school's latest inspection of any kind. */`. In `OfstedInspection`, replace `grade_source` with: + +```ts + /** The overall grade still in force, dated by the inspection that awarded + * ("graded") or confirmed ("confirmed", an ungraded visit) it. Null when the + * latest inspection gave no overall grade, or for a report card. */ + current_grade?: OfstedCurrentGrade | null; + /** The school's most recent inspection of any kind. */ + latest_visit?: OfstedLatestVisit | null; +``` + +and add near `ReportCardEntry`: + +```ts +export interface OfstedCurrentGrade { + grade: 1 | 2 | 3 | 4; + date: string | null; + basis: 'graded' | 'confirmed'; +} + +export interface OfstedLatestVisit { + date: string; + kind: 'report_card' | 'graded' | 'ungraded'; + outcome: string | null; +} +``` + +Change the `overall_effectiveness` comment to "The graded inspection's own overall grade; never carried forward." + +`lib/ofstedStatus.ts`: + +```ts +/** + * The sentences the school page and the compare page print about where an + * Ofsted grade came from and what the latest visit was. One wording, two pages. + * Rule: docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md + */ + +import type { OfstedCurrentGrade, OfstedInspection, OfstedLatestVisit } from './types'; + +export function formatOfstedDate(iso: string | null | undefined): string { + if (!iso) return ''; + const d = new Date(iso); + if (Number.isNaN(d.getTime())) return ''; + return d.toLocaleDateString('en-GB', { day: 'numeric', month: 'long', year: 'numeric' }); +} + +const VISIT_KIND: Record = { + report_card: 'Report card inspection', + graded: 'Graded inspection', + ungraded: 'Ungraded inspection', +}; + +/** "Graded inspection, 6 July 2016" or "Confirmed at an ungraded inspection, 14 March 2023". */ +export function gradeSourceLine(current: OfstedCurrentGrade): string { + const when = formatOfstedDate(current.date); + const what = current.basis === 'confirmed' ? 'Confirmed at an ungraded inspection' : 'Graded inspection'; + return when ? `${what}, ${when}` : what; +} + +/** "Ungraded inspection, 13 November 2024: Standards maintained". */ +export function latestVisitLine(visit: OfstedLatestVisit): string { + const head = `${VISIT_KIND[visit.kind]}, ${formatOfstedDate(visit.date)}`; + return visit.outcome ? `${head}: ${visit.outcome}` : head; +} + +/** + * Whether the latest visit needs its own line: when it is not where the grade + * came from, or when there is no grade but an ungraded outcome to report. A + * graded visit without a grade is already dated by the section title. + */ +export function showLatestVisitLine(ofsted: Pick): boolean { + const visit = ofsted.latest_visit; + if (!visit || visit.kind === 'report_card') return false; + const current = ofsted.current_grade; + if (current) return current.date !== visit.date; + return visit.kind === 'ungraded'; +} +``` + +`lib/utils.ts` — `buildOfstedListBadge`: add `ofsted_grade_date?: string | null;` to the parameter type, update the doc comment's second bullet to "Current grade (ofsted_grade set): grade word + the year it was awarded or confirmed", and replace the body after the report-card branch with: + +```ts + const yearOf = (iso?: string | null) => (iso ? new Date(iso).getFullYear() : null); + + if (school.ofsted_grade) { + const labels: Record = { + 1: 'Outstanding', + 2: 'Good', + 3: 'Req. Improvement', + 4: 'Inadequate', + }; + const gradeYear = yearOf(school.ofsted_grade_date); + return { + label: `${labels[school.ofsted_grade]}${gradeYear ? ` · ${gradeYear}` : ''}`, + cssClass: `ofsted${school.ofsted_grade}`, + }; + } + + // An inspection is on record but no overall grade is in force: every + // inspection from Sept 2024 to Nov 2025, or an ungraded visit whose outcome + // names no grade. Dated by the latest visit. + if (school.ofsted_date != null || school.ofsted_framework != null) { + const visitYear = yearOf(school.ofsted_date); + return { label: `Inspected${visitYear ? ` · ${visitYear}` : ''}`, cssClass: 'ofstedInspected' }; + } +``` + +`lib/compareLogic.ts` — replace `OfstedDisplay` and `ofstedDisplay` with: + +```ts +export type OfstedDisplay = + | { kind: 'none' } + | { kind: 'graded' | 'confirmed'; grade: number; gradeLabel: string; gradeDate: string | null } + | { kind: 'no_overall_grade' } + | { kind: 'report_card'; summary: ReportCardSummary }; + +export function ofstedDisplay( + ofsted: OfstedInspection | null | undefined, +): OfstedDisplay { + if (!ofsted) return { kind: 'none' }; + + // A report card is the newest inspection format; when present it wins. + if (ofsted.report_card && Object.keys(ofsted.report_card).length > 0) { + return { kind: 'report_card', summary: summariseReportCard(ofsted) }; + } + + const current = ofsted.current_grade; + const gradeLabel = current ? OFSTED_LEGACY_GRADES[current.grade] : undefined; + if (current && gradeLabel !== undefined) { + return { + kind: current.basis === 'confirmed' ? 'confirmed' : 'graded', + grade: current.grade, + gradeLabel, + gradeDate: current.date, + }; + } + return ofsted.latest_visit ? { kind: 'no_overall_grade' } : { kind: 'none' }; +} +``` + +- [ ] **Step 4: Run** `npx jest __tests__/lib` and `npx tsc --noEmit`. Expected: the new tests pass; `tsc` reports errors only in the components Tasks 9–10 change (`carried_forward`, `transitional`, `carriedForward`) — fix those in Tasks 9 and 10, not by loosening types. + +- [ ] **Step 5: Commit** — `fix(ofsted): badge and compare read the grade still in force` + +### Task 9: School page Ofsted section + +**Files:** +- Modify: `nextjs-app/components/school/OfstedSection.tsx` +- Modify: `nextjs-app/components/school/PrimarySchoolSections.tsx:57-80`, `SecondarySchoolSections.tsx:63-85` +- Test: create `nextjs-app/__tests__/components/OfstedSection.test.tsx`; update `__tests__/support/schoolFixtures.ts` (`grade_source` → `current_grade` + `latest_visit`) and `__tests__/components/schoolDetail.characterization.test.tsx` if its Ofsted assertions change. + +**Interfaces:** Consumes `ofstedDisplay`, `gradeSourceLine`, `latestVisitLine`, `showLatestVisitLine`, `formatOfstedDate`, `ofstedLegacyAreas`. `OfstedSection` props become `{ ofsted, urn, isReportCard, ofstedInspectedDate, oeifAllSameGrade, oeifAreas }` (the `variant` prop is removed; its callers stop passing it). + +- [ ] **Step 1: Write the failing tests** + +`__tests__/components/OfstedSection.test.tsx`: + +```tsx +import { render, screen } from '@testing-library/react'; +import { OfstedSection } from '@/components/school/OfstedSection'; +import { ofstedLegacyAreas } from '@/lib/utils'; +import type { OfstedInspection } from '@/lib/types'; + +const empty = { + framework: null, inspection_date: null, inspection_type: null, overall_effectiveness: null, + quality_of_education: null, behaviour_attitudes: null, personal_development: null, + leadership_management: null, early_years_provision: null, sixth_form_provision: null, + previous_overall: null, rc_safeguarding_met: null, rc_inclusion: null, rc_curriculum_teaching: null, + rc_achievement: null, rc_attendance_behaviour: null, rc_personal_development: null, + rc_leadership_governance: null, rc_early_years: null, rc_sixth_form: null, report_url: null, + report_card: {}, +} as OfstedInspection; + +function renderSection(o: Partial) { + const ofsted = { ...empty, ...o } as OfstedInspection; + render( + , + ); +} + +it('shows no overall grade, not an older one, after a no-grade inspection (Rabbsfarm)', () => { + renderSection({ + inspection_date: '2025-06-17', quality_of_education: 3, behaviour_attitudes: 3, + personal_development: 2, leadership_management: 3, early_years_provision: 2, + current_grade: null, latest_visit: { date: '2025-06-17', kind: 'graded', outcome: null }, + }); + expect(screen.getByText(/Inspected 17 June 2025/)).toBeInTheDocument(); + expect(screen.getByText('No overall grade')).toBeInTheDocument(); + expect(screen.queryByText('Good', { selector: 'span' })).not.toBeInTheDocument(); + expect(screen.queryByText('Not rated')).not.toBeInTheDocument(); + expect(screen.getAllByText('Requires Improvement')).toHaveLength(3); +}); + +it('keeps the sixth-form judgement when there is no overall grade (audit M2)', () => { + renderSection({ + inspection_date: '2025-04-01', quality_of_education: 1, behaviour_attitudes: 1, + personal_development: 1, leadership_management: 1, sixth_form_provision: 2, + current_grade: null, latest_visit: { date: '2025-04-01', kind: 'graded', outcome: null }, + }); + expect(screen.getByText(/sixth form/i)).toBeInTheDocument(); + expect(screen.getByText('Good')).toBeInTheDocument(); +}); + +it('dates a grade by its source and prints a later visit separately (Washwood Heath)', () => { + renderSection({ + inspection_date: '2020-03-03', overall_effectiveness: 2, quality_of_education: 2, + current_grade: { grade: 2, date: '2020-03-03', basis: 'graded' }, + latest_visit: { date: '2025-05-21', kind: 'ungraded', outcome: 'Standards maintained' }, + }); + expect(screen.getByText(/Inspected 21 May 2025/)).toBeInTheDocument(); + expect(screen.getByText('Graded inspection, 3 March 2020')).toBeInTheDocument(); + expect(screen.getByText(/Latest visit: Ungraded inspection, 21 May 2025: Standards maintained/)).toBeInTheDocument(); +}); + +it('says a grade was confirmed at an ungraded visit (Robins Lane)', () => { + renderSection({ + inspection_date: '2020-01-07', overall_effectiveness: 2, + current_grade: { grade: 2, date: '2024-07-18', basis: 'confirmed' }, + latest_visit: { date: '2024-07-18', kind: 'ungraded', outcome: 'School remains Good' }, + }); + expect(screen.getByText('Confirmed at an ungraded inspection, 18 July 2024')).toBeInTheDocument(); + expect(screen.queryByText(/Latest visit/)).not.toBeInTheDocument(); +}); + +it('prints an ungraded outcome when there is no grade (Oakgrove)', () => { + renderSection({ + current_grade: null, + latest_visit: { date: '2024-11-13', kind: 'ungraded', outcome: 'Standards maintained' }, + }); + expect(screen.getByText('No overall grade')).toBeInTheDocument(); + expect(screen.getByText(/Ungraded inspection, 13 November 2024: Standards maintained/)).toBeInTheDocument(); +}); +``` + +- [ ] **Step 2: Run** `npx jest __tests__/components/OfstedSection.test.tsx` → fails. + +- [ ] **Step 3: Implement** + +Replace the non-report-card part of `OfstedSection` (everything after the report-card branch of the ternary, both the "Old OEIF layout" and the "Secondary only" branches) with one branch, and drop the `variant` prop: + +```tsx + ) : ( + <> + {display.kind === 'graded' || display.kind === 'confirmed' ? ( +
+ + {display.gradeLabel} + +
+ ) : ( +
+ No overall grade +
+ )} +

+ {ofsted.current_grade + ? gradeSourceLine(ofsted.current_grade) + : 'Ofsted stopped giving overall grades in September 2024.'} +

+ {showLatestVisitLine(ofsted) && ofsted.latest_visit && ( +

Latest visit: {latestVisitLine(ofsted.latest_visit)}

+ )} + {oeifAllSameGrade && display.kind !== 'no_overall_grade' && display.kind !== 'none' ? ( +

+ Rated {display.gradeLabel} across all inspected areas: Quality of Teaching, Behaviour, Pupils' Development and Leadership. +

+ ) : oeifAreas.length > 0 ? ( + <> + {ofsted.inspection_date && ofsted.inspection_date !== ofsted.latest_visit?.date && ( +

+ Area judgements from the graded inspection, {formatOfstedDate(ofsted.inspection_date)}. +

+ )} +
+ {oeifAreas.map(({ label, value }) => ( +
+
{label}
+
+ {OFSTED_LABELS[value]} +
+
+ ))} +
+ + ) : null} + + )} +``` + +with `const display = ofstedDisplay(ofsted);` at the top of the component and imports from `@/lib/compareLogic` and `@/lib/ofstedStatus`. Update the header comment: the primary/secondary fork is gone because both now follow the current-status rule. Keep the report-card branch and the title/link untouched. + +In both section composers: + +```tsx + const ofstedInspectedDate = isReportCard + ? ofsted?.rc_inspection_date ?? null + : ofsted?.latest_visit?.date ?? null; + ... + const oeifAllSameGrade = + !!ofsted && + !isReportCard && + ofsted.overall_effectiveness != null && + oeifAreas.length >= 3 && + oeifAreas.every((a) => a.value === ofsted.overall_effectiveness); +``` + +and remove `variant="primary"`/`variant="secondary"` from the `` calls. + +Fixtures: in `schoolFixtures.ts` replace `grade_source: 'graded'` with `current_grade: { grade: , date: , basis: 'graded' }, latest_visit: { date: , kind: 'graded', outcome: null }`. + +- [ ] **Step 4: Run** `npx jest` (whole suite) and `npx tsc --noEmit`. Fix characterization assertions only where the spec changes the copy (e.g. "Not rated" → "No overall grade"); any other change is a regression. + +- [ ] **Step 5: Commit** — `fix(school): the Ofsted section shows the grade still in force and the latest visit` + +### Task 10: Compare Ofsted rows + +**Files:** +- Modify: `nextjs-app/components/compare/CompareAtAGlance.tsx:77-105`, `components/compare/CompareOfsted.tsx:42-77`, `:186-206` +- Test: `nextjs-app/__tests__/components/CompareOfsted.test.tsx` + +- [ ] **Step 1: Write the failing tests** (append to `CompareOfsted.test.tsx`, using its existing render helper; if it has none, render `` with a minimal `School` fixture from `__tests__/support/schoolFixtures.ts`): + +```tsx +it('dates "Inspected" by the latest visit, not the graded inspection (Washwood Heath)', () => { + renderCompare({ + inspection_date: '2020-03-03', overall_effectiveness: 2, + current_grade: { grade: 2, date: '2020-03-03', basis: 'graded' }, + latest_visit: { date: '2025-05-21', kind: 'ungraded', outcome: 'Standards maintained' }, + }); + expect(screen.getByText(/21 May 2025/)).toBeInTheDocument(); + expect(screen.queryByText('4+ years ago')).not.toBeInTheDocument(); + expect(screen.getByText('Graded inspection, 3 March 2020')).toBeInTheDocument(); +}); + +it('shows no overall grade after a no-grade inspection', () => { + renderCompare({ + inspection_date: '2025-06-17', quality_of_education: 3, + current_grade: null, latest_visit: { date: '2025-06-17', kind: 'graded', outcome: null }, + }); + expect(screen.getAllByText('No overall grade').length).toBeGreaterThan(0); + expect(screen.queryByText('Good')).not.toBeInTheDocument(); +}); +``` + +Remove tests that assert "Grade carried forward" or "transitional framework". + +- [ ] **Step 2: Run** `npx jest __tests__/components/CompareOfsted.test.tsx` → fails. + +- [ ] **Step 3: Implement** + +`CompareOfsted.tsx` `ResultCell` — replace the `transitional` branch and the final branch with: + +```tsx + if (display.kind === 'no_overall_grade') { + return ( + <> + + No overall grade + + Ofsted stopped giving overall grades in September 2024 + + ); + } + return ( + <> + + {display.gradeLabel} + + {current && {gradeSourceLine(current)}} + + ); +``` + +passing `current={ofsted?.current_grade ?? null}` into `ResultCell` (add the prop: `{ display, current }: { display: OfstedDisplay; current: OfstedCurrentGrade | null }`). + +"Inspected" measure: `const dateIso = ofsted?.latest_visit?.date ?? ofsted?.rc_inspection_date ?? null;` (delete the report-card special case and its comment, since `latest_visit` is the report card's own date when there is one) and render the visit kind under the date when present: + +```tsx + {formatInspectionDate(dateIso)}{' '} + {age != null && age > 4 && 4+ years ago} + {ofsted?.latest_visit?.outcome && {ofsted.latest_visit.outcome}} +``` + +Update the file's header comment ("three regimes (legacy graded, interim carried-forward, …)") to "legacy graded, no overall grade, renewed-framework report card". + +`CompareAtAGlance.tsx` — replace the `graded || carried_forward` and `transitional` branches with: + +```tsx + {(display.kind === 'graded' || display.kind === 'confirmed') && ( + <> + + {display.gradeLabel} + + {display.kind === 'confirmed' && Confirmed at an ungraded inspection} + + )} + {display.kind === 'no_overall_grade' && ( + + No overall grade + + )} +``` + +- [ ] **Step 4: Run** `npx jest` and `npx tsc --noEmit` → all pass. + +- [ ] **Step 5: Commit** — `fix(compare): Ofsted rows show the grade still in force and the latest visit` + +### Task 11: Remove the unused helpers that encode the old rule + +**Files:** Modify `nextjs-app/lib/utils.ts` (delete `buildOfstedHeroChip`, `OfstedHeroChip`/`HeroTone` types if only it uses them, `buildSchoolSummary`, and `formatOfstedMonth` if nothing else uses it); delete their tests in `__tests__/lib/utils.test.ts`. + +- [ ] **Step 1:** `grep -rn "buildOfstedHeroChip\|buildSchoolSummary\|formatOfstedMonth\|OfstedHeroChip\|HeroTone" nextjs-app/components nextjs-app/lib nextjs-app/app nextjs-app/__tests__` — confirm only `lib/utils.ts` and its test reference them. +- [ ] **Step 2:** Delete them and their tests. +- [ ] **Step 3:** `npx tsc --noEmit && npx jest` → pass. +- [ ] **Step 4: Commit** — `refactor(utils): drop the unused Ofsted hero chip and summary sentence` + +### Task 12: E2E journeys + +**Files:** Modify `e2e/tests/journeys.spec.ts` (add after the report-card journey). + +- [ ] **Step 1: Write the journeys** + +```ts +test('a school whose latest inspection gave no grade is not shown with an older grade', async ({ page }) => { + // Rabbsfarm (102408): Ofsted's 17 June 2025 inspection gave no overall + // grade and rated three areas Requires Improvement. The site used to carry + // a 2020 "remains Good" forward and print "Good · 2025". + const URN = 102408; + const res = await page.request.get(`/api/schools/${URN}`); + expect(res.ok()).toBeTruthy(); + const ofsted = (await res.json()).ofsted; + expect(ofsted.current_grade).toBeNull(); + expect(ofsted.latest_visit.date).toBe('2025-06-17'); + + await searchByName(page, 'Rabbsfarm Primary'); + const row = page.locator(`a[href*="${URN}"]`).first(); + await expect(row).toBeVisible({ timeout: 15_000 }); + await expect(page.getByText('Inspected · 2025').first()).toBeVisible(); + await expect(page.getByText(/Good · 2025/)).toHaveCount(0); + + await page.goto(`/school/${URN}`); + const section = page.locator('#ofsted'); + await expect(section.getByText('No overall grade')).toBeVisible(); + await expect(section.getByText('Requires Improvement').first()).toBeVisible(); +}); + +test('a grade confirmed at an ungraded visit says so and is dated by it', async ({ page }) => { + const URN = 104762; // Robins Lane: graded Good Jan 2020, "School remains Good" 18 July 2024 + await page.goto(`/school/${URN}`); + const section = page.locator('#ofsted'); + await expect(section).toBeVisible({ timeout: 15_000 }); + await expect(section.getByText('Confirmed at an ungraded inspection, 18 July 2024')).toBeVisible(); +}); +``` + +- [ ] **Step 2:** These run against staging after merge (the gate is post-merge). Before opening the PR, run `npx playwright test --list` from the repo root to confirm they parse. If either school is missing on staging (`/api/schools/{urn}` 404), replace it with another URN from the same rule row using the audit's MI file. +- [ ] **Step 3: Commit** — `test(e2e): no grade is carried past a no-grade inspection` + +### Task 13: Open PR 2 + +- [ ] **Step 1:** Full checks: `cd nextjs-app && npx tsc --noEmit && npm test`; backend/pipeline `pytest` as in Task 5; `dbt compile --select dim_school`. +- [ ] **Step 2:** `git status --short`; push; open the PR. The body leads with **"Do not merge until `school_data_monthly_ofsted` has run on staging with PR 1 — the backend reads `marts.fact_ofsted_latest` and has no fallback"**, then lists the visible change (about 932 badges, place-page counts, Typesense rating), the checks, and the post-merge verification: daily DAG run (for `dim_school` and Typesense), E2E, and re-running the audit's C1/M1/M2 checks (`scratchpad/audit/check_render.py` against staging) expecting 0. From 358705bf3934e555b609a2a29adc0a2f8a8e4249 Mon Sep 17 00:00:00 2001 From: Tudor Date: Mon, 5 Oct 2026 16:22:49 +0100 Subject: [PATCH 3/5] feat(pipeline): keep Ofsted's graded, ungraded and report-card dates apart int_ofsted_latest needs to know which inspection came last. Report-card-only schools are no longer dropped (123 schools, audit H3); inspection_date stays for current readers and now falls back to the report-card date, so fact_ofsted_inspection's not_null test still holds. Co-Authored-By: Claude Opus 5.5 --- .../models/staging/stg_ofsted_inspections.sql | 67 ++++++++++--------- .../models/staging/stg_ofsted_inspections.yml | 25 +++++++ 2 files changed, 61 insertions(+), 31 deletions(-) create mode 100644 pipeline/transform/models/staging/stg_ofsted_inspections.yml diff --git a/pipeline/transform/models/staging/stg_ofsted_inspections.sql b/pipeline/transform/models/staging/stg_ofsted_inspections.sql index 6bb7574..efdf118 100644 --- a/pipeline/transform/models/staging/stg_ofsted_inspections.sql +++ b/pipeline/transform/models/staging/stg_ofsted_inspections.sql @@ -1,5 +1,10 @@ -- Staging model: Ofsted inspection records --- Handles both OEIF (pre-Nov 2025) and Report Card (post-Nov 2025) frameworks +-- Handles both OEIF (pre-Nov 2025) and Report Card (post-Nov 2025) frameworks. +-- +-- Ofsted's MI carries up to three inspections per school: the latest graded +-- one, the latest ungraded one and the latest report card. Their dates stay +-- separate here so int_ofsted_latest can tell which came last and which one a +-- grade belongs to (docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md). with source as ( select * from {{ source('raw', 'ofsted_inspections') }} @@ -8,17 +13,12 @@ with source as ( renamed as ( select cast(urn as integer) as urn, - -- Inspection event date: the graded inspection when present, otherwise the - -- ungraded (Section 8) inspection so schools with only an ungraded - -- inspection are still retained. - coalesce( - to_date(nullif(trim(inspection_date), 'NULL'), 'DD/MM/YYYY'), - to_date(nullif(trim(ungraded_inspection_date), 'NULL'), 'DD/MM/YYYY') - ) as inspection_date, + to_date(nullif(trim(inspection_date), 'NULL'), 'DD/MM/YYYY') as graded_inspection_date, + to_date(nullif(trim(ungraded_inspection_date), 'NULL'), 'DD/MM/YYYY') as ungraded_inspection_date, inspection_type, event_type_grouping as framework, - -- OEIF grades (1-4 scale) + -- OEIF grades (1-4 scale; 9 = not applicable) {{ safe_numeric('overall_effectiveness') }}::integer as overall_effectiveness, {{ safe_numeric('quality_of_education') }}::integer as quality_of_education, {{ safe_numeric('behaviour_and_attitudes') }}::integer as behaviour_attitudes, @@ -27,9 +27,9 @@ renamed as ( {{ safe_numeric('early_years_provision') }}::integer as early_years_provision, {{ safe_numeric('sixth_form_provision') }}::integer as sixth_form_provision, - -- Ungraded (Section 8) inspection outcome — free text, plus a grade - -- parsed from it (1/2/null) used as a last-resort fallback for schools - -- with no graded overall effectiveness. + -- Ungraded (Section 8) inspection outcome — free text, plus the grade + -- it confirms ("School remains Good" → 2); null for outcomes that + -- name no grade. nullif(trim(ungraded_outcome), 'NULL') as ungraded_outcome, {{ parse_ungraded_outcome('ungraded_outcome') }}::integer as ungraded_grade, @@ -50,31 +50,36 @@ renamed as ( {{ parse_report_card_grade('rc_sixth_form') }}::integer as rc_sixth_form, -- Start date of the latest FULL inspection (the report-card - -- inspection in the renewed framework). Guarded in the final select: - -- only kept when the row actually carries report-card grades, because - -- in legacy-format files this column is the legacy inspection date. + -- inspection in the renewed framework). Only kept when the row + -- carries report-card grades, because in legacy-format files this + -- column is the legacy inspection date. to_date(nullif(trim(rc_inspection_date), 'NULL'), 'DD/MM/YYYY') as rc_inspection_date_raw, nullif(trim(report_url), 'NULL') as report_url from source where urn is not null - and ( - nullif(trim(inspection_date), 'NULL') is not null - or nullif(trim(ungraded_inspection_date), 'NULL') is not null - ) +), + +dated as ( + select + *, + case + when rc_safeguarding_met is not null + or rc_inclusion is not null + or rc_curriculum_teaching is not null + or rc_achievement is not null + or rc_attendance_behaviour is not null + or rc_personal_development is not null + or rc_leadership_governance is not null + then rc_inspection_date_raw + end as rc_inspection_date + from renamed ) select *, - case - when rc_safeguarding_met is not null - or rc_inclusion is not null - or rc_curriculum_teaching is not null - or rc_achievement is not null - or rc_attendance_behaviour is not null - or rc_personal_development is not null - or rc_leadership_governance is not null - then rc_inspection_date_raw - end as rc_inspection_date -from renamed -where inspection_date is not null + -- For readers that predate the separate dates (fact_ofsted_inspection and + -- the backend until it reads fact_ofsted_latest). Never null below. + coalesce(graded_inspection_date, ungraded_inspection_date, rc_inspection_date) as inspection_date +from dated +where coalesce(graded_inspection_date, ungraded_inspection_date, rc_inspection_date) is not null diff --git a/pipeline/transform/models/staging/stg_ofsted_inspections.yml b/pipeline/transform/models/staging/stg_ofsted_inspections.yml new file mode 100644 index 0000000..4b993d2 --- /dev/null +++ b/pipeline/transform/models/staging/stg_ofsted_inspections.yml @@ -0,0 +1,25 @@ +version: 2 + +unit_tests: + - name: stg_ofsted_keeps_the_three_dates_apart + description: The graded and ungraded dates must stay separate so the latest visit can be found. + model: stg_ofsted_inspections + given: + - input: source('raw', 'ofsted_inspections') + rows: + - {urn: 102408, inspection_date: '17/06/2025', ungraded_inspection_date: '06/02/2020', overall_effectiveness: 'Not judged', ungraded_outcome: 'School remains Good', rc_inspection_date: '17/06/2025', rc_inclusion: 'NULL'} + expect: + rows: + - {urn: 102408, graded_inspection_date: '2025-06-17', ungraded_inspection_date: '2020-02-06', rc_inspection_date: null, inspection_date: '2025-06-17', ungraded_grade: 2} + + - name: stg_ofsted_keeps_report_card_only_rows + description: A school whose only inspection is a report card used to be dropped (audit H3). + model: stg_ofsted_inspections + given: + - input: source('raw', 'ofsted_inspections') + rows: + - {urn: 149612, inspection_date: 'NULL', ungraded_inspection_date: 'NULL', rc_inspection_date: '10/02/2026', rc_inclusion: 'Expected standard', rc_safeguarding_met: 'Met'} + - {urn: 1, inspection_date: 'NULL', ungraded_inspection_date: 'NULL', rc_inspection_date: 'NULL'} + expect: + rows: + - {urn: 149612, graded_inspection_date: null, ungraded_inspection_date: null, rc_inspection_date: '2026-02-10', inspection_date: '2026-02-10', rc_inclusion: 3} From e6babe21f5f56cb816a9bcaf3ab5f03aed06d6d7 Mon Sep 17 00:00:00 2001 From: Tudor Date: Mon, 5 Oct 2026 16:23:38 +0100 Subject: [PATCH 4/5] feat(pipeline): work out each school's current Ofsted status once int_ofsted_latest now picks each school's latest visit (report card, graded or ungraded inspection) and the overall grade still in force, dated by the inspection that awarded or confirmed it. A 'School remains Good' from an older ungraded visit is no longer carried past a newer inspection that gave no grade (audit C1). Duplicate monthly rows resolve to the newest visit, so a newer report card always wins. dim_school's columns are unchanged. Co-Authored-By: Claude Opus 5.5 --- .../models/intermediate/int_ofsted_latest.sql | 83 ++++++++++++++-- .../models/intermediate/int_ofsted_latest.yml | 99 +++++++++++++++++++ 2 files changed, 175 insertions(+), 7 deletions(-) create mode 100644 pipeline/transform/models/intermediate/int_ofsted_latest.yml diff --git a/pipeline/transform/models/intermediate/int_ofsted_latest.sql b/pipeline/transform/models/intermediate/int_ofsted_latest.sql index 2b09768..ac3727b 100644 --- a/pipeline/transform/models/intermediate/int_ofsted_latest.sql +++ b/pipeline/transform/models/intermediate/int_ofsted_latest.sql @@ -1,18 +1,88 @@ --- Intermediate model: Latest Ofsted inspection per URN --- Picks the most recent inspection for each school +-- Intermediate model: the current Ofsted status per URN +-- One row per school: its latest visit (report card, graded or ungraded +-- inspection) and the overall grade still in force, if any. A grade is dated +-- by the inspection that awarded or confirmed it, never by a later visit. +-- Rule and examples: docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md -with ranked as ( +with inspections as ( select *, + -- Report cards began in Nov 2025, after the last legacy inspections, + -- so a school's report card is always its latest visit. + coalesce( + rc_inspection_date, + greatest(graded_inspection_date, ungraded_inspection_date) + ) as latest_visit_date + from {{ ref('stg_ofsted_inspections') }} +), + +ranked as ( + select + *, + -- Monthly loads can leave several rows per school. The newest visit + -- wins; the tie-breaks keep the choice deterministic. row_number() over ( partition by urn - order by inspection_date desc + order by latest_visit_date desc, + rc_inspection_date desc nulls last, + graded_inspection_date desc nulls last, + ungraded_inspection_date desc nulls last ) as rn - from {{ ref('stg_ofsted_inspections') }} + from inspections +), + +latest as ( + select + *, + case + when rc_inspection_date is not null then 'report_card' + when graded_inspection_date = latest_visit_date then 'graded' + else 'ungraded' + end as latest_visit_kind + from ranked + where rn = 1 +), + +graded as ( + select + *, + -- The grade still in force. A graded inspection's own overall grade + -- (1-4; "Not judged" and the sentinel 9 are no grade); else an + -- ungraded visit's "School remains X"; else, after an ungraded visit + -- that names no grade, the graded inspection's grade. + case + when latest_visit_kind = 'graded' and overall_effectiveness between 1 and 4 + then 'graded_latest' + when latest_visit_kind = 'ungraded' and ungraded_grade is not null + then 'confirmed' + when latest_visit_kind = 'ungraded' and overall_effectiveness between 1 and 4 + then 'graded_earlier' + end as grade_case + from latest ) select urn, + latest_visit_date, + latest_visit_kind, + case when latest_visit_kind = 'ungraded' then ungraded_outcome end as latest_visit_outcome, + case grade_case + when 'confirmed' then ungraded_grade + when 'graded_latest' then overall_effectiveness + when 'graded_earlier' then overall_effectiveness + end as current_grade, + case grade_case + when 'confirmed' then ungraded_inspection_date + when 'graded_latest' then graded_inspection_date + when 'graded_earlier' then graded_inspection_date + end as current_grade_date, + case grade_case + when 'confirmed' then 'confirmed' + when 'graded_latest' then 'graded' + when 'graded_earlier' then 'graded' + end as current_grade_basis, + graded_inspection_date, + ungraded_inspection_date, inspection_date, inspection_type, framework, @@ -36,5 +106,4 @@ select rc_sixth_form, rc_inspection_date, report_url -from ranked -where rn = 1 +from graded diff --git a/pipeline/transform/models/intermediate/int_ofsted_latest.yml b/pipeline/transform/models/intermediate/int_ofsted_latest.yml new file mode 100644 index 0000000..c436b2e --- /dev/null +++ b/pipeline/transform/models/intermediate/int_ofsted_latest.yml @@ -0,0 +1,99 @@ +version: 2 + +unit_tests: + - name: graded_not_judged_has_no_grade + description: Rabbsfarm (102408). The 2025 inspection gave no overall grade, so the 2020 "remains Good" is not carried forward (audit C1). + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 102408, graded_inspection_date: '2025-06-17', ungraded_inspection_date: '2020-02-06', overall_effectiveness: null, ungraded_grade: 2, ungraded_outcome: 'School remains Good'} + expect: + rows: + - {urn: 102408, latest_visit_date: '2025-06-17', latest_visit_kind: graded, latest_visit_outcome: null, current_grade: null, current_grade_date: null, current_grade_basis: null} + + - name: graded_with_overall_grade + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 1, graded_inspection_date: '2019-06-01', overall_effectiveness: 2} + expect: + rows: + - {urn: 1, latest_visit_date: '2019-06-01', latest_visit_kind: graded, current_grade: 2, current_grade_date: '2019-06-01', current_grade_basis: graded} + + - name: ungraded_remains_good_confirms_the_grade + description: Robins Lane (104762). Graded Good 2020, "School remains Good" July 2024 — Good, dated by the confirming visit. + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 104762, graded_inspection_date: '2020-01-07', ungraded_inspection_date: '2024-07-18', overall_effectiveness: 2, ungraded_grade: 2, ungraded_outcome: 'School remains Good'} + expect: + rows: + - {urn: 104762, latest_visit_date: '2024-07-18', latest_visit_kind: ungraded, latest_visit_outcome: 'School remains Good', current_grade: 2, current_grade_date: '2024-07-18', current_grade_basis: confirmed} + + - name: post_2024_ungraded_keeps_graded_grade_with_its_own_date + description: Washwood Heath (139888). Graded Good 2020, "Standards maintained" May 2025 — Good, dated 2020; latest visit May 2025 (audit M1). + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 139888, graded_inspection_date: '2020-03-03', ungraded_inspection_date: '2025-05-21', overall_effectiveness: 2, ungraded_grade: null, ungraded_outcome: 'Standards maintained'} + expect: + rows: + - {urn: 139888, latest_visit_date: '2025-05-21', latest_visit_kind: ungraded, latest_visit_outcome: 'Standards maintained', current_grade: 2, current_grade_date: '2020-03-03', current_grade_basis: graded} + + - name: ungraded_only_standards_maintained + description: Oakgrove (136454). Only an ungraded visit, outcome names no grade. + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 136454, ungraded_inspection_date: '2024-11-13', ungraded_grade: null, ungraded_outcome: 'Standards maintained'} + expect: + rows: + - {urn: 136454, latest_visit_date: '2024-11-13', latest_visit_kind: ungraded, latest_visit_outcome: 'Standards maintained', current_grade: null, current_grade_date: null, current_grade_basis: null} + + - name: report_card_wins + description: The Willink School (110048). A report card is the latest visit and no legacy grade stays in force. + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 110048, ungraded_inspection_date: '2023-10-05', ungraded_grade: 2, ungraded_outcome: 'School remains Good', rc_inspection_date: '2026-05-06', rc_inclusion: 3} + expect: + rows: + - {urn: 110048, latest_visit_date: '2026-05-06', latest_visit_kind: report_card, latest_visit_outcome: null, current_grade: null, current_grade_date: null, current_grade_basis: null} + + - name: duplicate_rows_newer_report_card_wins + description: Monthly loads can leave an older row beside a newer one for the same graded date; the row with the report card must win (audit H3). + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 138186, graded_inspection_date: '2023-06-13', overall_effectiveness: 3} + - {urn: 138186, graded_inspection_date: '2023-06-13', overall_effectiveness: 3, rc_inspection_date: '2026-06-02', rc_inclusion: 3} + expect: + rows: + - {urn: 138186, latest_visit_date: '2026-06-02', latest_visit_kind: report_card, current_grade: null} + + - name: same_day_graded_wins + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 2, graded_inspection_date: '2024-03-01', ungraded_inspection_date: '2024-03-01', overall_effectiveness: 1, ungraded_grade: 2, ungraded_outcome: 'School remains Good'} + expect: + rows: + - {urn: 2, latest_visit_kind: graded, current_grade: 1, current_grade_basis: graded} + + - name: overall_sentinel_is_not_a_grade + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 3, graded_inspection_date: '2018-05-01', overall_effectiveness: 9} + expect: + rows: + - {urn: 3, latest_visit_kind: graded, current_grade: null} From dfa4928641aef2462e320e99100e03b429bb0e3d Mon Sep 17 00:00:00 2001 From: Tudor Date: Mon, 5 Oct 2026 16:24:41 +0100 Subject: [PATCH 5/5] feat(pipeline): add fact_ofsted_latest, one current Ofsted status per school The backend will read this instead of picking the latest row of fact_ofsted_inspection itself (twice, with arbitrary ties). Schema tests and assert_ofsted_current_grade_consistent pin its invariants. Built by the monthly Ofsted DAG (int_ofsted_latest+), not the daily one. Co-Authored-By: Claude Opus 5.5 --- .../transform/models/marts/_marts_schema.yml | 24 ++++++++++++ .../models/marts/fact_ofsted_latest.sql | 37 +++++++++++++++++++ ...assert_ofsted_current_grade_consistent.sql | 10 +++++ 3 files changed, 71 insertions(+) create mode 100644 pipeline/transform/models/marts/fact_ofsted_latest.sql create mode 100644 pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql diff --git a/pipeline/transform/models/marts/_marts_schema.yml b/pipeline/transform/models/marts/_marts_schema.yml index 89f2301..b198b6b 100644 --- a/pipeline/transform/models/marts/_marts_schema.yml +++ b/pipeline/transform/models/marts/_marts_schema.yml @@ -125,6 +125,30 @@ models: - name: inspection_date tests: [not_null] + - name: fact_ofsted_latest + description: > + Current Ofsted status, one row per URN: the latest visit and the overall + grade still in force. See int_ofsted_latest for the rule. + columns: + - name: urn + tests: [not_null, unique] + - name: latest_visit_date + tests: [not_null] + - name: latest_visit_kind + tests: + - not_null + - accepted_values: + values: ['report_card', 'graded', 'ungraded'] + - name: current_grade + tests: + - accepted_values: + values: [1, 2, 3, 4] + quote: false + - name: current_grade_basis + tests: + - accepted_values: + values: ['graded', 'confirmed'] + - name: fact_pupil_characteristics description: Pupil demographics — one row per URN per year columns: diff --git a/pipeline/transform/models/marts/fact_ofsted_latest.sql b/pipeline/transform/models/marts/fact_ofsted_latest.sql new file mode 100644 index 0000000..12c5cb0 --- /dev/null +++ b/pipeline/transform/models/marts/fact_ofsted_latest.sql @@ -0,0 +1,37 @@ +-- Mart: current Ofsted status — one row per URN +-- The backend reads this instead of choosing the latest row of +-- fact_ofsted_inspection itself. The rule lives in int_ofsted_latest. + +select + urn, + latest_visit_date, + latest_visit_kind, + latest_visit_outcome, + current_grade, + current_grade_date, + current_grade_basis, + graded_inspection_date, + ungraded_inspection_date, + rc_inspection_date, + inspection_type, + framework, + overall_effectiveness, + quality_of_education, + behaviour_attitudes, + personal_development, + leadership_management, + early_years_provision, + sixth_form_provision, + ungraded_outcome, + ungraded_grade, + rc_safeguarding_met, + rc_inclusion, + rc_curriculum_teaching, + rc_achievement, + rc_attendance_behaviour, + rc_personal_development, + rc_leadership_governance, + rc_early_years, + rc_sixth_form, + report_url +from {{ ref('int_ofsted_latest') }} diff --git a/pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql b/pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql new file mode 100644 index 0000000..a1e36ea --- /dev/null +++ b/pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql @@ -0,0 +1,10 @@ +-- A grade is dated by the inspection that awarded or confirmed it, never +-- after the latest visit, and has a date and a basis exactly when it exists. +-- A report card leaves no legacy grade in force. + +select urn +from {{ ref('fact_ofsted_latest') }} +where current_grade_date > latest_visit_date + or (current_grade is null) <> (current_grade_date is null) + or (current_grade is null) <> (current_grade_basis is null) + or (latest_visit_kind = 'report_card' and current_grade is not null)