From 93f17211b1046f06fd8cf9f0210b104308c8aae1 Mon Sep 17 00:00:00 2001 From: Tudor Date: Mon, 5 Oct 2026 16:39:24 +0100 Subject: [PATCH 1/2] fix(pipeline): read the latest Ofsted visit from the dates int_ofsted_latest treated any report card as the latest visit, relying on report cards postdating legacy inspections. That holds today (0 of 2,451 report-card schools in the 31 Aug 2026 MI) but is now not assumed: the latest visit is the newest of the three dates. A report card still leaves no legacy grade in force, which is the spec's rule rather than an inference. New unit test newer_legacy_visit_after_a_report_card failed RED on the old logic. Review on #183. Co-Authored-By: Claude Opus 5.5 --- .../models/intermediate/int_ofsted_latest.sql | 19 +++++++++++-------- .../models/intermediate/int_ofsted_latest.yml | 14 ++++++++++++++ ...assert_ofsted_current_grade_consistent.sql | 2 +- 3 files changed, 26 insertions(+), 9 deletions(-) diff --git a/pipeline/transform/models/intermediate/int_ofsted_latest.sql b/pipeline/transform/models/intermediate/int_ofsted_latest.sql index ac3727b..9dde247 100644 --- a/pipeline/transform/models/intermediate/int_ofsted_latest.sql +++ b/pipeline/transform/models/intermediate/int_ofsted_latest.sql @@ -7,12 +7,11 @@ with inspections as ( select *, - -- Report cards began in Nov 2025, after the last legacy inspections, - -- so a school's report card is always its latest visit. - coalesce( - rc_inspection_date, - greatest(graded_inspection_date, ungraded_inspection_date) - ) as latest_visit_date + -- The newest of the three inspections, read from the dates. (Report + -- cards began in Nov 2025, after the last legacy inspections, so today + -- a report card is always the latest; nothing below relies on that.) + greatest(rc_inspection_date, graded_inspection_date, ungraded_inspection_date) + as latest_visit_date from {{ ref('stg_ofsted_inspections') }} ), @@ -34,8 +33,9 @@ ranked as ( latest as ( select *, + -- Same-day ties resolve report card, then graded, then ungraded. case - when rc_inspection_date is not null then 'report_card' + when rc_inspection_date = latest_visit_date then 'report_card' when graded_inspection_date = latest_visit_date then 'graded' else 'ungraded' end as latest_visit_kind @@ -46,11 +46,14 @@ latest as ( graded as ( select *, - -- The grade still in force. A graded inspection's own overall grade + -- The grade still in force. A report card replaced overall grades, so + -- none survives it. Otherwise: a graded inspection's own overall grade -- (1-4; "Not judged" and the sentinel 9 are no grade); else an -- ungraded visit's "School remains X"; else, after an ungraded visit -- that names no grade, the graded inspection's grade. case + when rc_inspection_date is not null + then null when latest_visit_kind = 'graded' and overall_effectiveness between 1 and 4 then 'graded_latest' when latest_visit_kind = 'ungraded' and ungraded_grade is not null diff --git a/pipeline/transform/models/intermediate/int_ofsted_latest.yml b/pipeline/transform/models/intermediate/int_ofsted_latest.yml index c436b2e..da5ae0f 100644 --- a/pipeline/transform/models/intermediate/int_ofsted_latest.yml +++ b/pipeline/transform/models/intermediate/int_ofsted_latest.yml @@ -97,3 +97,17 @@ unit_tests: expect: rows: - {urn: 3, latest_visit_kind: graded, current_grade: null} + + - name: newer_legacy_visit_after_a_report_card + description: > + Not in Ofsted's data today (0 of 2,451 report-card schools in the 31 Aug + 2026 MI), but the latest visit is read from the dates, not assumed. The + report card still leaves no legacy grade in force. + model: int_ofsted_latest + given: + - input: ref('stg_ofsted_inspections') + rows: + - {urn: 4, ungraded_inspection_date: '2026-03-02', ungraded_grade: 2, ungraded_outcome: 'School remains Good', rc_inspection_date: '2025-12-01', rc_inclusion: 3} + expect: + rows: + - {urn: 4, latest_visit_date: '2026-03-02', latest_visit_kind: ungraded, latest_visit_outcome: 'School remains Good', current_grade: null, current_grade_date: null, current_grade_basis: null} diff --git a/pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql b/pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql index a1e36ea..78d11d5 100644 --- a/pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql +++ b/pipeline/transform/tests/assert_ofsted_current_grade_consistent.sql @@ -7,4 +7,4 @@ from {{ ref('fact_ofsted_latest') }} where current_grade_date > latest_visit_date or (current_grade is null) <> (current_grade_date is null) or (current_grade is null) <> (current_grade_basis is null) - or (latest_visit_kind = 'report_card' and current_grade is not null) + or (rc_inspection_date is not null and current_grade is not null) From 480ac4b9a97f840d7bc656a281a8529576080b13 Mon Sep 17 00:00:00 2001 From: Tudor Date: Mon, 5 Oct 2026 16:39:24 +0100 Subject: [PATCH 2/2] docs: plan names the session's trailer; spec reads the latest visit from dates The plan hard-coded one model's Co-Authored-By trailer; an executor should use the one its own session specifies. The spec's report-card rule now matches int_ofsted_latest. Review on #183. Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-10-05-ofsted-current-status.md | 4 ++-- .../specs/2026-10-05-ofsted-current-status-design.md | 10 ++++++---- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/plans/2026-10-05-ofsted-current-status.md b/docs/superpowers/plans/2026-10-05-ofsted-current-status.md index d8944c8..8b68047 100644 --- a/docs/superpowers/plans/2026-10-05-ofsted-current-status.md +++ b/docs/superpowers/plans/2026-10-05-ofsted-current-status.md @@ -17,7 +17,7 @@ - PR 2 must not be merged before `school_data_monthly_ofsted` has run on staging with PR 1, nor promoted before it has run on production. - No compatibility code in the backend for marts built before PR 1 (agreed in brainstorming). - Copy follows the site's factual rule: print what Ofsted published; no derived summaries. Dates in prose use long months ("14 March 2023"). -- Commits end with `Co-Authored-By: Claude Opus 5.5 `; PR bodies end with the Claude Code line. Never push to `main`. +- Commits end with the Co-Authored-By trailer the executing session's harness specifies (not a fixed model name); PR bodies end with the Claude Code line. Never push to `main`. ## Review Focus @@ -289,7 +289,7 @@ Expected: unit tests PASS; `fact_ofsted_inspection`'s `not_null` on `inspection_ ```bash git add pipeline/transform/models/staging/stg_ofsted_inspections.sql pipeline/transform/models/staging/stg_ofsted_inspections.yml -git commit -m "feat(pipeline): keep Ofsted's graded, ungraded and report-card dates apart" -m "int_ofsted_latest needs to know which inspection came last. Report-card-only schools are no longer dropped; inspection_date stays for current readers and now falls back to the report-card date." -m "Co-Authored-By: Claude Opus 5.5 " +git commit -m "feat(pipeline): keep Ofsted's graded, ungraded and report-card dates apart" -m "int_ofsted_latest needs to know which inspection came last. Report-card-only schools are no longer dropped; inspection_date stays for current readers and now falls back to the report-card date." -m "" ``` ### Task 3: `int_ofsted_latest` computes the current status diff --git a/docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md b/docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md index abee389..75dfb4f 100644 --- a/docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md +++ b/docs/superpowers/specs/2026-10-05-ofsted-current-status-design.md @@ -81,10 +81,12 @@ Computed once per URN in `int_ofsted_latest`. | Latest is ungraded, any other outcome ("Standards maintained", "Improved significantly", "Some aspects not as strong") | the graded inspection's overall grade if it is 1–4, else null | G when a grade is kept | `graded` when a grade is kept | | No inspection | null | null | null | -Report cards began in November 2025, after the last legacy graded and ungraded -inspections, so when a school has a report card it is always the latest visit, -and no legacy grade stays in force beside it. Ties between G, U and RC on the -same date resolve in the order report card, graded, ungraded. +A report card replaced overall grades, so no legacy grade stays in force beside +one. The latest visit is read from the dates, not assumed: report cards began in +November 2025, after the last legacy graded and ungraded inspections, and in +Ofsted's 31 Aug 2026 data no school has a legacy visit newer than its report +card, but the rule does not depend on that. Ties between G, U and RC on the same +date resolve in the order report card, graded, ungraded. Invariants: `current_grade` is null or 1–4; `current_grade_date <= latest_visit_date`; `current_grade_basis` is null exactly when `current_grade`