Commit Graph
1021 Commits
Author SHA1 Message Date
TudorandClaude Opus 5.5 32715af568 fix(ofsted): a gov.uk outage skips the FE streams, not the Ofsted extract (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:55:22 +01:00
TudorandClaude Opus 5.5 c16dd0044f fix(dags): build stg_gias_ukprn monthly, never daily (review)
Rebuilding the view in the daily build dropped the Ofsted views on it
(CASCADE), which dim_school reads: every daily run would have failed. A
selector test now rejects any build that drops views it then reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:54:51 +01:00
TudorandClaude Opus 5.5 cfc9367bb3 fix(dbt): stg_gias_ukprn reads URN as the bigint target-postgres loads (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:28:51 +01:00
TudorandClaude Opus 5.5 8e82e2e4cf feat(dbt): dim_school.inspectorate and ofsted_listed (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:26:42 +01:00
TudorandClaude Opus 5.5 6cc135ec58 feat(dbt): further education status where a school has no school record (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:26:05 +01:00
TudorandClaude Opus 5.5 222886465a feat(dbt): further education status per URN under the C1 rule (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:25:26 +01:00
TudorandClaude Opus 5.5 6abee62c2a feat(dbt): stage Ofsted's further education and skills MI (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:25:04 +01:00
TudorandClaude Opus 5.5 ab3e12e1ac feat(ofsted): extract Ofsted's further education and skills MI (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:24:37 +01:00
TudorandClaude Opus 5.5 1949e85c58 feat(gias): load UKPRN and the inspectorate GIAS names (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:23:31 +01:00
TudorandClaude Opus 5.5 000626c5d6 docs: plan for inspection status beyond Ofsted's school data (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:00:54 +01:00
TudorandClaude Opus 5.5 5f7528b399 docs: design for inspection status beyond Ofsted's school data (H4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:00:38 +01:00
tudor 2b5e681482 Merge pull request 'fix(site): DfE LA averages for "vs LA avg"; no gap for independents (H2, part 2 of 2)' (#186) from fix/la-averages-site into main
Stage (build -> staging -> E2E gate) / prepare (push) Successful in 1s
Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 21s
Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m38s
Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 13s
Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 26s
Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Successful in 3m30s
Reviewed-on: #186
2026-10-06 15:07:36 +00:00
tudor 3bd1dbd276 Merge pull request 'fix(pipeline): load 2023/24 KS4/KS2 data and DfE LA averages (C2, H2, part 1 of 2)' (#185) from fix/ks4-2023-24-pipeline into main
Stage (build -> staging -> E2E gate) / prepare (push) Successful in 0s
Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 45s
Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m33s
Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 2m6s
Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 6s
Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Successful in 3m27s
Reviewed-on: #185
2026-10-06 12:42:19 +00:00
TudorandClaude Opus 5.5 870af949ee test(e2e): prove the independent row against a state row's gap (review)
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m14s
PR Checks / Backend Smoke (pull_request) Successful in 10s
PR Checks / Build Backend (no push) (pull_request) Successful in 33s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m26s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 1m16s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 21s
The journey passed whenever the LA map was empty or not yet rendered. It now
waits for a state school in the same LA to show its gap, and both LA journeys
fail rather than skip on an empty map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:51:18 +01:00
TudorandClaude Opus 5.5 5f93a7ecd2 docs: name the benchmarks computed from our dataset (review)
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m19s
PR Checks / Backend Smoke (pull_request) Successful in 11s
PR Checks / Build Backend (no push) (pull_request) Successful in 35s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m36s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 1m17s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 35s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:50:40 +01:00
TudorandClaude Opus 5.5 951c666f8a test(dbt): warn when DfE's LA averages fall behind the school results (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:50:27 +01:00
TudorandClaude Opus 5.5 e09d7a202b fix(ees): read the year from suffixed release slugs (review)
A '2025-26-revised' release read as an unknown year went last, behind the
'2025-26' release, which then owned the year and dropped every revised row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:50:10 +01:00
TudorandClaude Opus 5.5 c6ff77f05b test(e2e): 2023/24 results and no LA gap for independents (C2, H2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:37:02 +01:00
TudorandClaude Opus 5.5 c7ddf0505d fix(search): no LA comparison for independent schools (H2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:36:00 +01:00
TudorandClaude Opus 5.5 e659867590 fix(api): serve DfE's LA averages for "vs LA avg" (H2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:35:11 +01:00
TudorandClaude Opus 5.5 4ae3853277 fix(dbt): read percentages DfE writes as "34%" (C2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:33:36 +01:00
TudorandClaude Opus 5.5 242c603aec test(dbt): fail the build when a published year loads empty (C2, H2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:33:03 +01:00
TudorandClaude Opus 5.5 59dd20ab00 feat(dbt): fact_ks4_la_averages from DfE's LA rows (H2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:32:22 +01:00
TudorandClaude Opus 5.5 dbb60acff3 feat(ees): keep DfE's KS4 LA averages from the summary data set (H2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:46:53 +01:00
TudorandClaude Opus 5.5 967b1f0eed fix(ees): read 2023/24 KS4 school information under DfE's older names (C2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:45:57 +01:00
TudorandClaude Opus 5.5 4db1131d0f fix(ees): newest KS4 release owns every year it contains (C2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:45:14 +01:00
TudorandClaude Opus 5.5 0e987ef06e docs: plan for 2023/24 KS4/KS2 data and DfE LA averages (C2, H2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:34:00 +01:00
TudorandClaude Opus 5.5 ac5b7ccd7f docs: design for 2023/24 KS4/KS2 data and DfE LA averages (C2, H2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:08:25 +01:00
tudor c26b65246f Merge pull request 'fix: show the Ofsted grade still in force and the latest visit (C1/M1/M2, part 2 of 2)' (#184) from fix/ofsted-current-status-site into main
Stage (build -> staging -> E2E gate) / prepare (push) Successful in 0s
Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 21s
Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m36s
Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 5s
Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Successful in 3m22s
Reviewed-on: #184
2026-10-05 21:47:34 +00:00
tudor 3728a63275 Merge pull request 'feat(pipeline): one current Ofsted status per school (C1/M1, part 1 of 2)' (#183) from fix/ofsted-current-status-pipeline into main
Stage (build -> staging -> E2E gate) / prepare (push) Successful in 1s
Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 20s
Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m31s
Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m19s
Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 6s
Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Successful in 3m24s
Reviewed-on: #183
2026-10-05 15:44:59 +00:00
Tudor 90b7d09086 Merge #183's review fixes into the site branch
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m14s
PR Checks / Backend Smoke (pull_request) Successful in 10s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m26s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 52s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 26s
2026-10-05 16:39:34 +01:00
TudorandClaude Opus 5.5 480ac4b9a9 docs: plan names the session's trailer; spec reads the latest visit from dates
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m14s
PR Checks / Backend Smoke (pull_request) Successful in 12s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m27s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 53s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 27s
The plan hard-coded one model's Co-Authored-By trailer; an executor should use the one its own session specifies. The spec's report-card rule now matches int_ofsted_latest. Review on #183.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:39:24 +01:00
TudorandClaude Opus 5.5 93f17211b1 fix(pipeline): read the latest Ofsted visit from the dates
int_ofsted_latest treated any report card as the latest visit, relying on report cards postdating legacy inspections. That holds today (0 of 2,451 report-card schools in the 31 Aug 2026 MI) but is now not assumed: the latest visit is the newest of the three dates. A report card still leaves no legacy grade in force, which is the spec's rule rather than an inference. New unit test newer_legacy_visit_after_a_report_card failed RED on the old logic. Review on #183.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:39:24 +01:00
TudorandClaude Opus 5.5 f86d6f45d1 test(e2e): no grade is carried past a no-grade inspection
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m14s
PR Checks / Backend Smoke (pull_request) Successful in 10s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m25s
PR Checks / Build Pipeline (no push) (pull_request) Canceled after 45s
PR Checks / AI Code Review (Claude) (pull_request) Canceled after 0s
Rabbsfarm (102408) must read 'Inspected · 2025' with no overall grade, and Robins Lane (104762) must say its Good was confirmed at an ungraded inspection on 18 July 2024. Both fail against staging today (no current_grade field, no confirmation line) and pass once PR #183's pipeline run and this branch are deployed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:33:53 +01:00
TudorandClaude Opus 5.5 5eed09dfa7 refactor(utils): drop the unused Ofsted hero chip and summary sentence
buildOfstedHeroChip and buildSchoolSummary were rendered nowhere and encoded the old carried-forward rule (and a framework value, 'ReportCard', the API never sends).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:32:53 +01:00
TudorandClaude Opus 5.5 d37baa572b fix(compare): Ofsted rows show the grade still in force and the latest visit
'Inspected' is the school's latest visit, so Washwood Heath reads 21 May 2025 rather than '3 Mar 2020 · 4+ years ago' (audit M1). The Result row names the inspection a grade came from, or says 'No overall grade'; 'Grade carried forward' and 'transitional framework' go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:32:19 +01:00
TudorandClaude Opus 5.5 9ef48da4b1 fix(school): the Ofsted section shows the grade still in force and the latest visit
The section is dated by the latest visit and headlines the grade still in force with the inspection that awarded or confirmed it, or 'No overall grade'. A later visit that isn't the grade's source gets its own line, with an ungraded outcome such as 'Standards maintained'. The primary and secondary no-grade branches merge, so the secondary page no longer drops the sixth-form and early-years judgements (audit M2), and 'Not rated' goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:31:26 +01:00
TudorandClaude Opus 5.5 a42c586cc8 fix(ofsted): badge and compare read the grade still in force
Search badges date a grade by the inspection that awarded or confirmed it (ofsted_grade_date) and say 'Inspected · year' when the latest inspection gave no grade. ofstedDisplay's kinds become graded / confirmed / no_overall_grade, read from current_grade rather than a carried-forward overall_effectiveness. lib/ofstedStatus.ts holds the two sentences the school and compare pages share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:29:54 +01:00
TudorandClaude Opus 5.5 a9e3a6a700 fix(backend): serve the current Ofsted status from fact_ofsted_latest
The list query and the batch Ofsted fetch each picked 'the latest row' of fact_ofsted_inspection themselves, and _ofsted_block carried an older ungraded grade forward when the latest graded inspection gave none. All three now read marts.fact_ofsted_latest. List rows: ofsted_grade is the grade still in force, ofsted_grade_date when it was awarded or confirmed, ofsted_date the latest visit. The ofsted block gains current_grade and latest_visit and loses grade_source; overall_effectiveness is the graded inspection's own result. A school with only an inspection stays publishable in the sitemap.

Requires fact_ofsted_latest (PR #183's pipeline run) on the database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:28:30 +01:00
TudorandClaude Opus 5.5 96732c56d1 fix(pipeline): dim_school's Ofsted grade is the one still in force
dim_school.ofsted_grade (Typesense's rating) coalesced the graded grade with an older ungraded visit's, so Rabbsfarm's 2020 'remains Good' survived its 2025 no-grade inspection. It now reads int_ofsted_latest's current_grade, and ofsted_date is the latest visit. Unit tests pin both (audit C1, M1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:26:51 +01:00
Tudor befaf950a7 Merge PR 1 (fix/ofsted-current-status-pipeline, #183) into the site branch
PR 2 reads the columns PR 1 adds. Merge main back in once #183 lands.
2026-10-05 16:25:49 +01:00
TudorandClaude Opus 5.5 dfa4928641 feat(pipeline): add fact_ofsted_latest, one current Ofsted status per school
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m18s
PR Checks / Backend Smoke (pull_request) Successful in 10s
PR Checks / Build Backend (no push) (pull_request) Successful in 20s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m31s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 50s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 32s
The backend will read this instead of picking the latest row of fact_ofsted_inspection itself (twice, with arbitrary ties). Schema tests and assert_ofsted_current_grade_consistent pin its invariants. Built by the monthly Ofsted DAG (int_ofsted_latest+), not the daily one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:24:41 +01:00
TudorandClaude Opus 5.5 e6babe21f5 feat(pipeline): work out each school's current Ofsted status once
int_ofsted_latest now picks each school's latest visit (report card, graded or ungraded inspection) and the overall grade still in force, dated by the inspection that awarded or confirmed it. A 'School remains Good' from an older ungraded visit is no longer carried past a newer inspection that gave no grade (audit C1). Duplicate monthly rows resolve to the newest visit, so a newer report card always wins. dim_school's columns are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:23:38 +01:00
TudorandClaude Opus 5.5 358705bf39 feat(pipeline): keep Ofsted's graded, ungraded and report-card dates apart
int_ofsted_latest needs to know which inspection came last. Report-card-only schools are no longer dropped (123 schools, audit H3); inspection_date stays for current readers and now falls back to the report-card date, so fact_ofsted_inspection's not_null test still holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:22:49 +01:00
TudorandClaude Opus 5.5 3be902e98f docs(plan): implementation plan for the current Ofsted status
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:20:49 +01:00
TudorandClaude Opus 5.5 12c52244ee docs(spec): one current Ofsted status per school
Design for audit findings C1 and M1. A grade carried forward from an older
ungraded visit is shown under a newer inspection's date ("Good · 2025" for
Rabbsfarm, whose 2025 inspection gave no grade), and "Inspected" dates show the
last graded inspection rather than the latest visit. The rule moves into
int_ofsted_latest and a new fact_ofsted_latest mart, shipped as a pipeline PR
and then a backend/UI PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:00:35 +01:00
tudor b6e48c4930 Merge pull request 'fix(pipeline): decode GIAS extracts as Windows-1252' (#182) from fix/gias-encoding into main
Stage (build -> staging -> E2E gate) / prepare (push) Successful in 1s
Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 21s
Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m32s
Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m37s
Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 5s
Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Successful in 3m21s
Reviewed-on: #182
2026-10-05 06:31:30 +00:00
tudor 9f4f2507cc Merge pull request 'fix(pipeline): keep the destinations marts out of the scheduled builds' (#181) from fix/scheduled-dbt-selectors into main
Stage (build -> staging -> E2E gate) / prepare (push) Successful in 1s
Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 47s
Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m35s
Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m20s
Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 3s
Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Successful in 3m26s
Reviewed-on: #181
2026-10-05 06:19:44 +00:00
tudor e1373fb6df Merge pull request 'fix(school): drop the nearby section's lede, which repeated its heading' (#180) from fix/nearby-drop-lede into main
Stage (build -> staging -> E2E gate) / prepare (push) Successful in 1s
Stage (build -> staging -> E2E gate) / Build Backend (FastAPI) (push) Successful in 19s
Stage (build -> staging -> E2E gate) / Build Frontend (Next.js) (push) Successful in 1m31s
Stage (build -> staging -> E2E gate) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 2m8s
Stage (build -> staging -> E2E gate) / Deploy to Staging (push) Successful in 4s
Stage (build -> staging -> E2E gate) / E2E Journeys against Staging (push) Failing after 45s
Reviewed-on: #180
2026-10-04 09:13:48 +00:00
TudorandClaude Opus 5.5 94bfac9caf fix(pipeline): decode GIAS extracts as Windows-1252
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m16s
PR Checks / Backend Smoke (pull_request) Successful in 10s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m27s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 1m17s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 19s
GIAS publishes its CSVs in Windows-1252 and sends no charset. The tap read
resp.text, so requests guessed the codec, and the encoding="latin-1" passed
to read_csv did nothing on already-decoded text. On 3 Oct 2026 the guess was
windows-1250, and "St Thomas à Becket" (138950, 149557) was stored as
"St Thomas ŕ Becket". A different guess on another day would garble other
accented names.

Both streams now decode the downloaded bytes themselves (gias_csv.py). A byte
Windows-1252 leaves undefined becomes U+FFFD with a logged warning instead of
failing the load, so one odd name cannot stop the daily refresh. None of the
nine extracts checked (1 Jul to 3 Oct 2026) contains such a byte.

Checked by running the tap on the real 3 Oct extract with .text forced to
windows-1250: all 52,586 rows decode, with no "ŕ" and no replacement
characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 10:09:39 +01:00