Data-foundation PR for the compare-screen redesign — implements spec §5 + §8 of docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md (PR 1 of 3; backend and frontend PRs follow).
New mart columns (all additive; backend mapping comes in the next PR)
fact_ks2_national_averages: year floor widened to 2015/16 (row exists at source)
fact_ofsted_inspection:rc_* report-card columns now populated from the MI file (5-point scale coded 1–5; safeguarding as three-state boolean). rc_early_years/rc_sixth_form remain deliberate NULL stubs — no such columns exist in the MI file.
Evidence-first approach
pipeline/scripts/diagnose_compare_gaps.py carries committed evidence blocks (FINDINGS / TASK 6 VERIFICATION / TASK 7 VALUE SAMPLE) that justify each judgement call:
GPS/science/scaled-score national mapping is correct end-to-end; prod NULLs are stale raw data — the next ees_ks2_national extract run backfills them. No code change needed.
2021/22 school-level KS2 is a permanent DfE source gap (DfE never published it — verified against EES, the CSP download wizard, and DfE release notes). Spec updated; no filebrowser upload needed; frontend copy should say "DfE didn't publish school-level figures for 2021/22".
Ofsted report-card scale vocabulary sampled live: Exceptional / Strong standard / Expected standard / Needs attention / Urgent improvement ("Needs attention", not the consultation draft's "Attention needed").
Drive-by fix
Pre-existing bug: the KS4 union (int_ks4_with_lineage) aligns by position and the two staging branches had sen_ehcp_pct/sen_support_pct swapped — legacy-lineage schools had swapped SEN values in the mart. Fixed; full 26-column alignment audited.
Deploy notes
Until the next monthly Ofsted tap run adds rc_* to raw.ofsted_inspections, a manual full dbt build would fail compiling stg_ofsted_inspections; the DAGs are safe (extract runs before build; verified no other DAG selects that model).
progress_8_banding accepted_values ships at severity: warn (real values unsampled); tighten to error once the weekly EES run confirms the vocabulary.
Cross-LA staging identifiers are double-quoted per the file's FSM_eligible_percent precedent; if the DAG run errors on casing, the documented fallback is to drop the quotes.
Post-merge verification (plan Task 8)
/api/national-averages gains a 201516 primary row and non-NULL gps/science/scaled scores after the next national extract.
New mart columns spot-checked in Postgres (select count(*) from marts.fact_admissions where second_preference_offers is not null; etc.).
Data-foundation PR for the compare-screen redesign — implements spec §5 + §8 of `docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md` (PR 1 of 3; backend and frontend PRs follow).
## New mart columns (all additive; backend mapping comes in the next PR)
- **fact_admissions:** `total_offers`, `second_preference_offers`, `third_preference_offers`, `cross_la_applications`, `cross_la_offers`
- **fact_ks2_performance:** `reading/writing/maths_progress_lower_ci` + `_upper_ci`, `writing_working_towards_pct`
- **fact_ks4_performance:** `progress_8_banding`, `attainment_8_disadvantage_gap`, `progress_8_disadvantage_gap`
- **fact_ks2_national_averages:** year floor widened to 2015/16 (row exists at source)
- **fact_ofsted_inspection:** `rc_*` report-card columns now populated from the MI file (5-point scale coded 1–5; safeguarding as three-state boolean). `rc_early_years`/`rc_sixth_form` remain deliberate NULL stubs — no such columns exist in the MI file.
## Evidence-first approach
`pipeline/scripts/diagnose_compare_gaps.py` carries committed evidence blocks (FINDINGS / TASK 6 VERIFICATION / TASK 7 VALUE SAMPLE) that justify each judgement call:
- GPS/science/scaled-score national mapping is **correct end-to-end**; prod NULLs are stale raw data — the next `ees_ks2_national` extract run backfills them. No code change needed.
- **2021/22 school-level KS2 is a permanent DfE source gap** (DfE never published it — verified against EES, the CSP download wizard, and DfE release notes). Spec updated; no filebrowser upload needed; frontend copy should say "DfE didn't publish school-level figures for 2021/22".
- Ofsted report-card scale vocabulary sampled live: `Exceptional / Strong standard / Expected standard / Needs attention / Urgent improvement` ("Needs attention", not the consultation draft's "Attention needed").
## Drive-by fix
Pre-existing bug: the KS4 union (`int_ks4_with_lineage`) aligns by position and the two staging branches had `sen_ehcp_pct`/`sen_support_pct` swapped — legacy-lineage schools had swapped SEN values in the mart. Fixed; full 26-column alignment audited.
## Deploy notes
- Until the next monthly Ofsted tap run adds `rc_*` to `raw.ofsted_inspections`, a *manual* full `dbt build` would fail compiling `stg_ofsted_inspections`; the DAGs are safe (extract runs before build; verified no other DAG selects that model).
- `progress_8_banding` accepted_values ships at `severity: warn` (real values unsampled); tighten to error once the weekly EES run confirms the vocabulary.
- Cross-LA staging identifiers are double-quoted per the file's `FSM_eligible_percent` precedent; if the DAG run errors on casing, the documented fallback is to drop the quotes.
## Post-merge verification (plan Task 8)
- `/api/national-averages` gains a 201516 primary row and non-NULL gps/science/scaled scores after the next national extract.
- New mart columns spot-checked in Postgres (`select count(*) from marts.fact_admissions where second_preference_offers is not null;` etc.).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
int_ks4_with_lineage.sql unions stg_ees_ks4 and stg_legacy_ks4 via
`select *`, which PostgreSQL aligns positionally. stg_legacy_ks4 listed
sen_support_pct before sen_ehcp_pct while stg_ees_ks4 lists sen_ehcp_pct
before sen_support_pct, swapping the two values for legacy-sourced rows
in marts.fact_ks4_performance. Reordered stg_legacy_ks4's final select
to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
Widen the year filter in stg_ees_ks2_national.sql from >= 201617 to
>= 201516 so the England national-averages line no longer starts a
year late; the catalogue CSV has a real, comparable 201516 row (2015/16
was the first year of the current expected-standard tests, so it's the
correct floor).
GPS/science/scaled-score national columns confirmed present at source
with correct mapping; prod NULLs are stale raw data, backfilled by the
next extract run. No _KS2_NATIONAL_COL_MAP change accompanies this fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
DfE never published school-level KS2 2021/22 data publicly (confirmed via
EES release notes and by walking the Compare School Performance download
wizard, which has no ks2 checkbox for 2021-2022, same as the COVID-cancelled
2020-2021 year). No archive exists to verify column headers against or
upload to the filebrowser; Task 6 is blocked at the source-data level.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
Wires the tap TODO in stg_ofsted_inspections.sql: maps the 7 confirmed
report-card MI columns (Safeguarding standards, Inclusion, Curriculum
and teaching, Achievement, Attendance and behaviour, Personal
development and wellbeing, Leadership and governance) into rc_*
fields, parsed via the new parse_report_card_grade macro against
real sampled grade values (Exceptional/Strong standard/Expected
standard/Needs attention/Urgent improvement). rc_safeguarding_met
becomes boolean from Met/Not met. rc_early_years/rc_sixth_form have
no MI column yet and are intentionally omitted from COLUMN_PRIORITY,
staying NULL.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
Evidence trail for the rc_* mapping in the prior commit: real value_counts()
over the 7 MI report-card columns, confirming the 5-value grade vocabulary
and that 'Achievement'/'Safeguarding standards' match by exact string only
(no legacy OEIF column accidentally consumed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
This PR promotes several raw-but-unstored fields into dbt marts (admissions preference breakdown, KS2 progress confidence intervals, KS4 Progress 8 banding/disadvantage gaps, Ofsted report-card grades) plus supporting docs/specs and a diagnostic script. The KS4 and admissions changes are internally consistent, but the KS2 change breaks the union between EES and legacy KS2 data.
🔴 Severe (blocks merge)
pipeline/transform/models/staging/stg_ees_ks2.sql: Adds 7 new columns (reading/writing/maths_progress_lower_ci, reading/writing/maths_progress_upper_ci, writing_working_towards_pct) to stg_ees_ks2's output, but pipeline/transform/models/intermediate/int_ks2_with_lineage.sql (unchanged in this PR) does select * from stg_ees_ks2 union all select * from stg_legacy_ks2, and stg_legacy_ks2.sql (also unchanged) does not have these columns. This UNION ALL requires identical column counts/order between the two selects, so dbt run/dbt build will fail on this model with a column-count mismatch. Even if it didn't, int_ks2_with_lineage's current_ks2/predecessor_ks2 CTEs enumerate columns explicitly and don't include the 7 new ones, so fact_ks2_performance.sql's select reading_progress_lower_ci, ... would fail to compile since int_ks2_with_lineage never emits them. This is exactly the pattern the PR got right for KS4 (stg_legacy_ks4.sql was given matching null placeholder columns and int_ks4_with_lineage.sql was updated to pass the new columns through in both CTEs) but missed for KS2. Since local gating is only dbt parse (no DB, doesn't validate UNION column counts), this would only surface when the Airflow DAG runs the real dbt build on the server, breaking the core KS2 performance mart used site-wide (not just the compare feature).
🟡 Minor
pipeline/transform/models/staging/stg_ees_ks2_national.sql: The comment claims the GPS/science NULLs in prod are due to 'stale raw.ees_ks2_national data' rather than a mapping bug, but this PR makes no change to actually refresh that raw data or otherwise verify/fix the NULLs — only the year floor was widened (201617→201516). The stated Task 5 goal (fixing GPS/science NULLs) is asserted as resolved-by-refresh but isn't actually verified or addressed by any code in this diff.
pipeline/transform/models/staging/stg_ees_admissions.sql: Uses double-quoted mixed-case identifiers ("all_applications_from_another_LA", "offers_to_applicants_from_another_LA") inside safe_numeric(), which will fail to resolve if Postgres actually stores these raw columns lower-cased (the surrounding comment itself flags this as an unverified assumption to be checked against a real DAG run/DB, with no such verification evident in this diff).
## 🤖 AI Code Review (Claude Code)
This PR promotes several raw-but-unstored fields into dbt marts (admissions preference breakdown, KS2 progress confidence intervals, KS4 Progress 8 banding/disadvantage gaps, Ofsted report-card grades) plus supporting docs/specs and a diagnostic script. The KS4 and admissions changes are internally consistent, but the KS2 change breaks the union between EES and legacy KS2 data.
### 🔴 Severe (blocks merge)
- **pipeline/transform/models/staging/stg_ees_ks2.sql**: Adds 7 new columns (reading/writing/maths_progress_lower_ci, reading/writing/maths_progress_upper_ci, writing_working_towards_pct) to stg_ees_ks2's output, but pipeline/transform/models/intermediate/int_ks2_with_lineage.sql (unchanged in this PR) does `select * from stg_ees_ks2 union all select * from stg_legacy_ks2`, and stg_legacy_ks2.sql (also unchanged) does not have these columns. This UNION ALL requires identical column counts/order between the two selects, so `dbt run`/`dbt build` will fail on this model with a column-count mismatch. Even if it didn't, int_ks2_with_lineage's current_ks2/predecessor_ks2 CTEs enumerate columns explicitly and don't include the 7 new ones, so fact_ks2_performance.sql's `select reading_progress_lower_ci, ...` would fail to compile since int_ks2_with_lineage never emits them. This is exactly the pattern the PR got right for KS4 (stg_legacy_ks4.sql was given matching null placeholder columns and int_ks4_with_lineage.sql was updated to pass the new columns through in both CTEs) but missed for KS2. Since local gating is only `dbt parse` (no DB, doesn't validate UNION column counts), this would only surface when the Airflow DAG runs the real `dbt build` on the server, breaking the core KS2 performance mart used site-wide (not just the compare feature).
### 🟡 Minor
- **pipeline/transform/models/staging/stg_ees_ks2_national.sql**: The comment claims the GPS/science NULLs in prod are due to 'stale raw.ees_ks2_national data' rather than a mapping bug, but this PR makes no change to actually refresh that raw data or otherwise verify/fix the NULLs — only the year floor was widened (201617→201516). The stated Task 5 goal (fixing GPS/science NULLs) is asserted as resolved-by-refresh but isn't actually verified or addressed by any code in this diff.
- **pipeline/transform/models/staging/stg_ees_admissions.sql**: Uses double-quoted mixed-case identifiers (`"all_applications_from_another_LA"`, `"offers_to_applicants_from_another_LA"`) inside safe_numeric(), which will fail to resolve if Postgres actually stores these raw columns lower-cased (the surrounding comment itself flags this as an unverified assumption to be checked against a real DAG run/DB, with no such verification evident in this diff).
The AI review gate caught that stg_ees_ks2's 7 new columns broke the
positional UNION ALL with stg_legacy_ks2 in int_ks2_with_lineage, and
that the lineage CTEs never emitted them (same class of bug fixed for
KS4 in 34a5de2). Legacy gets typed null placeholders at matching
positions; both lineage CTEs pass the columns through. 45/45 columns
verified name-identical in order across both union branches.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
This PR is mostly documentation (a compare-screen redesign spec, expert review, and implementation plan) plus a focused pipeline/dbt change that promotes several raw-but-unstored fields into marts: admissions preference/cross-LA breakdowns, KS2 progress confidence intervals, KS4 Progress 8 banding/disadvantage gaps, a widened national-averages year filter, and real Ofsted report-card (rc_) values replacing NULL stubs. The SQL changes are careful and internally consistent: column order in every union all/select * combination (stg_ees_ks2 vs stg_legacy_ks2, stg_ees_ks4 vs stg_legacy_ks4) lines up correctly, new raw field names match the tap schemas exactly, and the new boolean/integer types for rc_ columns match what backend/models.py and migration.py already expect. The diagnostic script backing this PR correctly identified that 2021/22 school-level KS2 and the 2022/23 subject-level file are genuine DfE publication gaps, and the plan's speculative 'Attention needed' grade variant was correctly dropped from the shipped macro since it never appears in real data.
🟡 Minor
pipeline/transform/models/staging/stg_ees_admissions.sql: The new cross-LA columns reference double-quoted mixed-case raw identifiers ("all_applications_from_another_LA", "offers_to_applicants_from_another_LA") by analogy with the existing FSM_eligible_percent precedent, but this casing was never actually verified against target-postgres's column-name folding (unlike Tasks 1/5/6/7, no diagnostic check confirmed it). Since local CI only runs dbt parse (no DB access), a casing mismatch would only surface as a column does not exist failure when the Airflow DAG runs dbt build in production, breaking stg_ees_admissions/fact_admissions for that run.
pipeline/transform/models/marts/_marts_schema.yml: The progress_8_banding accepted_values list ('Well above average' … 'Well below average') is asserted without having been diagnostically verified against real EES progress8_banding text (Task 1 verified GPS/science, KS2 subjects, and Ofsted rc columns, but not this field). It's mitigated by severity: warn so it won't fail CI, but the labels could be wrong until someone checks the actual data.
## 🤖 AI Code Review (Claude Code)
This PR is mostly documentation (a compare-screen redesign spec, expert review, and implementation plan) plus a focused pipeline/dbt change that promotes several raw-but-unstored fields into marts: admissions preference/cross-LA breakdowns, KS2 progress confidence intervals, KS4 Progress 8 banding/disadvantage gaps, a widened national-averages year filter, and real Ofsted report-card (rc_*) values replacing NULL stubs. The SQL changes are careful and internally consistent: column order in every `union all`/`select *` combination (stg_ees_ks2 vs stg_legacy_ks2, stg_ees_ks4 vs stg_legacy_ks4) lines up correctly, new raw field names match the tap schemas exactly, and the new boolean/integer types for rc_* columns match what backend/models.py and migration.py already expect. The diagnostic script backing this PR correctly identified that 2021/22 school-level KS2 and the 2022/23 subject-level file are genuine DfE publication gaps, and the plan's speculative 'Attention needed' grade variant was correctly dropped from the shipped macro since it never appears in real data.
### 🟡 Minor
- **pipeline/transform/models/staging/stg_ees_admissions.sql**: The new cross-LA columns reference double-quoted mixed-case raw identifiers (`"all_applications_from_another_LA"`, `"offers_to_applicants_from_another_LA"`) by analogy with the existing FSM_eligible_percent precedent, but this casing was never actually verified against target-postgres's column-name folding (unlike Tasks 1/5/6/7, no diagnostic check confirmed it). Since local CI only runs `dbt parse` (no DB access), a casing mismatch would only surface as a `column does not exist` failure when the Airflow DAG runs `dbt build` in production, breaking stg_ees_admissions/fact_admissions for that run.
- **pipeline/transform/models/marts/_marts_schema.yml**: The `progress_8_banding` accepted_values list ('Well above average' … 'Well below average') is asserted without having been diagnostically verified against real EES `progress8_banding` text (Task 1 verified GPS/science, KS2 subjects, and Ofsted rc columns, but not this field). It's mitigated by `severity: warn` so it won't fail CI, but the labels could be wrong until someone checks the actual data.
tudor
merged commit 3754947fd6 into main2026-07-13 12:52:11 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Data-foundation PR for the compare-screen redesign — implements spec §5 + §8 of
docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md(PR 1 of 3; backend and frontend PRs follow).New mart columns (all additive; backend mapping comes in the next PR)
total_offers,second_preference_offers,third_preference_offers,cross_la_applications,cross_la_offersreading/writing/maths_progress_lower_ci+_upper_ci,writing_working_towards_pctprogress_8_banding,attainment_8_disadvantage_gap,progress_8_disadvantage_gaprc_*report-card columns now populated from the MI file (5-point scale coded 1–5; safeguarding as three-state boolean).rc_early_years/rc_sixth_formremain deliberate NULL stubs — no such columns exist in the MI file.Evidence-first approach
pipeline/scripts/diagnose_compare_gaps.pycarries committed evidence blocks (FINDINGS / TASK 6 VERIFICATION / TASK 7 VALUE SAMPLE) that justify each judgement call:ees_ks2_nationalextract run backfills them. No code change needed.Exceptional / Strong standard / Expected standard / Needs attention / Urgent improvement("Needs attention", not the consultation draft's "Attention needed").Drive-by fix
Pre-existing bug: the KS4 union (
int_ks4_with_lineage) aligns by position and the two staging branches hadsen_ehcp_pct/sen_support_pctswapped — legacy-lineage schools had swapped SEN values in the mart. Fixed; full 26-column alignment audited.Deploy notes
rc_*toraw.ofsted_inspections, a manual fulldbt buildwould fail compilingstg_ofsted_inspections; the DAGs are safe (extract runs before build; verified no other DAG selects that model).progress_8_bandingaccepted_values ships atseverity: warn(real values unsampled); tighten to error once the weekly EES run confirms the vocabulary.FSM_eligible_percentprecedent; if the DAG run errors on casing, the documented fallback is to drop the quotes.Post-merge verification (plan Task 8)
/api/national-averagesgains a 201516 primary row and non-NULL gps/science/scaled scores after the next national extract.select count(*) from marts.fact_admissions where second_preference_offers is not null;etc.).🤖 Generated with Claude Code
https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
🤖 AI Code Review (Claude Code)
This PR promotes several raw-but-unstored fields into dbt marts (admissions preference breakdown, KS2 progress confidence intervals, KS4 Progress 8 banding/disadvantage gaps, Ofsted report-card grades) plus supporting docs/specs and a diagnostic script. The KS4 and admissions changes are internally consistent, but the KS2 change breaks the union between EES and legacy KS2 data.
🔴 Severe (blocks merge)
select * from stg_ees_ks2 union all select * from stg_legacy_ks2, and stg_legacy_ks2.sql (also unchanged) does not have these columns. This UNION ALL requires identical column counts/order between the two selects, sodbt run/dbt buildwill fail on this model with a column-count mismatch. Even if it didn't, int_ks2_with_lineage's current_ks2/predecessor_ks2 CTEs enumerate columns explicitly and don't include the 7 new ones, so fact_ks2_performance.sql'sselect reading_progress_lower_ci, ...would fail to compile since int_ks2_with_lineage never emits them. This is exactly the pattern the PR got right for KS4 (stg_legacy_ks4.sql was given matching null placeholder columns and int_ks4_with_lineage.sql was updated to pass the new columns through in both CTEs) but missed for KS2. Since local gating is onlydbt parse(no DB, doesn't validate UNION column counts), this would only surface when the Airflow DAG runs the realdbt buildon the server, breaking the core KS2 performance mart used site-wide (not just the compare feature).🟡 Minor
"all_applications_from_another_LA","offers_to_applicants_from_another_LA") inside safe_numeric(), which will fail to resolve if Postgres actually stores these raw columns lower-cased (the surrounding comment itself flags this as an unverified assumption to be checked against a real DAG run/DB, with no such verification evident in this diff).🤖 AI Code Review (Claude Code)
This PR is mostly documentation (a compare-screen redesign spec, expert review, and implementation plan) plus a focused pipeline/dbt change that promotes several raw-but-unstored fields into marts: admissions preference/cross-LA breakdowns, KS2 progress confidence intervals, KS4 Progress 8 banding/disadvantage gaps, a widened national-averages year filter, and real Ofsted report-card (rc_) values replacing NULL stubs. The SQL changes are careful and internally consistent: column order in every
union all/select *combination (stg_ees_ks2 vs stg_legacy_ks2, stg_ees_ks4 vs stg_legacy_ks4) lines up correctly, new raw field names match the tap schemas exactly, and the new boolean/integer types for rc_ columns match what backend/models.py and migration.py already expect. The diagnostic script backing this PR correctly identified that 2021/22 school-level KS2 and the 2022/23 subject-level file are genuine DfE publication gaps, and the plan's speculative 'Attention needed' grade variant was correctly dropped from the shipped macro since it never appears in real data.🟡 Minor
"all_applications_from_another_LA","offers_to_applicants_from_another_LA") by analogy with the existing FSM_eligible_percent precedent, but this casing was never actually verified against target-postgres's column-name folding (unlike Tasks 1/5/6/7, no diagnostic check confirmed it). Since local CI only runsdbt parse(no DB access), a casing mismatch would only surface as acolumn does not existfailure when the Airflow DAG runsdbt buildin production, breaking stg_ees_admissions/fact_admissions for that run.progress_8_bandingaccepted_values list ('Well above average' … 'Well below average') is asserted without having been diagnostically verified against real EESprogress8_bandingtext (Task 1 verified GPS/science, KS2 subjects, and Ofsted rc columns, but not this field). It's mitigated byseverity: warnso it won't fail CI, but the labels could be wrong until someone checks the actual data.