diff --git a/docs/superpowers/plans/2026-07-12-compare-data-foundation.md b/docs/superpowers/plans/2026-07-12-compare-data-foundation.md new file mode 100644 index 0000000..30a3bb8 --- /dev/null +++ b/docs/superpowers/plans/2026-07-12-compare-data-foundation.md @@ -0,0 +1,591 @@ +# Compare-Screen Data Foundation (Pipeline PR) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Land every pipeline/dbt change the compare-screen redesign needs (spec §5 + §8 of `docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md`): promote raw-but-unstored fields to marts, close the national-averages gaps, and wire the Ofsted report-card columns. + +**Architecture:** Meltano Singer taps load `raw.*` tables; dbt builds `staging` → `marts` (read-only for the backend). All changes here are additive columns/rows — no breaking changes to existing marts. The full `dbt build` runs on the server via the Airflow DAGs; locally we gate with `dbt parse` (no DB needed) plus network-only diagnostic scripts. + +**Tech Stack:** Python (Singer SDK taps), dbt-postgres ~1.10 (invoked as `python -m dbt.cli.main`), Meltano, PostgreSQL. + +## Global Constraints + +- **No new external sources** (spec §5): only fields already in the `raw` schema or in files the taps already download. The one sanctioned tap change is the Ofsted MI report-card columns (spec §5, §8.4) and the legacy-KS2 year addition (same DfE performance-tables source). +- **Additive only:** never rename or drop existing mart columns; the backend maps them 1:1 in `backend/models.py`. +- **Never push to `main`.** Branch: `feat/compare-data-foundation`; PR checks must pass. +- Backend `models.py` changes belong to the follow-up backend PR, not this one. +- dbt invocation is always `python -m dbt.cli.main` (a bare `dbt` resolves to the wrong binary — see `pipeline/dags/school_data_pipeline.py:27`). +- EES suppression codes `z`/`c`/`x` must go through the `safe_numeric` macro. +- Computed benchmarks (FSM/EAL/SEN medians, disadvantaged national average) are **backend work** (spec §5) — explicitly out of scope here. + +--- + +### Task 0: Create the branch + +**Files:** none + +- [ ] **Step 1:** `git checkout main && git pull && git checkout -b feat/compare-data-foundation` + +--- + +### Task 1: Diagnostics — pin the three unknowns + +The spec flags three facts we must confirm from the actual files before wiring code: (a) why `gps_expected_pct`/`science_expected_pct` are NULL in `marts.fact_ks2_national_averages` despite being mapped end-to-end; (b) what the KS2 attainment long file calls its subjects/years for 2021/22 and 2022/23 (subject-level 2022/23 is NULL in prod; school-level 2021/22 is absent); (c) the exact report-card column headers in the current Ofsted MI CSV. + +**Files:** +- Create: `pipeline/scripts/diagnose_compare_gaps.py` + +**Interfaces:** +- Produces: a printed findings report; Tasks 5, 6, 7 consume the confirmed column/label names. Precedent: `pipeline/scripts/diagnose_ees_ks4.py`. + +- [ ] **Step 1: Write the diagnostic script** + +```python +"""Diagnose the three data gaps blocking the compare-screen redesign. + +Run from repo root (network access required, no DB needed): + python pipeline/scripts/diagnose_compare_gaps.py +""" +import io +import re +import sys +import zipfile + +import pandas as pd +import requests + +sys.path.insert(0, "pipeline/plugins/extractors/tap-uk-ees") +sys.path.insert(0, "pipeline/plugins/extractors/tap-uk-ofsted") +from tap_uk_ees.tap import ( # noqa: E402 + _KS2_NATIONAL_COL_MAP, + _KS2_NATIONAL_CSV_URL, + download_release_zip, + get_all_releases, +) +from tap_uk_ofsted.tap import discover_csv_url # noqa: E402 + +TIMEOUT = 120 + + +def check_national_gps_science(): + print("\n=== (a) National catalogue CSV: GPS/science columns ===") + resp = requests.get(_KS2_NATIONAL_CSV_URL, timeout=TIMEOUT) + resp.raise_for_status() + df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False) + df.columns = [c.strip().lower() for c in df.columns] + for csv_col in ("pt_gps_exp", "pt_scita_exp", "avg_readscore", "avg_matscore", "avg_gpsscore"): + status = "PRESENT" if csv_col in df.columns else "MISSING" + print(f" {csv_col}: {status}") + gps_like = [c for c in df.columns if "gps" in c or "scita" in c or "sci" in c] + print(f" all gps/science-ish columns: {gps_like}") + nat = df[df.get("geographic_level", "").str.strip().str.lower() == "national"] + print(f" national rows time_periods: {sorted(nat['time_period'].unique())}") + # Sample the values our map would read for the latest year + latest = nat[nat["time_period"] == nat["time_period"].max()] + for csv_col, field in _KS2_NATIONAL_COL_MAP.items(): + val = latest.iloc[0].get(csv_col, "") if len(latest) else "" + print(f" {field} <- {csv_col} = {val!r}") + + +def check_ks2_attainment_years_subjects(): + print("\n=== (b) EES KS2 attainment: years & subject labels ===") + releases = get_all_releases("key-stage-2-attainment") + print(f" releases found: {[r['time_period'] for r in releases]}") + for release in releases: + zf = download_release_zip(release["id"]) + name = next((n for n in zf.namelist() + if "ks2_school_attainment_data" in n and n.endswith(".csv")), None) + if not name: + print(f" {release['time_period']}: NO school attainment CSV in ZIP") + continue + with zf.open(name) as f: + df = pd.read_csv(f, dtype=str, keep_default_na=False, nrows=200000) + years = sorted(df["time_period"].unique()) + subjects = sorted(df["subject"].unique()) + print(f" release {release['time_period']}: time_periods={years}") + print(f" subjects={subjects}") + + +def check_ofsted_report_card_columns(): + print("\n=== (c) Ofsted MI CSV: report-card columns ===") + url = discover_csv_url() + print(f" MI file: {url}") + resp = requests.get(url, timeout=TIMEOUT) + resp.raise_for_status() + df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False, nrows=5) + rc_like = [c for c in df.columns + if re.search(r"report card|inclusion|curriculum|achievement|safeguard|well.?being|governance", c, re.I)] + print(f" candidate report-card columns ({len(rc_like)}):") + for c in rc_like: + print(f" - {c!r}") + + +if __name__ == "__main__": + check_national_gps_science() + check_ks2_attainment_years_subjects() + check_ofsted_report_card_columns() +``` + +Note: if `_KS2_NATIONAL_CSV_URL` is named differently in `tap_uk_ees/tap.py` (it is defined near the `_KS2_NATIONAL_COL_MAP` around line ~490), import whatever constant holds the catalogue CSV URL. + +- [ ] **Step 2: Run it and record findings** + +Run: `python pipeline/scripts/diagnose_compare_gaps.py 2>&1 | tee /tmp/compare-gaps-findings.txt` +Expected: three sections printed. Paste the findings as a comment block at the bottom of the script (so they're committed evidence), e.g. `# FINDINGS 2026-07-12: pt_gps_exp MISSING (actual col: ...), 202122 present in release X, rc columns: [...]`. + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/scripts/diagnose_compare_gaps.py +git commit -m "chore(pipeline): diagnostic for compare-screen data gaps" +``` + +--- + +### Task 2: Admissions preference detail → mart + +Staging already extracts `second_preference_offers`, `third_preference_offers`, `total_offers` (`stg_ees_admissions.sql:26-29`) — the mart drops them. The cross-LA fields are declared in the tap (`all_applications_from_another_LA`, `offers_to_applicants_from_another_LA`) but not selected in staging. + +**Files:** +- Modify: `pipeline/transform/models/staging/stg_ees_admissions.sql` (after line 33, in `renamed`) +- Modify: `pipeline/transform/models/marts/fact_admissions.sql` +- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_admissions block, ~line 120) + +**Interfaces:** +- Produces mart columns: `total_offers int`, `second_preference_offers int`, `third_preference_offers int`, `cross_la_applications int`, `cross_la_offers int`. The backend PR will map these in `FactAdmissions`. + +- [ ] **Step 1: Add cross-LA columns to staging** + +In `stg_ees_admissions.sql`, after the `first_preference_applications` line (line 33): + +```sql + -- Cross-borough demand: applications naming this school from families + -- living in another local authority, and offers made to them. + {{ safe_numeric('"all_applications_from_another_LA"') }}::integer as cross_la_applications, + {{ safe_numeric('"offers_to_applicants_from_another_LA"') }}::integer as cross_la_offers, +``` + +(Quote the identifiers — the tap emits them with mixed case, same trap as `FSM_eligible_percent`, see the header comment in that file. If `dbt parse` or the DAG run later shows the raw columns are lower-cased in Postgres, drop the double quotes.) + +- [ ] **Step 2: Pass everything through the mart** + +Replace the full select list in `fact_admissions.sql`: + +```sql +-- Mart: School admissions — one row per URN per year + +select + urn, + year, + school_phase, + places_offered, + total_offers, + total_applications, + first_preference_applications, + first_preference_offers, + second_preference_offers, + third_preference_offers, + cross_la_applications, + cross_la_offers, + first_preference_offer_pct, + oversubscription_ratio, + oversubscribed, + admissions_policy +from {{ ref('stg_ees_admissions') }} +``` + +- [ ] **Step 3: Add schema tests** + +In `_marts_schema.yml` under `fact_admissions.columns`, append: + +```yaml + - name: second_preference_offers + - name: third_preference_offers + - name: cross_la_applications + - name: cross_la_offers + - name: total_offers +``` + +- [ ] **Step 4: Parse gate** + +Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` +Expected: `Done.` with no compilation errors. + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/staging/stg_ees_admissions.sql pipeline/transform/models/marts/fact_admissions.sql pipeline/transform/models/marts/_marts_schema.yml +git commit -m "feat(pipeline): admissions preference breakdown and cross-LA demand in marts" +``` + +--- + +### Task 3: KS2 progress confidence intervals + writing working-towards + +The tap already emits `progress_measure_lower_conf_interval`, `progress_measure_upper_conf_interval`, `working_towards_expected_standard_pupil_percent` (tap.py:203-206). The staging pivot drops them. These power the CI-based Above/Average/Below progress chips (spec §8, first-review item on statistical honesty). + +**Files:** +- Modify: `pipeline/transform/models/staging/stg_ees_ks2.sql` (inside the `pivoted` CTE, next to each subject's `progress_measure_score` case, lines ~41/55/72, and in the final select ~lines 145-152) +- Modify: `pipeline/transform/models/marts/fact_ks2_performance.sql` +- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_ks2_performance block, ~line 82) + +**Interfaces:** +- Produces mart columns: `reading_progress_lower_ci`, `reading_progress_upper_ci`, `writing_progress_lower_ci`, `writing_progress_upper_ci`, `maths_progress_lower_ci`, `maths_progress_upper_ci` (float), `writing_working_towards_pct` (float). + +- [ ] **Step 1: Add pivot cases in staging** + +After the `reading_progress` case (line ~41), add: + +```sql + max(case when subject = 'Reading' + and breakdown_topic = 'All pupils' and breakdown = 'Total' + then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as reading_progress_lower_ci, + max(case when subject = 'Reading' + and breakdown_topic = 'All pupils' and breakdown = 'Total' + then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as reading_progress_upper_ci, +``` + +After the `writing_progress` case (line ~55), add: + +```sql + max(case when subject = 'Writing' + and breakdown_topic = 'All pupils' and breakdown = 'Total' + then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as writing_progress_lower_ci, + max(case when subject = 'Writing' + and breakdown_topic = 'All pupils' and breakdown = 'Total' + then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as writing_progress_upper_ci, + max(case when subject = 'Writing' + and breakdown_topic = 'All pupils' and breakdown = 'Total' + then {{ safe_numeric('working_towards_expected_standard_pupil_percent') }} end) as writing_working_towards_pct, +``` + +After the `maths_progress` case (line ~72), add: + +```sql + max(case when subject = 'Maths' + and breakdown_topic = 'All pupils' and breakdown = 'Total' + then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as maths_progress_lower_ci, + max(case when subject = 'Maths' + and breakdown_topic = 'All pupils' and breakdown = 'Total' + then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as maths_progress_upper_ci, +``` + +Then add the seven new columns to the model's final select (next to the existing `p.reading_progress` / `p.writing_progress` / `p.maths_progress` lines ~145-152): + +```sql + p.reading_progress_lower_ci, + p.reading_progress_upper_ci, + p.writing_progress_lower_ci, + p.writing_progress_upper_ci, + p.writing_working_towards_pct, + p.maths_progress_lower_ci, + p.maths_progress_upper_ci, +``` + +- [ ] **Step 2: Pass through the mart** + +In `fact_ks2_performance.sql`, add the same seven column names to the select list immediately after the existing `maths_progress` line (this mart selects staging columns by name; match the file's existing alias style — if columns are selected bare, add them bare). + +- [ ] **Step 3: Schema tests** + +In `_marts_schema.yml` under `fact_ks2_performance.columns`, append the seven names (no tests beyond presence — values are legitimately NULL for 2023/24+ since progress measures ended with 2022/23, spec §4.3): + +```yaml + - name: reading_progress_lower_ci + - name: reading_progress_upper_ci + - name: writing_progress_lower_ci + - name: writing_progress_upper_ci + - name: writing_working_towards_pct + - name: maths_progress_lower_ci + - name: maths_progress_upper_ci +``` + +- [ ] **Step 4: Parse gate** + +Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` +Expected: `Done.` + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/staging/stg_ees_ks2.sql pipeline/transform/models/marts/fact_ks2_performance.sql pipeline/transform/models/marts/_marts_schema.yml +git commit -m "feat(pipeline): KS2 progress confidence intervals and writing working-towards" +``` + +--- + +### Task 4: KS4 — Progress 8 banding and disadvantage gaps + +The tap's `ees_ks4_info` stream already declares `progress8_banding` (DfE's own "well above average … well below average" label — the ready-made secondary chip), `attainment8_diffn` and `progress8_diffn` (tap.py:338-340). Wire them through staging into the mart. + +**Files:** +- Modify: `pipeline/transform/models/staging/stg_ees_ks4.sql` (the CTE that reads `ees_ks4_info` — the same one that already surfaces `sen_pct`; add three columns to its select and to the final joined select) +- Modify: `pipeline/transform/models/marts/fact_ks4_performance.sql` (add after `progress_8_upper_ci`) +- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_ks4_performance block, ~line 93) + +**Interfaces:** +- Produces mart columns: `progress_8_banding text`, `attainment_8_disadvantage_gap float`, `progress_8_disadvantage_gap float`. + +- [ ] **Step 1: Staging — select from the info source** + +In the info CTE of `stg_ees_ks4.sql` add: + +```sql + nullif(trim(progress8_banding), '') as progress_8_banding, + {{ safe_numeric('attainment8_diffn') }} as attainment_8_disadvantage_gap, + {{ safe_numeric('progress8_diffn') }} as progress_8_disadvantage_gap, +``` + +and add the three names to the model's final select (aliased the same way the CTE's other columns are). + +- [ ] **Step 2: Mart passthrough** + +In `fact_ks4_performance.sql`, after the `progress_8_upper_ci,` line: + +```sql + progress_8_banding, + attainment_8_disadvantage_gap, + progress_8_disadvantage_gap, +``` + +- [ ] **Step 3: Schema tests** — append the three names under `fact_ks4_performance.columns`, plus an accepted-values guard that tolerates NULL: + +```yaml + - name: progress_8_banding + tests: + - accepted_values: + values: ['Well above average', 'Above average', 'Average', 'Below average', 'Well below average'] + config: + where: "progress_8_banding is not null" + - name: attainment_8_disadvantage_gap + - name: progress_8_disadvantage_gap +``` + +(If the DAG run later shows different capitalisation in the data, fix the accepted values to match the data, not vice versa.) + +- [ ] **Step 4: Parse gate** — `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.` + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/staging/stg_ees_ks4.sql pipeline/transform/models/marts/fact_ks4_performance.sql pipeline/transform/models/marts/_marts_schema.yml +git commit -m "feat(pipeline): Progress 8 banding and KS4 disadvantage gaps in marts" +``` + +--- + +### Task 5: National averages — 2015/16 row and GPS/science/scaled-score fix + +Two changes. (1) `stg_ees_ks2_national.sql:34` filters `>= 201617`, which is exactly why the England line starts a year late (2015/16 RWM = 53% exists in the catalogue). (2) GPS/science expected are NULL in prod despite full end-to-end mapping — Task 1's findings say whether the catalogue CSV column names differ from `_KS2_NATIONAL_COL_MAP` (`pt_gps_exp`, `pt_scita_exp`) or whether values are suppressed at source. + +**Files:** +- Modify: `pipeline/transform/models/staging/stg_ees_ks2_national.sql:34` +- Modify (conditional on Task 1 findings): `pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py` (`_KS2_NATIONAL_COL_MAP`) + +**Interfaces:** +- Produces: a 201516 row in `marts.fact_ks2_national_averages`; non-NULL `gps_expected_pct`, `science_expected_pct`, `reading_avg_score`, `maths_avg_score`, `gps_avg_score` for years the DfE publishes them. Backend/frontend consume via `/api/national-averages` unchanged (additive year + newly non-NULL fields). + +- [ ] **Step 1: Widen the year filter** + +In `stg_ees_ks2_national.sql`, change line 34: + +```sql + and cast(trim(time_period) as integer) >= 201516 +``` + +(2015/16 was the first year of the current expected-standard tests; nothing earlier is comparable, so keep a floor.) + +- [ ] **Step 2: Fix the column map per Task 1 findings** + +If Task 1 reported the actual CSV column names for GPS/science/scaled scores differ, update `_KS2_NATIONAL_COL_MAP` in `tap.py` accordingly, e.g. (illustrative — use the diagnosed names): + +```python +_KS2_NATIONAL_COL_MAP = { + # ... existing entries ... + "pt_gps_exp": "gps_expected_pct", # replace key with diagnosed name + "pt_scita_exp": "science_expected_pct", # replace key with diagnosed name +} +``` + +If Task 1 showed the columns are present but suppressed (`x`) at national level for all years, instead delete the two entries from the map, delete the corresponding lines from `stg_ees_ks2_national.sql` and `fact_ks2_national_averages.sql`, and record in the PR description that GPS/science England ticks stay "not in dataset" (the mockups already carry that caveat). + +- [ ] **Step 3: Parse gate** — `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.` + +- [ ] **Step 4: Commit** + +```bash +git add pipeline/transform/models/staging/stg_ees_ks2_national.sql pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py +git commit -m "fix(pipeline): include 2015/16 national averages; fix GPS/science national mapping" +``` + +--- + +### Task 6: Legacy KS2 — load the 2021/22 school-level year + +School-level 2021/22 exists in DfE performance-tables archives (same source as the four legacy years already loaded) but in neither our legacy config (stops at 201819, `pipeline/meltano.yml:33-37`) nor EES (starts 2022/23) — unless Task 1's finding (b) showed an EES release carrying 202122, in which case skip this task and note why in the PR. + +The legacy URLs point at the self-hosted filebrowser (`10.0.1.224:8081`) — **the 2021/22 DfE archive must be uploaded there first; this is the one human dependency in this plan.** + +**Files:** +- Modify: `pipeline/meltano.yml` (legacy_ks2_urls block, line ~33) + +**Interfaces:** +- Produces: `raw.legacy_ks2` rows with `year = '202122'`, flowing through `stg_legacy_ks2` → `fact_ks2_performance` unchanged (the stream maps old column names already; 2021/22 CSVs use the same `PTRWM_EXP`-style headers as 2018/19). + +- [ ] **Step 1: Verify the 2021/22 CSV headers match `_LEGACY_KS2_COLUMN_MAP`** + +Download the DfE 2021/22 KS2 revised archive (gov.uk "Compare School Performance data download": 2021-2022 all-schools ZIP), then: + +Run: `python -c "import zipfile,io,pandas as pd; zf=zipfile.ZipFile('/path/to/2021-2022.zip'); n=[x for x in zf.namelist() if 'ks2final' in x.lower() and x.endswith('.csv')][0]; df=pd.read_csv(zf.open(n), dtype=str, nrows=5); import sys; sys.path.insert(0,'pipeline/plugins/extractors/tap-uk-ees'); from tap_uk_ees.tap import _LEGACY_KS2_COLUMN_MAP as m; missing=[c for c in m if c not in df.columns]; print('missing legacy columns:', missing)"` +Expected: `missing legacy columns: []` (progress columns `READPROG` etc. may legitimately be missing/blank in 2021/22 — acceptable, they load as NULL). + +- [ ] **Step 2: Upload the archive to the filebrowser and add the config entry** + +In `pipeline/meltano.yml` under `legacy_ks2_urls`, add (with the real share URL from the filebrowser upload): + +```yaml + "202122": "http://10.0.1.224:8081/filebrowser/api/public/dl/?inline=true" +``` + +- [ ] **Step 3: Commit** + +```bash +git add pipeline/meltano.yml +git commit -m "feat(pipeline): load 2021/22 school-level KS2 from legacy performance tables" +``` + +- [ ] **Step 4 (only if Task 1(b) showed 2022/23 subject labels differ):** widen the subject matchers in `stg_ees_ks2.sql` the same way GPS already is (`subject ilike '%grammar%' or subject = 'GPS'`), e.g. `subject in ('Reading', 'reading')` → use the diagnosed labels. Parse-gate and commit as `fix(pipeline): match 2022/23 KS2 subject labels`. + +--- + +### Task 7: Ofsted report-card columns (rc_*) + +Resolves the tap TODO (`stg_ofsted_inspections.sql:37`). The marts/backed columns already exist as stubs; this wires real values. Uses Task 1(c)'s confirmed MI column names — the candidates below follow the MI file's existing naming style and must be corrected against the diagnostic output. + +**Files:** +- Modify: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (COLUMN_PRIORITY ~line 19-72, schema ~line 100-114) +- Create: `pipeline/transform/macros/parse_report_card_grade.sql` +- Modify: `pipeline/transform/models/staging/stg_ofsted_inspections.sql:36-46` + +**Interfaces:** +- Produces mart columns (already declared in `fact_ofsted_inspection`): `rc_safeguarding_met boolean`, and `rc_inclusion` … `rc_sixth_form` as integers on the 5-point scale `1=Exceptional, 2=Strong standard, 3=Expected standard, 4=Needs attention/Attention needed, 5=Urgent improvement`. The backend translates codes to labels (same pattern as `gias_codes.py`), verifying wording against Ofsted's published toolkit (spec §8.4). + +- [ ] **Step 1: Add tap column mappings** + +In `COLUMN_PRIORITY` add (replace candidate strings with Task 1(c)'s exact headers — keep them as priority lists so older files degrade to blank): + +```python + "rc_safeguarding_met": ["Report card safeguarding", "Safeguarding"], + "rc_inclusion": ["Report card inclusion", "Inclusion"], + "rc_curriculum_teaching": ["Report card curriculum and teaching", "Curriculum and teaching"], + "rc_achievement": ["Report card achievement", "Achievement"], + "rc_attendance_behaviour": ["Report card attendance and behaviour", "Attendance and behaviour"], + "rc_personal_development": ["Report card personal development and well-being", "Personal development and well-being"], + "rc_leadership_governance": ["Report card leadership and governance", "Leadership and governance"], + "rc_early_years": ["Report card early years", "Early years"], + "rc_sixth_form": ["Report card sixth form", "Sixth form"], +``` + +And in the stream schema (next to `report_url`, ~line 114): + +```python + th.Property("rc_safeguarding_met", th.StringType), + th.Property("rc_inclusion", th.StringType), + th.Property("rc_curriculum_teaching", th.StringType), + th.Property("rc_achievement", th.StringType), + th.Property("rc_attendance_behaviour", th.StringType), + th.Property("rc_personal_development", th.StringType), + th.Property("rc_leadership_governance", th.StringType), + th.Property("rc_early_years", th.StringType), + th.Property("rc_sixth_form", th.StringType), +``` + +- [ ] **Step 2: Write the grade-parsing macro** + +`pipeline/transform/macros/parse_report_card_grade.sql`: + +```sql +{% macro parse_report_card_grade(column_name) %} + case lower(trim(nullif({{ column_name }}, 'NULL'))) + when 'exceptional' then 1 + when 'strong standard' then 2 + when 'expected standard' then 3 + when 'needs attention' then 4 + when 'attention needed' then 4 + when 'urgent improvement' then 5 + end +{% endmacro %} +``` + +- [ ] **Step 3: Wire staging** + +Replace `stg_ofsted_inspections.sql` lines 36-46 (the NULL stubs) with: + +```sql + -- Report Card fields (post-Nov 2025 framework), 5-point scale: + -- 1 Exceptional · 2 Strong standard · 3 Expected standard + -- · 4 Needs attention · 5 Urgent improvement + (lower(trim(nullif(rc_safeguarding_met, 'NULL'))) = 'met') as rc_safeguarding_met, + {{ parse_report_card_grade('rc_inclusion') }}::integer as rc_inclusion, + {{ parse_report_card_grade('rc_curriculum_teaching') }}::integer as rc_curriculum_teaching, + {{ parse_report_card_grade('rc_achievement') }}::integer as rc_achievement, + {{ parse_report_card_grade('rc_attendance_behaviour') }}::integer as rc_attendance_behaviour, + {{ parse_report_card_grade('rc_personal_development') }}::integer as rc_personal_development, + {{ parse_report_card_grade('rc_leadership_governance') }}::integer as rc_leadership_governance, + {{ parse_report_card_grade('rc_early_years') }}::integer as rc_early_years, + {{ parse_report_card_grade('rc_sixth_form') }}::integer as rc_sixth_form, +``` + +Note `rc_safeguarding_met` becomes boolean (NULL when blank) — matching `fact_ofsted_inspection`'s `rc_safeguarding_met` Boolean column. If `fact_ofsted_inspection.sql` casts these columns, align its casts too (inspect that model; it currently passes the text stubs through). + +- [ ] **Step 4: Parse gate + tap smoke test** + +Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.` +Run: `python -c "import sys; sys.path.insert(0,'pipeline/plugins/extractors/tap-uk-ofsted'); from tap_uk_ofsted.tap import COLUMN_PRIORITY; assert 'rc_inclusion' in COLUMN_PRIORITY; print('ok')"` → `ok` + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py pipeline/transform/macros/parse_report_card_grade.sql pipeline/transform/models/staging/stg_ofsted_inspections.sql +git commit -m "feat(pipeline): extract Ofsted report-card judgements (rc_* columns)" +``` + +--- + +### Task 8: PR + post-merge verification + +**Files:** none new + +- [ ] **Step 1: Push and open the PR** (Gitea — use the git credential helper + basic-auth API pattern; token-header auth 401s): + +```bash +git push -u origin feat/compare-data-foundation +# then create the PR via the Gitea API with basic auth from `git credential fill` +``` + +PR body: link spec §5/§8, list the new mart columns, note the Task 6 human dependency (filebrowser upload) and the Task 1 findings file. + +- [ ] **Step 2: After merge, verify the DAG run picked everything up** + +The daily/monthly DAGs rebuild the affected models (`pipeline/dags/school_data_pipeline.py`). Spot-check via the public API (production after promotion, staging first at stx.schoolcompare.co.uk — note external /api is broken at the staging proxy, so check staging from the host): + +```bash +# 2015/16 national row exists +curl -sL "https://www.schoolcompare.co.uk/api/national-averages" | python3 -c "import json,sys; d=json.load(sys.stdin); assert any(r['year']==201516 and r['primary'] for r in d['by_year']), '2015/16 missing'; print('201516 ok')" +# 2021/22 school rows exist (Barclay) +curl -sL "https://www.schoolcompare.co.uk/api/schools/138690" | python3 -c "import json,sys; d=json.load(sys.stdin); ys=[r['year'] for r in d['yearly_data']]; assert 202122 in [int(y) for y in ys], ys; print('202122 ok')" +``` + +(The admissions/CI/KS4/rc_* columns aren't API-visible until the backend PR maps them — verify those directly in Postgres from the pipeline host: `select count(*) from marts.fact_admissions where second_preference_offers is not null;` etc.) + +- [ ] **Step 3: Update the spec** — tick off the §5 promotions this PR delivered (edit the spec's promotion list to note "landed in PR #NN") and commit to main via a docs PR or alongside the backend PR. + +--- + +## Out of scope (next plans) + +1. **Backend PR:** map new columns in `backend/models.py`, extend `/api/compare` with supplementary blocks + `national_averages`, computed benchmarks (FSM/EAL/SEN/size medians, disadvantaged national average), CI-based progress banding, report-card label translation (verify against Ofsted toolkit), Ofsted provider-page URLs, graded-vs-ungraded surfacing. +2. **Frontend PR:** rebuild `/compare` per the mockups + e2e journeys (promotion gate). +3. **Separate bug fix:** third school's series not rendering on the current production chart. +4. **Post-v1 (spec):** census ethnicity/young-carer promotion, IDACI display, attendance section, gender-split/absence tier-2 measures. +5. **Already in marts, no work needed:** KS4 EBacc entry/APS, grade 5+ English & maths, Progress 8 CIs — `fact_ks4_performance` carries them today; only the backend needs to expose them. diff --git a/docs/superpowers/specs/2026-07-11-compare-screen-expert-review.md b/docs/superpowers/specs/2026-07-11-compare-screen-expert-review.md new file mode 100644 index 0000000..d67821d --- /dev/null +++ b/docs/superpowers/specs/2026-07-11-compare-screen-expert-review.md @@ -0,0 +1,177 @@ +# Compare Screen Redesign — Expert Data Review + +**Date:** 2026-07-11 +**Reviewer:** subagent briefed as an English education-standards / DfE-Ofsted data expert +**Subject:** desktop + mobile compare mockups and the redesign spec +(`2026-07-11-compare-screen-redesign-design.md`) +**Status:** first-pass must-fixes applied 2026-07-12; second-pass +findings (below) applied 2026-07-12 — mockups + spec §4/§8 updated + +## Must-fix + +1. **COVID gap is wrong and drops a real results year.** KS2 tests were + cancelled 2019/20 and 2020/21 only; they resumed in 2021/22 with + published school-level results (England RWM ≈ 59%). The mockup charts + omit 2021/22 entirely and the tooltip claims no tests were held + 2019/20–2021/22. Fix: add 2021/22 to axis and all series; shrink the + gap band; optionally annotate 2021/22 with DfE's post-pandemic + comparability caution. +2. **Report-card at-a-glance summary miscounts areas.** Detail list has + 4 Strong / 2 Expected / 1 Attention needed + Safeguarding met, but + the summary says "3 areas Expected standard" — it counts safeguarding + as a graded area. Safeguarding is a separate binary judgement and + must be excluded from rating counts. +3. **"Where the offers went" derivation is unsound.** Places − 1st-pref + offers ≠ "second or third choices": the residual can include 4th–6th + preference offers (pan-London scheme) and LA-allocated children who + didn't choose the school; and offers don't necessarily equal PAN. + Use the real 2nd/3rd-preference fields being promoted from + `raw.ees_admissions`; until then drop the row. +4. **Ofsted timeline in the copy is wrong.** Overall grades were + abolished September 2024, not November 2025; Sept 2024–Nov 2025 + inspections kept the four key judgements without an overall grade + (ungraded inspections carried grades forward). Neither mockup shows + the interim regime, which will dominate real comparisons. Fix copy + and add an interim example. +5. **Barclay's "published an overall grade only — no area-by-area + detail" misdescribes inspections.** No inspection type does that; a + 2021 graded inspection necessarily had subgrades — the gap is in our + dataset. If it was an ungraded (s8) inspection, "Outstanding" is a + carried-forward grade and should say so. Fix: "We don't hold + area-by-area detail for this inspection", and distinguish graded vs + ungraded in the data model. + +## Should-fix + +6. Writing is teacher assessment, not a test — "national tests and + teacher assessments"; note TA caveat on the Writing strip. +7. Verify renewed-framework wording against Ofsted's final toolkit: + likely "Needs attention" (not "Attention needed") and "Personal + development and well-being" (which otherwise collides with the + identically-named legacy judgement). Pin every label to the + published toolkit. +8. "Expected standard" now means two things on one page (Ofsted area + rating vs KS2 measure) — disambiguate in tooltips. +9. Disadvantaged row: DfE definition includes looked-after / previously + looked-after children, not just FSM6; benchmark labels inconsistent + across desktop/mobile; subgroup percentages need cohort sizes or a + volatility threshold before chips are attached. +10. "Trend, last 7 years" spans ten years; sparklines render the COVID + gap as equal spacing (the exact defect the audit criticises) and + "Improved: 52% → 87%" endpoint-cherry-picks a volatile series. +11. At-a-glance "Getting a place" uses different metrics per school + (Barclay is also oversubscribed on total preferences but shows a + green chip). Standardise on first-preference success %. Explain the + equal-preference rule; condition "living close by matters" on the + school's actual oversubscription criteria. +12. "457 applications for 180 places" = total preferences at any rank, + not head-to-head applicants; lead with first preferences vs places. + Add offers-vs-final-intake (waiting lists/appeals) caveat. +13. Elmhurst's subgrade list is likely missing Early years provision + (school has a nursery) — possible pipeline gap. +14. "Ofsted rating" label is obsolete post-Sept-2024 — use "Latest + Ofsted inspection"; check whether Oct 2021 is the latest inspection + or merely the latest graded one. +15. SEN: "EHCP plans" is redundant; 28% SEN support often indicates + resourced provision — add a note; England SEN-support ≈ 14%, not 13%. + +## Nice-to-have + +16. Consistent labelling of official DfE vs dataset-computed benchmarks + (and medians shouldn't be called averages inconsistently). +17. England 2015/16 RWM (53%) exists in DfE publications — the null is + a dataset gap; source it or the England line looks broken. +18. "1 in 4 first choices missed out" — actually more than 1 in 4. +19. "1,273 of 1,260 places (full)" is over capacity; capacity figures + are often stale — say "at or above capacity". +20. State the actual suppression rule (DfE: ≤5 pupils suppressed, + small numbers rounded) instead of "a handful". +21. Spec §4.3 progress chips can't exist for displayed years: KS2 + progress ended with 2022/23 (no KS1 baseline) and returns + ~2027/28 with the reception baseline. Make explicit in the spec. + IDACI (spec §4.5) is absent from mockups; if shipped, caveat it + describes pupils' neighbourhoods, not the school. +22. Tooltips should give the official term "first preference" alongside + the plain-English "first choice". + +## Overall assessment (verbatim gist) + +The bones are genuinely good by education-data standards — +England-average anchoring, explicit non-comparability messaging across +Ofsted regimes, refusal to synthesise an overall grade, time-true +x-axis, neutral FSM/EAL framing — better than most commercial +school-comparison sites. But items 1–5 are outright factual errors or +misdescriptions that a well-informed parent or Ofsted would catch; +the admissions section needs the most conceptual work (equal +preference, preferences-vs-applicants, offers-vs-intake). Fix 1–5 +before user testing; the rest fold into the planned PRs. + +--- + +# Second-pass review (2026-07-12) + +Same reviewer, after the must-fixes and the new three-tier metric +exposure model were applied. + +## Verification of first-pass must-fixes + +- **1 (COVID/2021/22): resolved.** Time-true axis, band covers only the + cancelled years, England 58.7% consistent with official figures, + dataset gaps break lines honestly; reading/maths England series all + match published figures; RWM ≤ min(subject) checks pass. +- **2 (report-card count): resolved** — safeguarding excluded, spec §8.2. +- **3 (offers derivation): resolved** — row removed, spec §8.3 bans it. +- **4 (Ofsted timeline): resolved on desktop; mobile omits the interim + regime clause** (see finding 6). +- **5 (Barclay explanation): resolved.** + +## New findings + +1. **Should-fix — scaled-score strip domain contradicts caption.** + Caption says "scaled scores run 80–120", strips render 100–120; + truncated domain exaggerates small gaps and below-100 averages + would fall off the edge. Render 80–120, or caption the 100–120 + window honestly and define below-100 behaviour. +2. **Should-fix — scaled-score England ticks (106/105/105) unsourced.** + Plausible but hand-entered; verify against DfE 2024/25 tables and + add loading official England scaled scores to the pipeline list + (absent from §8.1/§8.6). +3. **Should-fix — "Writing" listed under "Higher standard" in the + picker.** Writing TA outcome is "greater depth" (GDS), never + "higher standard". Label "Writing — greater depth (teacher + assessment)"; tooltip the combined higher-standard composition. +4. Nice — "grammar & punctuation" summary line drops "spelling" (GPS). +5. Nice — science is teacher-assessed (no KS2 test since 2009) and + coarse; tooltip it like writing; reconsider its tier-2 slot. +6. **Should-fix — mobile Ofsted copy skips the interim regime** + (Sept 2024–Nov 2025) that desktop explains. One clause fixes it. +7. **Should-fix — benchmark provenance still inconsistent** (EAL + tooltip unsourced; FSM/disadvantaged chips vs tooltips use three + vocabularies; header note says all England averages are official). + Adopt one house style: official = "England average", computed = + "benchmark / typical state school (our dataset)". Also tighten EAL + definition to census wording ("first language known or believed to + be other than English"). +8. Nice — "community primaries" distance note attached to an academy + (Elmhurst); say "non-faith primaries" or condition on policy field. +9. Nice — "Improving since 2022" → "since 2022/23". +10. Nice — England chart tooltips show decimals; §7 mandates whole + percents. + +## Residual gaps not covered by spec §8 + +11. Spec promises IDACI-in-words, Attendance section, and tier-2 + gender/absence that the mockups never show — mark post-v1 or + demonstrate, so implementation scope is unambiguous. +12. Add official England scaled-score averages to the pipeline task + list. +13. Add the writing/greater-depth terminology rule to §8.7. + +## Verdict + +All must-fixes genuinely resolved; the tier model is conceptually +sound ("no measure is lost", honest dataset-gap breaks, grouped +picker). Remaining issues are contained: one internal contradiction +(80–120 vs 100–120), one provenance inconsistency, one terminology +error (writing/GDS). With findings 1–3 and 6–7 addressed, the data +framing is fit to put in front of parents. diff --git a/docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md b/docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md new file mode 100644 index 0000000..f160c36 --- /dev/null +++ b/docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md @@ -0,0 +1,318 @@ +# Compare Screen Redesign — Audit & Design + +**Date:** 2026-07-11 +**Status:** Draft — awaiting review +**Scope:** `/compare` page (nextjs-app), `/api/compare` endpoint (backend) + +## 1. Audit of the current screen + +The current compare page (`nextjs-app/components/ComparisonView.tsx`) is a +single-metric analyst tool: a `