docs: compare-screen redesign spec, expert review, and data-foundation plan

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-12 21:19:53 +01:00
parent 58e90fef61
commit 297bdbd12e
3 changed files with 1086 additions and 0 deletions
@@ -0,0 +1,591 @@
 # Compare-Screen Data Foundation (Pipeline PR) Implementation Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Land every pipeline/dbt change the compare-screen redesign needs (spec §5 + §8 of `docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md`): promote raw-but-unstored fields to marts, close the national-averages gaps, and wire the Ofsted report-card columns.
 **Architecture:** Meltano Singer taps load `raw.*` tables; dbt builds `staging` → `marts` (read-only for the backend). All changes here are additive columns/rows — no breaking changes to existing marts. The full `dbt build` runs on the server via the Airflow DAGs; locally we gate with `dbt parse` (no DB needed) plus network-only diagnostic scripts.
 **Tech Stack:** Python (Singer SDK taps), dbt-postgres ~1.10 (invoked as `python -m dbt.cli.main`), Meltano, PostgreSQL.
 ## Global Constraints
 - **No new external sources** (spec §5): only fields already in the `raw` schema or in files the taps already download. The one sanctioned tap change is the Ofsted MI report-card columns (spec §5, §8.4) and the legacy-KS2 year addition (same DfE performance-tables source).
 - **Additive only:** never rename or drop existing mart columns; the backend maps them 1:1 in `backend/models.py`.
 - **Never push to `main`.** Branch: `feat/compare-data-foundation`; PR checks must pass.
 - Backend `models.py` changes belong to the follow-up backend PR, not this one.
 - dbt invocation is always `python -m dbt.cli.main` (a bare `dbt` resolves to the wrong binary — see `pipeline/dags/school_data_pipeline.py:27`).
 - EES suppression codes `z`/`c`/`x` must go through the `safe_numeric` macro.
 - Computed benchmarks (FSM/EAL/SEN medians, disadvantaged national average) are **backend work** (spec §5) — explicitly out of scope here.
 ---
 ### Task 0: Create the branch
 **Files:** none
 - [ ] **Step 1:** `git checkout main && git pull && git checkout -b feat/compare-data-foundation`
 ---
 ### Task 1: Diagnostics — pin the three unknowns
 The spec flags three facts we must confirm from the actual files before wiring code: (a) why `gps_expected_pct`/`science_expected_pct` are NULL in `marts.fact_ks2_national_averages` despite being mapped end-to-end; (b) what the KS2 attainment long file calls its subjects/years for 2021/22 and 2022/23 (subject-level 2022/23 is NULL in prod; school-level 2021/22 is absent); (c) the exact report-card column headers in the current Ofsted MI CSV.
 **Files:**
 - Create: `pipeline/scripts/diagnose_compare_gaps.py`
 **Interfaces:**
 - Produces: a printed findings report; Tasks 5, 6, 7 consume the confirmed column/label names. Precedent: `pipeline/scripts/diagnose_ees_ks4.py`.
 - [ ] **Step 1: Write the diagnostic script**
 ```python
 """Diagnose the three data gaps blocking the compare-screen redesign.
 Run from repo root (network access required, no DB needed):
    python pipeline/scripts/diagnose_compare_gaps.py
 """
 import io
 import re
 import sys
 import zipfile
 import pandas as pd
 import requests
 sys.path.insert(0, "pipeline/plugins/extractors/tap-uk-ees")
 sys.path.insert(0, "pipeline/plugins/extractors/tap-uk-ofsted")
 from tap_uk_ees.tap import (  # noqa: E402
    _KS2_NATIONAL_COL_MAP,
    _KS2_NATIONAL_CSV_URL,
    download_release_zip,
    get_all_releases,
 )
 from tap_uk_ofsted.tap import discover_csv_url  # noqa: E402
 TIMEOUT = 120
 def check_national_gps_science():
    print("\n=== (a) National catalogue CSV: GPS/science columns ===")
    resp = requests.get(_KS2_NATIONAL_CSV_URL, timeout=TIMEOUT)
    resp.raise_for_status()
    df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False)
    df.columns = [c.strip().lower() for c in df.columns]
    for csv_col in ("pt_gps_exp", "pt_scita_exp", "avg_readscore", "avg_matscore", "avg_gpsscore"):
        status = "PRESENT" if csv_col in df.columns else "MISSING"
        print(f"  {csv_col}: {status}")
    gps_like = [c for c in df.columns if "gps" in c or "scita" in c or "sci" in c]
    print(f"  all gps/science-ish columns: {gps_like}")
    nat = df[df.get("geographic_level", "").str.strip().str.lower() == "national"]
    print(f"  national rows time_periods: {sorted(nat['time_period'].unique())}")
    # Sample the values our map would read for the latest year
    latest = nat[nat["time_period"] == nat["time_period"].max()]
    for csv_col, field in _KS2_NATIONAL_COL_MAP.items():
        val = latest.iloc[0].get(csv_col, "<col missing>") if len(latest) else "<no row>"
        print(f"  {field} <- {csv_col} = {val!r}")
 def check_ks2_attainment_years_subjects():
    print("\n=== (b) EES KS2 attainment: years & subject labels ===")
    releases = get_all_releases("key-stage-2-attainment")
    print(f"  releases found: {[r['time_period'] for r in releases]}")
    for release in releases:
        zf = download_release_zip(release["id"])
        name = next((n for n in zf.namelist()
                     if "ks2_school_attainment_data" in n and n.endswith(".csv")), None)
        if not name:
            print(f"  {release['time_period']}: NO school attainment CSV in ZIP")
            continue
        with zf.open(name) as f:
            df = pd.read_csv(f, dtype=str, keep_default_na=False, nrows=200000)
        years = sorted(df["time_period"].unique())
        subjects = sorted(df["subject"].unique())
        print(f"  release {release['time_period']}: time_periods={years}")
        print(f"    subjects={subjects}")
 def check_ofsted_report_card_columns():
    print("\n=== (c) Ofsted MI CSV: report-card columns ===")
    url = discover_csv_url()
    print(f"  MI file: {url}")
    resp = requests.get(url, timeout=TIMEOUT)
    resp.raise_for_status()
    df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False, nrows=5)
    rc_like = [c for c in df.columns
               if re.search(r"report card|inclusion|curriculum|achievement|safeguard|well.?being|governance", c, re.I)]
    print(f"  candidate report-card columns ({len(rc_like)}):")
    for c in rc_like:
        print(f"    - {c!r}")
 if __name__ == "__main__":
    check_national_gps_science()
    check_ks2_attainment_years_subjects()
    check_ofsted_report_card_columns()
 ```
 Note: if `_KS2_NATIONAL_CSV_URL` is named differently in `tap_uk_ees/tap.py` (it is defined near the `_KS2_NATIONAL_COL_MAP` around line ~490), import whatever constant holds the catalogue CSV URL.
 - [ ] **Step 2: Run it and record findings**
 Run: `python pipeline/scripts/diagnose_compare_gaps.py 2>&1 | tee /tmp/compare-gaps-findings.txt`
 Expected: three sections printed. Paste the findings as a comment block at the bottom of the script (so they're committed evidence), e.g. `# FINDINGS 2026-07-12: pt_gps_exp MISSING (actual col: ...), 202122 present in release X, rc columns: [...]`.
 - [ ] **Step 3: Commit**
 ```bash
 git add pipeline/scripts/diagnose_compare_gaps.py
 git commit -m "chore(pipeline): diagnostic for compare-screen data gaps"
 ```
 ---
 ### Task 2: Admissions preference detail → mart
 Staging already extracts `second_preference_offers`, `third_preference_offers`, `total_offers` (`stg_ees_admissions.sql:26-29`) — the mart drops them. The cross-LA fields are declared in the tap (`all_applications_from_another_LA`, `offers_to_applicants_from_another_LA`) but not selected in staging.
 **Files:**
 - Modify: `pipeline/transform/models/staging/stg_ees_admissions.sql` (after line 33, in `renamed`)
 - Modify: `pipeline/transform/models/marts/fact_admissions.sql`
 - Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_admissions block, ~line 120)
 **Interfaces:**
 - Produces mart columns: `total_offers int`, `second_preference_offers int`, `third_preference_offers int`, `cross_la_applications int`, `cross_la_offers int`. The backend PR will map these in `FactAdmissions`.
 - [ ] **Step 1: Add cross-LA columns to staging**
 In `stg_ees_admissions.sql`, after the `first_preference_applications` line (line 33):
 ```sql
        -- Cross-borough demand: applications naming this school from families
        -- living in another local authority, and offers made to them.
        {{ safe_numeric('"all_applications_from_another_LA"') }}::integer as cross_la_applications,
        {{ safe_numeric('"offers_to_applicants_from_another_LA"') }}::integer as cross_la_offers,
 ```
 (Quote the identifiers — the tap emits them with mixed case, same trap as `FSM_eligible_percent`, see the header comment in that file. If `dbt parse` or the DAG run later shows the raw columns are lower-cased in Postgres, drop the double quotes.)
 - [ ] **Step 2: Pass everything through the mart**
 Replace the full select list in `fact_admissions.sql`:
 ```sql
 -- Mart: School admissions — one row per URN per year
 select
    urn,
    year,
    school_phase,
    places_offered,
    total_offers,
    total_applications,
    first_preference_applications,
    first_preference_offers,
    second_preference_offers,
    third_preference_offers,
    cross_la_applications,
    cross_la_offers,
    first_preference_offer_pct,
    oversubscription_ratio,
    oversubscribed,
    admissions_policy
 from {{ ref('stg_ees_admissions') }}
 ```
 - [ ] **Step 3: Add schema tests**
 In `_marts_schema.yml` under `fact_admissions.columns`, append:
 ```yaml
      - name: second_preference_offers
      - name: third_preference_offers
      - name: cross_la_applications
      - name: cross_la_offers
      - name: total_offers
 ```
 - [ ] **Step 4: Parse gate**
 Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .`
 Expected: `Done.` with no compilation errors.
 - [ ] **Step 5: Commit**
 ```bash
 git add pipeline/transform/models/staging/stg_ees_admissions.sql pipeline/transform/models/marts/fact_admissions.sql pipeline/transform/models/marts/_marts_schema.yml
 git commit -m "feat(pipeline): admissions preference breakdown and cross-LA demand in marts"
 ```
 ---
 ### Task 3: KS2 progress confidence intervals + writing working-towards
 The tap already emits `progress_measure_lower_conf_interval`, `progress_measure_upper_conf_interval`, `working_towards_expected_standard_pupil_percent` (tap.py:203-206). The staging pivot drops them. These power the CI-based Above/Average/Below progress chips (spec §8, first-review item on statistical honesty).
 **Files:**
 - Modify: `pipeline/transform/models/staging/stg_ees_ks2.sql` (inside the `pivoted` CTE, next to each subject's `progress_measure_score` case, lines ~41/55/72, and in the final select ~lines 145-152)
 - Modify: `pipeline/transform/models/marts/fact_ks2_performance.sql`
 - Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_ks2_performance block, ~line 82)
 **Interfaces:**
 - Produces mart columns: `reading_progress_lower_ci`, `reading_progress_upper_ci`, `writing_progress_lower_ci`, `writing_progress_upper_ci`, `maths_progress_lower_ci`, `maths_progress_upper_ci` (float), `writing_working_towards_pct` (float).
 - [ ] **Step 1: Add pivot cases in staging**
 After the `reading_progress` case (line ~41), add:
 ```sql
        max(case when subject = 'Reading'
                  and breakdown_topic = 'All pupils' and breakdown = 'Total'
            then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as reading_progress_lower_ci,
        max(case when subject = 'Reading'
                  and breakdown_topic = 'All pupils' and breakdown = 'Total'
            then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as reading_progress_upper_ci,
 ```
 After the `writing_progress` case (line ~55), add:
 ```sql
        max(case when subject = 'Writing'
                  and breakdown_topic = 'All pupils' and breakdown = 'Total'
            then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as writing_progress_lower_ci,
        max(case when subject = 'Writing'
                  and breakdown_topic = 'All pupils' and breakdown = 'Total'
            then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as writing_progress_upper_ci,
        max(case when subject = 'Writing'
                  and breakdown_topic = 'All pupils' and breakdown = 'Total'
            then {{ safe_numeric('working_towards_expected_standard_pupil_percent') }} end) as writing_working_towards_pct,
 ```
 After the `maths_progress` case (line ~72), add:
 ```sql
        max(case when subject = 'Maths'
                  and breakdown_topic = 'All pupils' and breakdown = 'Total'
            then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as maths_progress_lower_ci,
        max(case when subject = 'Maths'
                  and breakdown_topic = 'All pupils' and breakdown = 'Total'
            then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as maths_progress_upper_ci,
 ```
 Then add the seven new columns to the model's final select (next to the existing `p.reading_progress` / `p.writing_progress` / `p.maths_progress` lines ~145-152):
 ```sql
    p.reading_progress_lower_ci,
    p.reading_progress_upper_ci,
    p.writing_progress_lower_ci,
    p.writing_progress_upper_ci,
    p.writing_working_towards_pct,
    p.maths_progress_lower_ci,
    p.maths_progress_upper_ci,
 ```
 - [ ] **Step 2: Pass through the mart**
 In `fact_ks2_performance.sql`, add the same seven column names to the select list immediately after the existing `maths_progress` line (this mart selects staging columns by name; match the file's existing alias style — if columns are selected bare, add them bare).
 - [ ] **Step 3: Schema tests**
 In `_marts_schema.yml` under `fact_ks2_performance.columns`, append the seven names (no tests beyond presence — values are legitimately NULL for 2023/24+ since progress measures ended with 2022/23, spec §4.3):
 ```yaml
      - name: reading_progress_lower_ci
      - name: reading_progress_upper_ci
      - name: writing_progress_lower_ci
      - name: writing_progress_upper_ci
      - name: writing_working_towards_pct
      - name: maths_progress_lower_ci
      - name: maths_progress_upper_ci
 ```
 - [ ] **Step 4: Parse gate**
 Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .`
 Expected: `Done.`
 - [ ] **Step 5: Commit**
 ```bash
 git add pipeline/transform/models/staging/stg_ees_ks2.sql pipeline/transform/models/marts/fact_ks2_performance.sql pipeline/transform/models/marts/_marts_schema.yml
 git commit -m "feat(pipeline): KS2 progress confidence intervals and writing working-towards"
 ```
 ---
 ### Task 4: KS4 — Progress 8 banding and disadvantage gaps
 The tap's `ees_ks4_info` stream already declares `progress8_banding` (DfE's own "well above average … well below average" label — the ready-made secondary chip), `attainment8_diffn` and `progress8_diffn` (tap.py:338-340). Wire them through staging into the mart.
 **Files:**
 - Modify: `pipeline/transform/models/staging/stg_ees_ks4.sql` (the CTE that reads `ees_ks4_info` — the same one that already surfaces `sen_pct`; add three columns to its select and to the final joined select)
 - Modify: `pipeline/transform/models/marts/fact_ks4_performance.sql` (add after `progress_8_upper_ci`)
 - Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_ks4_performance block, ~line 93)
 **Interfaces:**
 - Produces mart columns: `progress_8_banding text`, `attainment_8_disadvantage_gap float`, `progress_8_disadvantage_gap float`.
 - [ ] **Step 1: Staging — select from the info source**
 In the info CTE of `stg_ees_ks4.sql` add:
 ```sql
        nullif(trim(progress8_banding), '') as progress_8_banding,
        {{ safe_numeric('attainment8_diffn') }} as attainment_8_disadvantage_gap,
        {{ safe_numeric('progress8_diffn') }} as progress_8_disadvantage_gap,
 ```
 and add the three names to the model's final select (aliased the same way the CTE's other columns are).
 - [ ] **Step 2: Mart passthrough**
 In `fact_ks4_performance.sql`, after the `progress_8_upper_ci,` line:
 ```sql
    progress_8_banding,
    attainment_8_disadvantage_gap,
    progress_8_disadvantage_gap,
 ```
 - [ ] **Step 3: Schema tests** — append the three names under `fact_ks4_performance.columns`, plus an accepted-values guard that tolerates NULL:
 ```yaml
      - name: progress_8_banding
        tests:
          - accepted_values:
              values: ['Well above average', 'Above average', 'Average', 'Below average', 'Well below average']
              config:
                where: "progress_8_banding is not null"
      - name: attainment_8_disadvantage_gap
      - name: progress_8_disadvantage_gap
 ```
 (If the DAG run later shows different capitalisation in the data, fix the accepted values to match the data, not vice versa.)
 - [ ] **Step 4: Parse gate** — `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.`
 - [ ] **Step 5: Commit**
 ```bash
 git add pipeline/transform/models/staging/stg_ees_ks4.sql pipeline/transform/models/marts/fact_ks4_performance.sql pipeline/transform/models/marts/_marts_schema.yml
 git commit -m "feat(pipeline): Progress 8 banding and KS4 disadvantage gaps in marts"
 ```
 ---
 ### Task 5: National averages — 2015/16 row and GPS/science/scaled-score fix
 Two changes. (1) `stg_ees_ks2_national.sql:34` filters `>= 201617`, which is exactly why the England line starts a year late (2015/16 RWM = 53% exists in the catalogue). (2) GPS/science expected are NULL in prod despite full end-to-end mapping — Task 1's findings say whether the catalogue CSV column names differ from `_KS2_NATIONAL_COL_MAP` (`pt_gps_exp`, `pt_scita_exp`) or whether values are suppressed at source.
 **Files:**
 - Modify: `pipeline/transform/models/staging/stg_ees_ks2_national.sql:34`
 - Modify (conditional on Task 1 findings): `pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py` (`_KS2_NATIONAL_COL_MAP`)
 **Interfaces:**
 - Produces: a 201516 row in `marts.fact_ks2_national_averages`; non-NULL `gps_expected_pct`, `science_expected_pct`, `reading_avg_score`, `maths_avg_score`, `gps_avg_score` for years the DfE publishes them. Backend/frontend consume via `/api/national-averages` unchanged (additive year + newly non-NULL fields).
 - [ ] **Step 1: Widen the year filter**
 In `stg_ees_ks2_national.sql`, change line 34:
 ```sql
  and cast(trim(time_period) as integer) >= 201516
 ```
 (2015/16 was the first year of the current expected-standard tests; nothing earlier is comparable, so keep a floor.)
 - [ ] **Step 2: Fix the column map per Task 1 findings**
 If Task 1 reported the actual CSV column names for GPS/science/scaled scores differ, update `_KS2_NATIONAL_COL_MAP` in `tap.py` accordingly, e.g. (illustrative — use the diagnosed names):
 ```python
 _KS2_NATIONAL_COL_MAP = {
    # ... existing entries ...
    "pt_gps_exp": "gps_expected_pct",      # replace key with diagnosed name
    "pt_scita_exp": "science_expected_pct",  # replace key with diagnosed name
 }
 ```
 If Task 1 showed the columns are present but suppressed (`x`) at national level for all years, instead delete the two entries from the map, delete the corresponding lines from `stg_ees_ks2_national.sql` and `fact_ks2_national_averages.sql`, and record in the PR description that GPS/science England ticks stay "not in dataset" (the mockups already carry that caveat).
 - [ ] **Step 3: Parse gate** — `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.`
 - [ ] **Step 4: Commit**
 ```bash
 git add pipeline/transform/models/staging/stg_ees_ks2_national.sql pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py
 git commit -m "fix(pipeline): include 2015/16 national averages; fix GPS/science national mapping"
 ```
 ---
 ### Task 6: Legacy KS2 — load the 2021/22 school-level year
 School-level 2021/22 exists in DfE performance-tables archives (same source as the four legacy years already loaded) but in neither our legacy config (stops at 201819, `pipeline/meltano.yml:33-37`) nor EES (starts 2022/23) — unless Task 1's finding (b) showed an EES release carrying 202122, in which case skip this task and note why in the PR.
 The legacy URLs point at the self-hosted filebrowser (`10.0.1.224:8081`) — **the 2021/22 DfE archive must be uploaded there first; this is the one human dependency in this plan.**
 **Files:**
 - Modify: `pipeline/meltano.yml` (legacy_ks2_urls block, line ~33)
 **Interfaces:**
 - Produces: `raw.legacy_ks2` rows with `year = '202122'`, flowing through `stg_legacy_ks2` → `fact_ks2_performance` unchanged (the stream maps old column names already; 2021/22 CSVs use the same `PTRWM_EXP`-style headers as 2018/19).
 - [ ] **Step 1: Verify the 2021/22 CSV headers match `_LEGACY_KS2_COLUMN_MAP`**
 Download the DfE 2021/22 KS2 revised archive (gov.uk "Compare School Performance data download": 2021-2022 all-schools ZIP), then:
 Run: `python -c "import zipfile,io,pandas as pd; zf=zipfile.ZipFile('/path/to/2021-2022.zip'); n=[x for x in zf.namelist() if 'ks2final' in x.lower() and x.endswith('.csv')][0]; df=pd.read_csv(zf.open(n), dtype=str, nrows=5); import sys; sys.path.insert(0,'pipeline/plugins/extractors/tap-uk-ees'); from tap_uk_ees.tap import _LEGACY_KS2_COLUMN_MAP as m; missing=[c for c in m if c not in df.columns]; print('missing legacy columns:', missing)"`
 Expected: `missing legacy columns: []` (progress columns `READPROG` etc. may legitimately be missing/blank in 2021/22 — acceptable, they load as NULL).
 - [ ] **Step 2: Upload the archive to the filebrowser and add the config entry**
 In `pipeline/meltano.yml` under `legacy_ks2_urls`, add (with the real share URL from the filebrowser upload):
 ```yaml
          "202122": "http://10.0.1.224:8081/filebrowser/api/public/dl/<SHARE_ID>?inline=true"
 ```
 - [ ] **Step 3: Commit**
 ```bash
 git add pipeline/meltano.yml
 git commit -m "feat(pipeline): load 2021/22 school-level KS2 from legacy performance tables"
 ```
 - [ ] **Step 4 (only if Task 1(b) showed 2022/23 subject labels differ):** widen the subject matchers in `stg_ees_ks2.sql` the same way GPS already is (`subject ilike '%grammar%' or subject = 'GPS'`), e.g. `subject in ('Reading', 'reading')` → use the diagnosed labels. Parse-gate and commit as `fix(pipeline): match 2022/23 KS2 subject labels`.
 ---
 ### Task 7: Ofsted report-card columns (rc_*)
 Resolves the tap TODO (`stg_ofsted_inspections.sql:37`). The marts/backed columns already exist as stubs; this wires real values. Uses Task 1(c)'s confirmed MI column names — the candidates below follow the MI file's existing naming style and must be corrected against the diagnostic output.
 **Files:**
 - Modify: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (COLUMN_PRIORITY ~line 19-72, schema ~line 100-114)
 - Create: `pipeline/transform/macros/parse_report_card_grade.sql`
 - Modify: `pipeline/transform/models/staging/stg_ofsted_inspections.sql:36-46`
 **Interfaces:**
 - Produces mart columns (already declared in `fact_ofsted_inspection`): `rc_safeguarding_met boolean`, and `rc_inclusion` … `rc_sixth_form` as integers on the 5-point scale `1=Exceptional, 2=Strong standard, 3=Expected standard, 4=Needs attention/Attention needed, 5=Urgent improvement`. The backend translates codes to labels (same pattern as `gias_codes.py`), verifying wording against Ofsted's published toolkit (spec §8.4).
 - [ ] **Step 1: Add tap column mappings**
 In `COLUMN_PRIORITY` add (replace candidate strings with Task 1(c)'s exact headers — keep them as priority lists so older files degrade to blank):
 ```python
    "rc_safeguarding_met": ["Report card safeguarding", "Safeguarding"],
    "rc_inclusion": ["Report card inclusion", "Inclusion"],
    "rc_curriculum_teaching": ["Report card curriculum and teaching", "Curriculum and teaching"],
    "rc_achievement": ["Report card achievement", "Achievement"],
    "rc_attendance_behaviour": ["Report card attendance and behaviour", "Attendance and behaviour"],
    "rc_personal_development": ["Report card personal development and well-being", "Personal development and well-being"],
    "rc_leadership_governance": ["Report card leadership and governance", "Leadership and governance"],
    "rc_early_years": ["Report card early years", "Early years"],
    "rc_sixth_form": ["Report card sixth form", "Sixth form"],
 ```
 And in the stream schema (next to `report_url`, ~line 114):
 ```python
        th.Property("rc_safeguarding_met", th.StringType),
        th.Property("rc_inclusion", th.StringType),
        th.Property("rc_curriculum_teaching", th.StringType),
        th.Property("rc_achievement", th.StringType),
        th.Property("rc_attendance_behaviour", th.StringType),
        th.Property("rc_personal_development", th.StringType),
        th.Property("rc_leadership_governance", th.StringType),
        th.Property("rc_early_years", th.StringType),
        th.Property("rc_sixth_form", th.StringType),
 ```
 - [ ] **Step 2: Write the grade-parsing macro**
 `pipeline/transform/macros/parse_report_card_grade.sql`:
 ```sql
 {% macro parse_report_card_grade(column_name) %}
    case lower(trim(nullif({{ column_name }}, 'NULL')))
        when 'exceptional' then 1
        when 'strong standard' then 2
        when 'expected standard' then 3
        when 'needs attention' then 4
        when 'attention needed' then 4
        when 'urgent improvement' then 5
    end
 {% endmacro %}
 ```
 - [ ] **Step 3: Wire staging**
 Replace `stg_ofsted_inspections.sql` lines 36-46 (the NULL stubs) with:
 ```sql
        -- Report Card fields (post-Nov 2025 framework), 5-point scale:
        -- 1 Exceptional · 2 Strong standard · 3 Expected standard
        -- · 4 Needs attention · 5 Urgent improvement
        (lower(trim(nullif(rc_safeguarding_met, 'NULL'))) = 'met')        as rc_safeguarding_met,
        {{ parse_report_card_grade('rc_inclusion') }}::integer            as rc_inclusion,
        {{ parse_report_card_grade('rc_curriculum_teaching') }}::integer  as rc_curriculum_teaching,
        {{ parse_report_card_grade('rc_achievement') }}::integer          as rc_achievement,
        {{ parse_report_card_grade('rc_attendance_behaviour') }}::integer as rc_attendance_behaviour,
        {{ parse_report_card_grade('rc_personal_development') }}::integer as rc_personal_development,
        {{ parse_report_card_grade('rc_leadership_governance') }}::integer as rc_leadership_governance,
        {{ parse_report_card_grade('rc_early_years') }}::integer          as rc_early_years,
        {{ parse_report_card_grade('rc_sixth_form') }}::integer           as rc_sixth_form,
 ```
 Note `rc_safeguarding_met` becomes boolean (NULL when blank) — matching `fact_ofsted_inspection`'s `rc_safeguarding_met` Boolean column. If `fact_ofsted_inspection.sql` casts these columns, align its casts too (inspect that model; it currently passes the text stubs through).
 - [ ] **Step 4: Parse gate + tap smoke test**
 Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.`
 Run: `python -c "import sys; sys.path.insert(0,'pipeline/plugins/extractors/tap-uk-ofsted'); from tap_uk_ofsted.tap import COLUMN_PRIORITY; assert 'rc_inclusion' in COLUMN_PRIORITY; print('ok')"` → `ok`
 - [ ] **Step 5: Commit**
 ```bash
 git add pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py pipeline/transform/macros/parse_report_card_grade.sql pipeline/transform/models/staging/stg_ofsted_inspections.sql
 git commit -m "feat(pipeline): extract Ofsted report-card judgements (rc_* columns)"
 ```
 ---
 ### Task 8: PR + post-merge verification
 **Files:** none new
 - [ ] **Step 1: Push and open the PR** (Gitea — use the git credential helper + basic-auth API pattern; token-header auth 401s):
 ```bash
 git push -u origin feat/compare-data-foundation
 # then create the PR via the Gitea API with basic auth from `git credential fill`
 ```
 PR body: link spec §5/§8, list the new mart columns, note the Task 6 human dependency (filebrowser upload) and the Task 1 findings file.
 - [ ] **Step 2: After merge, verify the DAG run picked everything up**
 The daily/monthly DAGs rebuild the affected models (`pipeline/dags/school_data_pipeline.py`). Spot-check via the public API (production after promotion, staging first at stx.schoolcompare.co.uk — note external /api is broken at the staging proxy, so check staging from the host):
 ```bash
 # 2015/16 national row exists
 curl -sL "https://www.schoolcompare.co.uk/api/national-averages" | python3 -c "import json,sys; d=json.load(sys.stdin); assert any(r['year']==201516 and r['primary'] for r in d['by_year']), '2015/16 missing'; print('201516 ok')"
 # 2021/22 school rows exist (Barclay)
 curl -sL "https://www.schoolcompare.co.uk/api/schools/138690" | python3 -c "import json,sys; d=json.load(sys.stdin); ys=[r['year'] for r in d['yearly_data']]; assert 202122 in [int(y) for y in ys], ys; print('202122 ok')"
 ```
 (The admissions/CI/KS4/rc_* columns aren't API-visible until the backend PR maps them — verify those directly in Postgres from the pipeline host: `select count(*) from marts.fact_admissions where second_preference_offers is not null;` etc.)
 - [ ] **Step 3: Update the spec** — tick off the §5 promotions this PR delivered (edit the spec's promotion list to note "landed in PR #NN") and commit to main via a docs PR or alongside the backend PR.
 ---
 ## Out of scope (next plans)
 1. **Backend PR:** map new columns in `backend/models.py`, extend `/api/compare` with supplementary blocks + `national_averages`, computed benchmarks (FSM/EAL/SEN/size medians, disadvantaged national average), CI-based progress banding, report-card label translation (verify against Ofsted toolkit), Ofsted provider-page URLs, graded-vs-ungraded surfacing.
 2. **Frontend PR:** rebuild `/compare` per the mockups + e2e journeys (promotion gate).
 3. **Separate bug fix:** third school's series not rendering on the current production chart.
 4. **Post-v1 (spec):** census ethnicity/young-carer promotion, IDACI display, attendance section, gender-split/absence tier-2 measures.
 5. **Already in marts, no work needed:** KS4 EBacc entry/APS, grade 5+ English & maths, Progress 8 CIs — `fact_ks4_performance` carries them today; only the backend needs to expose them.
@@ -0,0 +1,177 @@
 # Compare Screen Redesign — Expert Data Review
 **Date:** 2026-07-11
 **Reviewer:** subagent briefed as an English education-standards / DfE-Ofsted data expert
 **Subject:** desktop + mobile compare mockups and the redesign spec
 (`2026-07-11-compare-screen-redesign-design.md`)
 **Status:** first-pass must-fixes applied 2026-07-12; second-pass
 findings (below) applied 2026-07-12 — mockups + spec §4/§8 updated
 ## Must-fix
 1. **COVID gap is wrong and drops a real results year.** KS2 tests were
   cancelled 2019/20 and 2020/21 only; they resumed in 2021/22 with
   published school-level results (England RWM ≈ 59%). The mockup charts
   omit 2021/22 entirely and the tooltip claims no tests were held
   2019/20–2021/22. Fix: add 2021/22 to axis and all series; shrink the
   gap band; optionally annotate 2021/22 with DfE's post-pandemic
   comparability caution.
 2. **Report-card at-a-glance summary miscounts areas.** Detail list has
   4 Strong / 2 Expected / 1 Attention needed + Safeguarding met, but
   the summary says "3 areas Expected standard" — it counts safeguarding
   as a graded area. Safeguarding is a separate binary judgement and
   must be excluded from rating counts.
 3. **"Where the offers went" derivation is unsound.** Places − 1st-pref
   offers ≠ "second or third choices": the residual can include 4th–6th
   preference offers (pan-London scheme) and LA-allocated children who
   didn't choose the school; and offers don't necessarily equal PAN.
   Use the real 2nd/3rd-preference fields being promoted from
   `raw.ees_admissions`; until then drop the row.
 4. **Ofsted timeline in the copy is wrong.** Overall grades were
   abolished September 2024, not November 2025; Sept 2024–Nov 2025
   inspections kept the four key judgements without an overall grade
   (ungraded inspections carried grades forward). Neither mockup shows
   the interim regime, which will dominate real comparisons. Fix copy
   and add an interim example.
 5. **Barclay's "published an overall grade only — no area-by-area
   detail" misdescribes inspections.** No inspection type does that; a
   2021 graded inspection necessarily had subgrades — the gap is in our
   dataset. If it was an ungraded (s8) inspection, "Outstanding" is a
   carried-forward grade and should say so. Fix: "We don't hold
   area-by-area detail for this inspection", and distinguish graded vs
   ungraded in the data model.
 ## Should-fix
 6. Writing is teacher assessment, not a test — "national tests and
   teacher assessments"; note TA caveat on the Writing strip.
 7. Verify renewed-framework wording against Ofsted's final toolkit:
   likely "Needs attention" (not "Attention needed") and "Personal
   development and well-being" (which otherwise collides with the
   identically-named legacy judgement). Pin every label to the
   published toolkit.
 8. "Expected standard" now means two things on one page (Ofsted area
   rating vs KS2 measure) — disambiguate in tooltips.
 9. Disadvantaged row: DfE definition includes looked-after / previously
   looked-after children, not just FSM6; benchmark labels inconsistent
   across desktop/mobile; subgroup percentages need cohort sizes or a
   volatility threshold before chips are attached.
 10. "Trend, last 7 years" spans ten years; sparklines render the COVID
    gap as equal spacing (the exact defect the audit criticises) and
    "Improved: 52% → 87%" endpoint-cherry-picks a volatile series.
 11. At-a-glance "Getting a place" uses different metrics per school
    (Barclay is also oversubscribed on total preferences but shows a
    green chip). Standardise on first-preference success %. Explain the
    equal-preference rule; condition "living close by matters" on the
    school's actual oversubscription criteria.
 12. "457 applications for 180 places" = total preferences at any rank,
    not head-to-head applicants; lead with first preferences vs places.
    Add offers-vs-final-intake (waiting lists/appeals) caveat.
 13. Elmhurst's subgrade list is likely missing Early years provision
    (school has a nursery) — possible pipeline gap.
 14. "Ofsted rating" label is obsolete post-Sept-2024 — use "Latest
    Ofsted inspection"; check whether Oct 2021 is the latest inspection
    or merely the latest graded one.
 15. SEN: "EHCP plans" is redundant; 28% SEN support often indicates
    resourced provision — add a note; England SEN-support ≈ 14%, not 13%.
 ## Nice-to-have
 16. Consistent labelling of official DfE vs dataset-computed benchmarks
    (and medians shouldn't be called averages inconsistently).
 17. England 2015/16 RWM (53%) exists in DfE publications — the null is
    a dataset gap; source it or the England line looks broken.
 18. "1 in 4 first choices missed out" — actually more than 1 in 4.
 19. "1,273 of 1,260 places (full)" is over capacity; capacity figures
    are often stale — say "at or above capacity".
 20. State the actual suppression rule (DfE: ≤5 pupils suppressed,
    small numbers rounded) instead of "a handful".
 21. Spec §4.3 progress chips can't exist for displayed years: KS2
    progress ended with 2022/23 (no KS1 baseline) and returns
    ~2027/28 with the reception baseline. Make explicit in the spec.
    IDACI (spec §4.5) is absent from mockups; if shipped, caveat it
    describes pupils' neighbourhoods, not the school.
 22. Tooltips should give the official term "first preference" alongside
    the plain-English "first choice".
 ## Overall assessment (verbatim gist)
 The bones are genuinely good by education-data standards —
 England-average anchoring, explicit non-comparability messaging across
 Ofsted regimes, refusal to synthesise an overall grade, time-true
 x-axis, neutral FSM/EAL framing — better than most commercial
 school-comparison sites. But items 1–5 are outright factual errors or
 misdescriptions that a well-informed parent or Ofsted would catch;
 the admissions section needs the most conceptual work (equal
 preference, preferences-vs-applicants, offers-vs-intake). Fix 1–5
 before user testing; the rest fold into the planned PRs.
 ---
 # Second-pass review (2026-07-12)
 Same reviewer, after the must-fixes and the new three-tier metric
 exposure model were applied.
 ## Verification of first-pass must-fixes
 - **1 (COVID/2021/22): resolved.** Time-true axis, band covers only the
  cancelled years, England 58.7% consistent with official figures,
  dataset gaps break lines honestly; reading/maths England series all
  match published figures; RWM ≤ min(subject) checks pass.
 - **2 (report-card count): resolved** — safeguarding excluded, spec §8.2.
 - **3 (offers derivation): resolved** — row removed, spec §8.3 bans it.
 - **4 (Ofsted timeline): resolved on desktop; mobile omits the interim
  regime clause** (see finding 6).
 - **5 (Barclay explanation): resolved.**
 ## New findings
 1. **Should-fix — scaled-score strip domain contradicts caption.**
   Caption says "scaled scores run 80–120", strips render 100–120;
   truncated domain exaggerates small gaps and below-100 averages
   would fall off the edge. Render 80–120, or caption the 100–120
   window honestly and define below-100 behaviour.
 2. **Should-fix — scaled-score England ticks (106/105/105) unsourced.**
   Plausible but hand-entered; verify against DfE 2024/25 tables and
   add loading official England scaled scores to the pipeline list
   (absent from §8.1/§8.6).
 3. **Should-fix — "Writing" listed under "Higher standard" in the
   picker.** Writing TA outcome is "greater depth" (GDS), never
   "higher standard". Label "Writing — greater depth (teacher
   assessment)"; tooltip the combined higher-standard composition.
 4. Nice — "grammar & punctuation" summary line drops "spelling" (GPS).
 5. Nice — science is teacher-assessed (no KS2 test since 2009) and
   coarse; tooltip it like writing; reconsider its tier-2 slot.
 6. **Should-fix — mobile Ofsted copy skips the interim regime**
   (Sept 2024–Nov 2025) that desktop explains. One clause fixes it.
 7. **Should-fix — benchmark provenance still inconsistent** (EAL
   tooltip unsourced; FSM/disadvantaged chips vs tooltips use three
   vocabularies; header note says all England averages are official).
   Adopt one house style: official = "England average", computed =
   "benchmark / typical state school (our dataset)". Also tighten EAL
   definition to census wording ("first language known or believed to
   be other than English").
 8. Nice — "community primaries" distance note attached to an academy
   (Elmhurst); say "non-faith primaries" or condition on policy field.
 9. Nice — "Improving since 2022" → "since 2022/23".
 10. Nice — England chart tooltips show decimals; §7 mandates whole
    percents.
 ## Residual gaps not covered by spec §8
 11. Spec promises IDACI-in-words, Attendance section, and tier-2
    gender/absence that the mockups never show — mark post-v1 or
    demonstrate, so implementation scope is unambiguous.
 12. Add official England scaled-score averages to the pipeline task
    list.
 13. Add the writing/greater-depth terminology rule to §8.7.
 ## Verdict
 All must-fixes genuinely resolved; the tier model is conceptually
 sound ("no measure is lost", honest dataset-gap breaks, grouped
 picker). Remaining issues are contained: one internal contradiction
 (80–120 vs 100–120), one provenance inconsistency, one terminology
 error (writing/GDS). With findings 1–3 and 6–7 addressed, the data
 framing is fit to put in front of parents.
@@ -0,0 +1,318 @@
 # Compare Screen Redesign — Audit & Design
 **Date:** 2026-07-11
 **Status:** Draft — awaiting review
 **Scope:** `/compare` page (nextjs-app), `/api/compare` endpoint (backend)
 ## 1. Audit of the current screen
 The current compare page (`nextjs-app/components/ComparisonView.tsx`) is a
 single-metric analyst tool: a `<select>` with ~40 KS2/GCSE metrics, one
 line chart over time, and a year-by-year table — all for the one selected
 metric. Observed on production with 3 primary schools:
 **What works**
 - URL-shareable state (`?urns=…&metric=…`), native share sheet.
 - Phase tabs (primary/secondary) with sensible auto-detection.
 - Colour-coded school cards tied to chart series.
 - Metric descriptions from `/api/metrics` (single source of truth).
 **What doesn't**
 1. **Performance-only.** The database already holds Ofsted inspections,
   admissions/oversubscription history, pupil characteristics (FSM/EAL),
   SEN, deprivation (IDACI), finance, capacity, faith, gender, trust —
   none of it reaches the compare screen. `/api/compare` returns only
   `yearly_data` + minimal `school_info`, while `/api/schools/{urn}`
   already returns all supplementary blocks.
 2. **One metric at a time.** A parent must know which of ~40 metrics
   matters, select each in turn, and hold results in their head. There is
   no side-by-side overview and no way to see two dimensions at once.
 3. **No benchmarks.** Numbers float without anchors: is 79% RWM good?
   The DB has official national averages (`fact_ks2_national_averages`)
   but the page never shows them.
 4. **Domain jargon untranslated.** "GPS Expected %", "Progress scores",
   "RWM Combined" assume DfE literacy. The only plain-English help is one
   note for progress scores.
 5. **Raw numbers, no judgement support.** 87.0% vs 92.0% vs 79.0% — the
   page never says "all three are well above the England average of 62%",
   which is the fact a parent actually needs.
 6. **Bugs/paper cuts observed:** the third school's series did not render
   on the production chart despite table data (worth a separate fix);
   the COVID gap (2018/19 → 2022/23) renders as equal spacing with no
   annotation; table shows "87.0%" precision that implies false accuracy.
 ## 2. Data inventory (available vs shown)
 | Domain | Source table | On detail page | On compare |
 |---|---|---|---|
 | KS2 attainment/progress | fact_ks2_performance | yes | **yes** (only thing shown) |
 | National averages | fact_ks2_national_averages | partial | no |
 | Ofsted (latest + subgrades + report-card fields) | fact_ofsted_inspection, dim_school | yes | no |
 | Admissions & oversubscription (multi-year) | fact_admissions | yes | no |
 | Pupil characteristics (FSM, EAL, gender split) | fact_pupil_characteristics | yes | no |
 | Context (SEN, disadvantaged, stability, absence) | fact_ks2_performance | via metric picker | buried in picker |
 | Deprivation (IDACI) | fact_deprivation | yes | no |
 | Finance (per-pupil spend) | fact_finance | yes | no |
 | School facts (capacity, faith, ages, trust, nursery, gender) | dim_school | yes | no |
 | Location/distance | dim_location | map | no |
 ## 3. Design goals
 1. **Answer parent questions, in order:** Is it a good school (Ofsted)?
   Do children do well there (academics vs England)? Will my child get a
   place (admissions)? What is the school like (size, community, faith)?
 2. **Every number gets an anchor** — the England average, rendered as a
   consistent visual tick, plus a plain-English chip
   (Above / Close to / Below England average).
 3. **Plain English first, jargon on demand.** Labels are questions or
   sentences ("Children reaching the expected standard in reading,
   writing and maths"), codes/acronyms live in tooltips.
 4. **Scan whole-picture first, drill down second.** The single-metric
   trend explorer survives, demoted to an "Explore trends" section at the
   bottom rather than being the entire page.
 ## 4. Proposed structure
 Columns = schools (max 4 visible on desktop, horizontal scroll beyond),
 rows = dimensions. Sticky compact school header keeps column identity
 while scrolling. Sections, in order:
 1. **At a glance** — verdict row per school: Ofsted badge, headline
   attainment vs England (dot strip + chip), oversubscription chip,
   size, distance (when a location is set).
 2. **Ofsted inspection** — must handle all three inspection regimes,
   which will coexist in comparisons for years:
   - **Legacy graded (pre-Sept 2024):** overall grade badge
     (Outstanding/Good/Requires improvement/Inadequate). Subgrades,
     where published, are rendered in the **same area-by-rating chip
     list UX as report cards** (one row per judgement area, rating as
     a chip) — one visual grammar for inspection detail across both
     regimes. Where our dataset has no subgrades for an inspection,
     say so honestly ("We don't hold area-by-area detail for this
     inspection") and point to the school's Ofsted page — never claim
     the inspection itself published no detail (graded inspections
     always have subgrades; if it was ungraded, the grade is
     carried forward and must be labelled as such).
   - **Interim ungraded (Sept 2024 – Nov 2025):** parsed outcome
     ("remains Good") shown as the effective grade, marked as such.
   - **Renewed framework report card (from Nov 2025):** no overall
     grade exists. Render the report card as an area-by-rating list
     using Ofsted's 5-point scale (Exceptional / Strong standard /
     Expected standard / Attention needed / Urgent improvement) across
     the evaluation areas we model (`rc_inclusion`,
     `rc_curriculum_teaching`, `rc_achievement`,
     `rc_attendance_behaviour`, `rc_personal_development`,
     `rc_leadership_governance`, `rc_early_years`, `rc_sixth_form`)
     plus the separate safeguarding met/not-met flag. **At-a-glance
     summary rule:** never an unlabelled colour strip — summarise by
     counting areas per rating, best first ("5 areas Strong standard ·
     3 areas Expected standard"), and always name any area rated
     Attention needed or Urgent improvement explicitly (never fold
     problems into a count), plus "Safeguarding not met" whenever that
     flag is false. When everything is Expected standard or better,
     add the reassurance line "No areas need attention".
   When a comparison mixes regimes, show a one-line comparability note
   ("Ofsted changed how it reports in Nov 2025 — a report card and an
   older overall grade aren't directly comparable"). Never derive a
   fake overall grade from report-card areas.
 3. **Academics (KS2)** — one dot-strip row per headline measure (RWM
   expected, RWM higher, reading/writing/maths expected), each with the
   England-average tick and per-school dots; copy must say "tests and
   teacher assessments" (writing is TA, not a test). Progress scores
   translated to Above/Average/Below chips (CI-based) — **but note KS2
   progress measures ended with 2022/23** (no KS1 baseline afterwards)
   and return only when the reception-baseline cohort reaches Y6
   (~2027/28), so progress chips apply to historical years in the
   trends explorer, not the headline view. Sparkline per school over
   the full published period, with an honest gap for the cancelled
   test years (2019/20–2020/21). Disadvantaged-pupils row under an
   "Equity" subheading, always with cohort size shown and DfE's full
   definition (FSM6 **or** looked-after/previously looked-after).
 4. **Getting a place** — oversubscription ratio as plain sentence
   ("184 applications for 80 places"), first-preference success %, trend
   vs last year, admissions policy.
 5. **Who goes there** — pupils on roll (vs capacity), boys/girls, FSM %,
   EAL %, SEN support %, faith, ages, nursery, trust. *Post-v1:* IDACI
   decile in words (needs a coverage check of `fact_deprivation` and
   the neighbourhood-not-school caveat, §8.7).
 6. **Attendance** — *post-v1.* The KS2 test-day absence fields are the
   only per-school absence data we hold; they're near-zero for most
   schools and easy to misread as general attendance. Ship only if a
   general-absence source lands.
 7. **Explore trends** (existing feature, collapsed) — metric picker +
   multi-year line chart + table, with an added England-average
   reference line and a COVID-gap annotation.
 **Metric exposure model (three tiers).** No measure from the current
 page is lost; they surface at three levels of prominence:
 - **Tier 1 — headline strips (always visible):** RWM expected,
  reading/writing/maths expected, RWM higher standard.
 - **Tier 2 — "More measures" expansion inside Academics:** GPS and
  science expected % (science labelled teacher-assessed), average
  scaled scores (reading/maths/GPS, same dot-strip grammar showing
  the 100–120 window of the 80–120 scale, widening below 100, with
  the England tick) — one tap/click away, same visual language.
  *Post-v1:* gender split and absence (see §4.6).
 - **Tier 3 — Explore trends:** the full grouped catalogue (the
  current page's ~40 metrics, including equity and school-context
  measures, and the GCSE set for secondary phase) drives the
  year-by-year chart and table via the grouped metric picker.
 The tier assignment is a content decision per phase (secondary:
 Attainment 8, Progress 8 banding, grade 5+ English & maths as tier 1;
 EBacc and subject entries as tier 2).
 Finance (per-pupil spend) is deliberately deferred: low parent value,
 risk of misreading. Revisit later.
 **Mobile (design target — mobile first):** the desktop grid is the
 adaptation, not the other way round. On mobile the layout goes
 *measure-first*: each row is one measure with all schools listed under
 it (colour dot + short name + value + chip), so comparison never
 requires horizontal swiping between school cards. A sticky horizontal
 school-chip bar keeps identity and add/remove available while
 scrolling. Dot strips already read measure-first and carry over
 unchanged. The trend chart scrolls horizontally inside its container.
 ## 5. Data strategy — existing dataset only
 Constraint (agreed 2026-07-11): use only data already in marts plus
 fields already present in the `raw` schema extracts we pull today.
 No new external sources.
 **Gaps in the mockup, resolved within this constraint:**
 | Mockup element | Resolution |
 |---|---|
 | England average for disadvantaged pupils | Compute from our own data: `stg_ees_ks2` already pivots the Disadvantaged breakdown per school; aggregate it (weighted by eligible pupils) into `fact_ks2_national_averages` or compute in the API. Label it "England average (state schools)". |
 | England context for FSM / EAL / SEN chips | Compute dataset-wide medians per phase, same pattern as `/api/national-averages` does for KS4. |
 | "Much larger than average" size label | Dataset median pupils-on-roll per phase. |
 | Ofsted link | We don't have deep links to the latest report, so always link to the school's Ofsted provider page, `https://reports.ofsted.gov.uk/provider/21/{urn}`, derived from URN (label it "the school's Ofsted page", not "the report"). |
 **Raw fields we already pull but don't store — promote to marts (one
 dbt/pipeline PR, no tap changes):**
 - `raw.ees_admissions`: 2nd/3rd preference applications and offers,
  total-preference counts, cross-LA applications and offers → richer
  "Getting a place" (e.g. "offers reached 2nd-choice families",
  competition from outside the borough).
 - `raw.ees_ks2_attainment`: progress-measure confidence intervals and
  "working towards" % → lets the Above/Average/Below progress chips be
  statistically honest (band by CI overlap with 0, mirroring DfE
  methodology) instead of thresholding the point estimate.
 - `raw.ees_ks4_performance` / `ees_ks4_info`: `progress8_banding`
  (DfE's own plain-English "well above average … well below average"
  label — exactly the chip we want for secondary), EBacc entry/APS,
  grade-5+ English & maths, `attainment8_diffn`/`progress8_diffn`
  (disadvantage gaps) → the secondary-phase version of the Academics
  section.
 - `raw.ees_census`: young-carer % and the ethnicity breakdown →
  optional "Who goes there" enrichment; hold for a later iteration
  (presentation needs care), but the data requires no new extract.
 - `raw.ofsted_inspections` / tap-uk-ofsted: the `rc_*` report-card
  columns exist in staging/marts but are stubbed `null` — the tap has a
  TODO to map the report-card column names from the Ofsted MI file
  (same monthly extract we already download; inspections from Nov 2025
  onward carry them). This is the one promotion that needs a small tap
  schema addition, and it's a prerequisite for the new-framework Ofsted
  display above.
 Explicitly out (not in any current extract): school-level phonics,
 workforce/teacher data, per-school attendance beyond the KS2 test-day
 absence fields, Ofsted report-card documents themselves.
 ## 6. API changes
 Extend `GET /api/compare` response per URN with the same supplementary
 blocks the detail endpoint already builds (`get_supplementary_data`):
 `ofsted`, `census`, `admissions` (+ `admissions_history`), `deprivation`,
 plus a top-level `national_averages` block for the latest year. Reuse the
 existing function; no new tables. Response stays backward-compatible
 (additive fields only). Add derived helper fields server-side or compute
 chips client-side from `national_averages` (client-side preferred — no
 schema churn).
 ## 7. Accessibility & comprehension devices
 - Verdict chips are text + colour + position (never colour alone).
 - Every acronym has a tooltip using existing `MetricTooltip`.
 - "How to read this" one-liner at the top of each section.
 - Chart palette: coral `#e07256`, teal `#00949b`, purple `#8664c9`
  (validated: lightness band, chroma, CVD separation, contrast — the
  current `--chart-2/-4` tokens fail chroma/contrast checks and should
  be nudged to these).
 - Numbers rounded to whole percents; England tick labelled on first use.
 ## 8. Expert-review requirements
 An adversarial review by an education-data expert (full findings in
 `2026-07-11-compare-screen-expert-review.md`) was applied to the
 mockups on 2026-07-12. The following are binding requirements for
 implementation, beyond what the mockups can show:
 1. **Chart truthfulness:** KS2 tests were cancelled 2019/20–2020/21
   only; 2021/22 results exist and must be loaded (school-level rows
   and the 2015/16 national figure, 53%, are currently missing from
   our dataset — pipeline task). Load official England averages for
   GPS/science expected % and the scaled scores (reading/maths/GPS) —
   the mockups' scaled-score ticks are indicative until then. Never
   render missing years as if time were continuous.
 2. **Report-card summaries** count graded areas only — safeguarding is
   a separate binary flag, never included in rating counts.
 3. **Admissions:** use the real preference-breakdown fields from
   `raw.ees_admissions`; never derive "lower-preference offers" as
   places − first-preference offers. Frame total applications as
   "named on N forms" (any rank), lead with first-preference success,
   and standardise at-a-glance chips on that one metric. Explain the
   equal-preference rule; caveat offers vs final intake (waiting
   lists/appeals); condition "distance decides" on the school's actual
   oversubscription criteria where we have the admissions-policy field.
 4. **Ofsted:** overall grades ended September 2024 (report cards from
   November 2025); the interim regime must be renderable. Distinguish
   graded (s5) vs ungraded (s8) inspections and surface carried-forward
   grades as such; "we don't hold the detail" is a statement about our
   dataset, never about the inspection. Verify every scale/area label
   against Ofsted's final published toolkit before launch (e.g. "Needs
   attention" vs "Attention needed"; "Personal development and
   well-being" vs the identically-named legacy judgement). Check
   whether a school's latest inspection is merely its latest *graded*
   one. Confirm Early years provision subgrades flow through the
   pipeline for schools with nurseries.
 5. **Subgroup honesty:** disadvantaged-pupil percentages carry cohort
   sizes and follow the DfE suppression rule (≤5 pupils suppressed);
   state the rule verbatim in the footer.
 6. **Benchmark provenance:** official DfE figures and
   dataset-computed benchmarks must be labelled distinctly and
   consistently everywhere (a computed median is a "benchmark",
   not an "England average").
 7. **Copy details:** "Latest Ofsted inspection" (not "Ofsted rating");
   "EHC plans"; SEN-support benchmark ≈14%; high SEN share may
   indicate resourced provision (say so neutrally); "at or above
   capacity" rather than "full" (capacity data is often stale);
   disambiguate Ofsted's "Expected standard" from the KS2 measure;
   give official terms ("first preference") alongside plain English.
   Writing has no "higher standard" — its TA outcome is "greater
   depth (GDS)"; never list writing under a higher-standard group.
   Science and writing are teacher-assessed and must be labelled as
   such (no KS2 science test since 2009). House style for benchmark
   provenance: official DfE figures say "England average"; computed
   figures say "state-school average (computed from our dataset)" —
   applied to every chip, tooltip, header note and section intro.
   EAL uses the census wording: first language known or believed to
   be other than English. If IDACI ships, caveat that it describes
   pupils' home neighbourhoods, not the school.
 ## 9. Rollout
 1. **PR 1 (backend):** extend `/api/compare` + tests.
 2. **PR 2 (frontend):** new compare layout behind the existing route;
   e2e journey updated in the same PR (promotion gate).
 3. **Fix separately:** missing third series on the current chart.
 ## 10. Open questions for review
 - Max schools: keep 10 in API but cap visible columns at 4 with scroll?
 - Should distance-from-home appear when the user searched by postcode
  (data exists via `dim_location`)?
 - Keep finance out of v1? (Recommended: yes, out.)