# Compare-Screen Data Foundation (Pipeline PR) Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Land every pipeline/dbt change the compare-screen redesign needs (spec §5 + §8 of `docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md`): promote raw-but-unstored fields to marts, close the national-averages gaps, and wire the Ofsted report-card columns. **Architecture:** Meltano Singer taps load `raw.*` tables; dbt builds `staging` → `marts` (read-only for the backend). All changes here are additive columns/rows — no breaking changes to existing marts. The full `dbt build` runs on the server via the Airflow DAGs; locally we gate with `dbt parse` (no DB needed) plus network-only diagnostic scripts. **Tech Stack:** Python (Singer SDK taps), dbt-postgres ~1.10 (invoked as `python -m dbt.cli.main`), Meltano, PostgreSQL. ## Global Constraints - **No new external sources** (spec §5): only fields already in the `raw` schema or in files the taps already download. The one sanctioned tap change is the Ofsted MI report-card columns (spec §5, §8.4) and the legacy-KS2 year addition (same DfE performance-tables source). - **Additive only:** never rename or drop existing mart columns; the backend maps them 1:1 in `backend/models.py`. - **Never push to `main`.** Branch: `feat/compare-data-foundation`; PR checks must pass. - Backend `models.py` changes belong to the follow-up backend PR, not this one. - dbt invocation is always `python -m dbt.cli.main` (a bare `dbt` resolves to the wrong binary — see `pipeline/dags/school_data_pipeline.py:27`). - EES suppression codes `z`/`c`/`x` must go through the `safe_numeric` macro. - Computed benchmarks (FSM/EAL/SEN medians, disadvantaged national average) are **backend work** (spec §5) — explicitly out of scope here. --- ### Task 0: Create the branch **Files:** none - [ ] **Step 1:** `git checkout main && git pull && git checkout -b feat/compare-data-foundation` --- ### Task 1: Diagnostics — pin the three unknowns The spec flags three facts we must confirm from the actual files before wiring code: (a) why `gps_expected_pct`/`science_expected_pct` are NULL in `marts.fact_ks2_national_averages` despite being mapped end-to-end; (b) what the KS2 attainment long file calls its subjects/years for 2021/22 and 2022/23 (subject-level 2022/23 is NULL in prod; school-level 2021/22 is absent); (c) the exact report-card column headers in the current Ofsted MI CSV. **Files:** - Create: `pipeline/scripts/diagnose_compare_gaps.py` **Interfaces:** - Produces: a printed findings report; Tasks 5, 6, 7 consume the confirmed column/label names. Precedent: `pipeline/scripts/diagnose_ees_ks4.py`. - [ ] **Step 1: Write the diagnostic script** ```python """Diagnose the three data gaps blocking the compare-screen redesign. Run from repo root (network access required, no DB needed): python pipeline/scripts/diagnose_compare_gaps.py """ import io import re import sys import zipfile import pandas as pd import requests sys.path.insert(0, "pipeline/plugins/extractors/tap-uk-ees") sys.path.insert(0, "pipeline/plugins/extractors/tap-uk-ofsted") from tap_uk_ees.tap import ( # noqa: E402 _KS2_NATIONAL_COL_MAP, _KS2_NATIONAL_CSV_URL, download_release_zip, get_all_releases, ) from tap_uk_ofsted.tap import discover_csv_url # noqa: E402 TIMEOUT = 120 def check_national_gps_science(): print("\n=== (a) National catalogue CSV: GPS/science columns ===") resp = requests.get(_KS2_NATIONAL_CSV_URL, timeout=TIMEOUT) resp.raise_for_status() df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False) df.columns = [c.strip().lower() for c in df.columns] for csv_col in ("pt_gps_exp", "pt_scita_exp", "avg_readscore", "avg_matscore", "avg_gpsscore"): status = "PRESENT" if csv_col in df.columns else "MISSING" print(f" {csv_col}: {status}") gps_like = [c for c in df.columns if "gps" in c or "scita" in c or "sci" in c] print(f" all gps/science-ish columns: {gps_like}") nat = df[df.get("geographic_level", "").str.strip().str.lower() == "national"] print(f" national rows time_periods: {sorted(nat['time_period'].unique())}") # Sample the values our map would read for the latest year latest = nat[nat["time_period"] == nat["time_period"].max()] for csv_col, field in _KS2_NATIONAL_COL_MAP.items(): val = latest.iloc[0].get(csv_col, "") if len(latest) else "" print(f" {field} <- {csv_col} = {val!r}") def check_ks2_attainment_years_subjects(): print("\n=== (b) EES KS2 attainment: years & subject labels ===") releases = get_all_releases("key-stage-2-attainment") print(f" releases found: {[r['time_period'] for r in releases]}") for release in releases: zf = download_release_zip(release["id"]) name = next((n for n in zf.namelist() if "ks2_school_attainment_data" in n and n.endswith(".csv")), None) if not name: print(f" {release['time_period']}: NO school attainment CSV in ZIP") continue with zf.open(name) as f: df = pd.read_csv(f, dtype=str, keep_default_na=False, nrows=200000) years = sorted(df["time_period"].unique()) subjects = sorted(df["subject"].unique()) print(f" release {release['time_period']}: time_periods={years}") print(f" subjects={subjects}") def check_ofsted_report_card_columns(): print("\n=== (c) Ofsted MI CSV: report-card columns ===") url = discover_csv_url() print(f" MI file: {url}") resp = requests.get(url, timeout=TIMEOUT) resp.raise_for_status() df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False, nrows=5) rc_like = [c for c in df.columns if re.search(r"report card|inclusion|curriculum|achievement|safeguard|well.?being|governance", c, re.I)] print(f" candidate report-card columns ({len(rc_like)}):") for c in rc_like: print(f" - {c!r}") if __name__ == "__main__": check_national_gps_science() check_ks2_attainment_years_subjects() check_ofsted_report_card_columns() ``` Note: if `_KS2_NATIONAL_CSV_URL` is named differently in `tap_uk_ees/tap.py` (it is defined near the `_KS2_NATIONAL_COL_MAP` around line ~490), import whatever constant holds the catalogue CSV URL. - [ ] **Step 2: Run it and record findings** Run: `python pipeline/scripts/diagnose_compare_gaps.py 2>&1 | tee /tmp/compare-gaps-findings.txt` Expected: three sections printed. Paste the findings as a comment block at the bottom of the script (so they're committed evidence), e.g. `# FINDINGS 2026-07-12: pt_gps_exp MISSING (actual col: ...), 202122 present in release X, rc columns: [...]`. - [ ] **Step 3: Commit** ```bash git add pipeline/scripts/diagnose_compare_gaps.py git commit -m "chore(pipeline): diagnostic for compare-screen data gaps" ``` --- ### Task 2: Admissions preference detail → mart Staging already extracts `second_preference_offers`, `third_preference_offers`, `total_offers` (`stg_ees_admissions.sql:26-29`) — the mart drops them. The cross-LA fields are declared in the tap (`all_applications_from_another_LA`, `offers_to_applicants_from_another_LA`) but not selected in staging. **Files:** - Modify: `pipeline/transform/models/staging/stg_ees_admissions.sql` (after line 33, in `renamed`) - Modify: `pipeline/transform/models/marts/fact_admissions.sql` - Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_admissions block, ~line 120) **Interfaces:** - Produces mart columns: `total_offers int`, `second_preference_offers int`, `third_preference_offers int`, `cross_la_applications int`, `cross_la_offers int`. The backend PR will map these in `FactAdmissions`. - [ ] **Step 1: Add cross-LA columns to staging** In `stg_ees_admissions.sql`, after the `first_preference_applications` line (line 33): ```sql -- Cross-borough demand: applications naming this school from families -- living in another local authority, and offers made to them. {{ safe_numeric('"all_applications_from_another_LA"') }}::integer as cross_la_applications, {{ safe_numeric('"offers_to_applicants_from_another_LA"') }}::integer as cross_la_offers, ``` (Quote the identifiers — the tap emits them with mixed case, same trap as `FSM_eligible_percent`, see the header comment in that file. If `dbt parse` or the DAG run later shows the raw columns are lower-cased in Postgres, drop the double quotes.) - [ ] **Step 2: Pass everything through the mart** Replace the full select list in `fact_admissions.sql`: ```sql -- Mart: School admissions — one row per URN per year select urn, year, school_phase, places_offered, total_offers, total_applications, first_preference_applications, first_preference_offers, second_preference_offers, third_preference_offers, cross_la_applications, cross_la_offers, first_preference_offer_pct, oversubscription_ratio, oversubscribed, admissions_policy from {{ ref('stg_ees_admissions') }} ``` - [ ] **Step 3: Add schema tests** In `_marts_schema.yml` under `fact_admissions.columns`, append: ```yaml - name: second_preference_offers - name: third_preference_offers - name: cross_la_applications - name: cross_la_offers - name: total_offers ``` - [ ] **Step 4: Parse gate** Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` Expected: `Done.` with no compilation errors. - [ ] **Step 5: Commit** ```bash git add pipeline/transform/models/staging/stg_ees_admissions.sql pipeline/transform/models/marts/fact_admissions.sql pipeline/transform/models/marts/_marts_schema.yml git commit -m "feat(pipeline): admissions preference breakdown and cross-LA demand in marts" ``` --- ### Task 3: KS2 progress confidence intervals + writing working-towards The tap already emits `progress_measure_lower_conf_interval`, `progress_measure_upper_conf_interval`, `working_towards_expected_standard_pupil_percent` (tap.py:203-206). The staging pivot drops them. These power the CI-based Above/Average/Below progress chips (spec §8, first-review item on statistical honesty). **Files:** - Modify: `pipeline/transform/models/staging/stg_ees_ks2.sql` (inside the `pivoted` CTE, next to each subject's `progress_measure_score` case, lines ~41/55/72, and in the final select ~lines 145-152) - Modify: `pipeline/transform/models/marts/fact_ks2_performance.sql` - Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_ks2_performance block, ~line 82) **Interfaces:** - Produces mart columns: `reading_progress_lower_ci`, `reading_progress_upper_ci`, `writing_progress_lower_ci`, `writing_progress_upper_ci`, `maths_progress_lower_ci`, `maths_progress_upper_ci` (float), `writing_working_towards_pct` (float). - [ ] **Step 1: Add pivot cases in staging** After the `reading_progress` case (line ~41), add: ```sql max(case when subject = 'Reading' and breakdown_topic = 'All pupils' and breakdown = 'Total' then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as reading_progress_lower_ci, max(case when subject = 'Reading' and breakdown_topic = 'All pupils' and breakdown = 'Total' then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as reading_progress_upper_ci, ``` After the `writing_progress` case (line ~55), add: ```sql max(case when subject = 'Writing' and breakdown_topic = 'All pupils' and breakdown = 'Total' then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as writing_progress_lower_ci, max(case when subject = 'Writing' and breakdown_topic = 'All pupils' and breakdown = 'Total' then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as writing_progress_upper_ci, max(case when subject = 'Writing' and breakdown_topic = 'All pupils' and breakdown = 'Total' then {{ safe_numeric('working_towards_expected_standard_pupil_percent') }} end) as writing_working_towards_pct, ``` After the `maths_progress` case (line ~72), add: ```sql max(case when subject = 'Maths' and breakdown_topic = 'All pupils' and breakdown = 'Total' then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as maths_progress_lower_ci, max(case when subject = 'Maths' and breakdown_topic = 'All pupils' and breakdown = 'Total' then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as maths_progress_upper_ci, ``` Then add the seven new columns to the model's final select (next to the existing `p.reading_progress` / `p.writing_progress` / `p.maths_progress` lines ~145-152): ```sql p.reading_progress_lower_ci, p.reading_progress_upper_ci, p.writing_progress_lower_ci, p.writing_progress_upper_ci, p.writing_working_towards_pct, p.maths_progress_lower_ci, p.maths_progress_upper_ci, ``` - [ ] **Step 2: Pass through the mart** In `fact_ks2_performance.sql`, add the same seven column names to the select list immediately after the existing `maths_progress` line (this mart selects staging columns by name; match the file's existing alias style — if columns are selected bare, add them bare). - [ ] **Step 3: Schema tests** In `_marts_schema.yml` under `fact_ks2_performance.columns`, append the seven names (no tests beyond presence — values are legitimately NULL for 2023/24+ since progress measures ended with 2022/23, spec §4.3): ```yaml - name: reading_progress_lower_ci - name: reading_progress_upper_ci - name: writing_progress_lower_ci - name: writing_progress_upper_ci - name: writing_working_towards_pct - name: maths_progress_lower_ci - name: maths_progress_upper_ci ``` - [ ] **Step 4: Parse gate** Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` Expected: `Done.` - [ ] **Step 5: Commit** ```bash git add pipeline/transform/models/staging/stg_ees_ks2.sql pipeline/transform/models/marts/fact_ks2_performance.sql pipeline/transform/models/marts/_marts_schema.yml git commit -m "feat(pipeline): KS2 progress confidence intervals and writing working-towards" ``` --- ### Task 4: KS4 — Progress 8 banding and disadvantage gaps The tap's `ees_ks4_info` stream already declares `progress8_banding` (DfE's own "well above average … well below average" label — the ready-made secondary chip), `attainment8_diffn` and `progress8_diffn` (tap.py:338-340). Wire them through staging into the mart. **Files:** - Modify: `pipeline/transform/models/staging/stg_ees_ks4.sql` (the CTE that reads `ees_ks4_info` — the same one that already surfaces `sen_pct`; add three columns to its select and to the final joined select) - Modify: `pipeline/transform/models/marts/fact_ks4_performance.sql` (add after `progress_8_upper_ci`) - Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_ks4_performance block, ~line 93) **Interfaces:** - Produces mart columns: `progress_8_banding text`, `attainment_8_disadvantage_gap float`, `progress_8_disadvantage_gap float`. - [ ] **Step 1: Staging — select from the info source** In the info CTE of `stg_ees_ks4.sql` add: ```sql nullif(trim(progress8_banding), '') as progress_8_banding, {{ safe_numeric('attainment8_diffn') }} as attainment_8_disadvantage_gap, {{ safe_numeric('progress8_diffn') }} as progress_8_disadvantage_gap, ``` and add the three names to the model's final select (aliased the same way the CTE's other columns are). - [ ] **Step 2: Mart passthrough** In `fact_ks4_performance.sql`, after the `progress_8_upper_ci,` line: ```sql progress_8_banding, attainment_8_disadvantage_gap, progress_8_disadvantage_gap, ``` - [ ] **Step 3: Schema tests** — append the three names under `fact_ks4_performance.columns`, plus an accepted-values guard that tolerates NULL: ```yaml - name: progress_8_banding tests: - accepted_values: values: ['Well above average', 'Above average', 'Average', 'Below average', 'Well below average'] config: where: "progress_8_banding is not null" - name: attainment_8_disadvantage_gap - name: progress_8_disadvantage_gap ``` (If the DAG run later shows different capitalisation in the data, fix the accepted values to match the data, not vice versa.) - [ ] **Step 4: Parse gate** — `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.` - [ ] **Step 5: Commit** ```bash git add pipeline/transform/models/staging/stg_ees_ks4.sql pipeline/transform/models/marts/fact_ks4_performance.sql pipeline/transform/models/marts/_marts_schema.yml git commit -m "feat(pipeline): Progress 8 banding and KS4 disadvantage gaps in marts" ``` --- ### Task 5: National averages — 2015/16 row and GPS/science/scaled-score fix Two changes. (1) `stg_ees_ks2_national.sql:34` filters `>= 201617`, which is exactly why the England line starts a year late (2015/16 RWM = 53% exists in the catalogue). (2) GPS/science expected are NULL in prod despite full end-to-end mapping — Task 1's findings say whether the catalogue CSV column names differ from `_KS2_NATIONAL_COL_MAP` (`pt_gps_exp`, `pt_scita_exp`) or whether values are suppressed at source. **Files:** - Modify: `pipeline/transform/models/staging/stg_ees_ks2_national.sql:34` - Modify (conditional on Task 1 findings): `pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py` (`_KS2_NATIONAL_COL_MAP`) **Interfaces:** - Produces: a 201516 row in `marts.fact_ks2_national_averages`; non-NULL `gps_expected_pct`, `science_expected_pct`, `reading_avg_score`, `maths_avg_score`, `gps_avg_score` for years the DfE publishes them. Backend/frontend consume via `/api/national-averages` unchanged (additive year + newly non-NULL fields). - [ ] **Step 1: Widen the year filter** In `stg_ees_ks2_national.sql`, change line 34: ```sql and cast(trim(time_period) as integer) >= 201516 ``` (2015/16 was the first year of the current expected-standard tests; nothing earlier is comparable, so keep a floor.) - [ ] **Step 2: Fix the column map per Task 1 findings** If Task 1 reported the actual CSV column names for GPS/science/scaled scores differ, update `_KS2_NATIONAL_COL_MAP` in `tap.py` accordingly, e.g. (illustrative — use the diagnosed names): ```python _KS2_NATIONAL_COL_MAP = { # ... existing entries ... "pt_gps_exp": "gps_expected_pct", # replace key with diagnosed name "pt_scita_exp": "science_expected_pct", # replace key with diagnosed name } ``` If Task 1 showed the columns are present but suppressed (`x`) at national level for all years, instead delete the two entries from the map, delete the corresponding lines from `stg_ees_ks2_national.sql` and `fact_ks2_national_averages.sql`, and record in the PR description that GPS/science England ticks stay "not in dataset" (the mockups already carry that caveat). - [ ] **Step 3: Parse gate** — `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.` - [ ] **Step 4: Commit** ```bash git add pipeline/transform/models/staging/stg_ees_ks2_national.sql pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py git commit -m "fix(pipeline): include 2015/16 national averages; fix GPS/science national mapping" ``` --- ### Task 6: Legacy KS2 — load the 2021/22 school-level year School-level 2021/22 exists in DfE performance-tables archives (same source as the four legacy years already loaded) but in neither our legacy config (stops at 201819, `pipeline/meltano.yml:33-37`) nor EES (starts 2022/23) — unless Task 1's finding (b) showed an EES release carrying 202122, in which case skip this task and note why in the PR. The legacy URLs point at the self-hosted filebrowser (`10.0.1.224:8081`) — **the 2021/22 DfE archive must be uploaded there first; this is the one human dependency in this plan.** **Files:** - Modify: `pipeline/meltano.yml` (legacy_ks2_urls block, line ~33) **Interfaces:** - Produces: `raw.legacy_ks2` rows with `year = '202122'`, flowing through `stg_legacy_ks2` → `fact_ks2_performance` unchanged (the stream maps old column names already; 2021/22 CSVs use the same `PTRWM_EXP`-style headers as 2018/19). - [ ] **Step 1: Verify the 2021/22 CSV headers match `_LEGACY_KS2_COLUMN_MAP`** Download the DfE 2021/22 KS2 revised archive (gov.uk "Compare School Performance data download": 2021-2022 all-schools ZIP), then: Run: `python -c "import zipfile,io,pandas as pd; zf=zipfile.ZipFile('/path/to/2021-2022.zip'); n=[x for x in zf.namelist() if 'ks2final' in x.lower() and x.endswith('.csv')][0]; df=pd.read_csv(zf.open(n), dtype=str, nrows=5); import sys; sys.path.insert(0,'pipeline/plugins/extractors/tap-uk-ees'); from tap_uk_ees.tap import _LEGACY_KS2_COLUMN_MAP as m; missing=[c for c in m if c not in df.columns]; print('missing legacy columns:', missing)"` Expected: `missing legacy columns: []` (progress columns `READPROG` etc. may legitimately be missing/blank in 2021/22 — acceptable, they load as NULL). - [ ] **Step 2: Upload the archive to the filebrowser and add the config entry** In `pipeline/meltano.yml` under `legacy_ks2_urls`, add (with the real share URL from the filebrowser upload): ```yaml "202122": "http://10.0.1.224:8081/filebrowser/api/public/dl/?inline=true" ``` - [ ] **Step 3: Commit** ```bash git add pipeline/meltano.yml git commit -m "feat(pipeline): load 2021/22 school-level KS2 from legacy performance tables" ``` - [ ] **Step 4 (only if Task 1(b) showed 2022/23 subject labels differ):** widen the subject matchers in `stg_ees_ks2.sql` the same way GPS already is (`subject ilike '%grammar%' or subject = 'GPS'`), e.g. `subject in ('Reading', 'reading')` → use the diagnosed labels. Parse-gate and commit as `fix(pipeline): match 2022/23 KS2 subject labels`. --- ### Task 7: Ofsted report-card columns (rc_*) Resolves the tap TODO (`stg_ofsted_inspections.sql:37`). The marts/backed columns already exist as stubs; this wires real values. Uses Task 1(c)'s confirmed MI column names — the candidates below follow the MI file's existing naming style and must be corrected against the diagnostic output. **Files:** - Modify: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (COLUMN_PRIORITY ~line 19-72, schema ~line 100-114) - Create: `pipeline/transform/macros/parse_report_card_grade.sql` - Modify: `pipeline/transform/models/staging/stg_ofsted_inspections.sql:36-46` **Interfaces:** - Produces mart columns (already declared in `fact_ofsted_inspection`): `rc_safeguarding_met boolean`, and `rc_inclusion` … `rc_sixth_form` as integers on the 5-point scale `1=Exceptional, 2=Strong standard, 3=Expected standard, 4=Needs attention/Attention needed, 5=Urgent improvement`. The backend translates codes to labels (same pattern as `gias_codes.py`), verifying wording against Ofsted's published toolkit (spec §8.4). - [ ] **Step 1: Add tap column mappings** In `COLUMN_PRIORITY` add (replace candidate strings with Task 1(c)'s exact headers — keep them as priority lists so older files degrade to blank): ```python "rc_safeguarding_met": ["Report card safeguarding", "Safeguarding"], "rc_inclusion": ["Report card inclusion", "Inclusion"], "rc_curriculum_teaching": ["Report card curriculum and teaching", "Curriculum and teaching"], "rc_achievement": ["Report card achievement", "Achievement"], "rc_attendance_behaviour": ["Report card attendance and behaviour", "Attendance and behaviour"], "rc_personal_development": ["Report card personal development and well-being", "Personal development and well-being"], "rc_leadership_governance": ["Report card leadership and governance", "Leadership and governance"], "rc_early_years": ["Report card early years", "Early years"], "rc_sixth_form": ["Report card sixth form", "Sixth form"], ``` And in the stream schema (next to `report_url`, ~line 114): ```python th.Property("rc_safeguarding_met", th.StringType), th.Property("rc_inclusion", th.StringType), th.Property("rc_curriculum_teaching", th.StringType), th.Property("rc_achievement", th.StringType), th.Property("rc_attendance_behaviour", th.StringType), th.Property("rc_personal_development", th.StringType), th.Property("rc_leadership_governance", th.StringType), th.Property("rc_early_years", th.StringType), th.Property("rc_sixth_form", th.StringType), ``` - [ ] **Step 2: Write the grade-parsing macro** `pipeline/transform/macros/parse_report_card_grade.sql`: ```sql {% macro parse_report_card_grade(column_name) %} case lower(trim(nullif({{ column_name }}, 'NULL'))) when 'exceptional' then 1 when 'strong standard' then 2 when 'expected standard' then 3 when 'needs attention' then 4 when 'attention needed' then 4 when 'urgent improvement' then 5 end {% endmacro %} ``` - [ ] **Step 3: Wire staging** Replace `stg_ofsted_inspections.sql` lines 36-46 (the NULL stubs) with: ```sql -- Report Card fields (post-Nov 2025 framework), 5-point scale: -- 1 Exceptional · 2 Strong standard · 3 Expected standard -- · 4 Needs attention · 5 Urgent improvement (lower(trim(nullif(rc_safeguarding_met, 'NULL'))) = 'met') as rc_safeguarding_met, {{ parse_report_card_grade('rc_inclusion') }}::integer as rc_inclusion, {{ parse_report_card_grade('rc_curriculum_teaching') }}::integer as rc_curriculum_teaching, {{ parse_report_card_grade('rc_achievement') }}::integer as rc_achievement, {{ parse_report_card_grade('rc_attendance_behaviour') }}::integer as rc_attendance_behaviour, {{ parse_report_card_grade('rc_personal_development') }}::integer as rc_personal_development, {{ parse_report_card_grade('rc_leadership_governance') }}::integer as rc_leadership_governance, {{ parse_report_card_grade('rc_early_years') }}::integer as rc_early_years, {{ parse_report_card_grade('rc_sixth_form') }}::integer as rc_sixth_form, ``` Note `rc_safeguarding_met` becomes boolean (NULL when blank) — matching `fact_ofsted_inspection`'s `rc_safeguarding_met` Boolean column. If `fact_ofsted_inspection.sql` casts these columns, align its casts too (inspect that model; it currently passes the text stubs through). - [ ] **Step 4: Parse gate + tap smoke test** Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.` Run: `python -c "import sys; sys.path.insert(0,'pipeline/plugins/extractors/tap-uk-ofsted'); from tap_uk_ofsted.tap import COLUMN_PRIORITY; assert 'rc_inclusion' in COLUMN_PRIORITY; print('ok')"` → `ok` - [ ] **Step 5: Commit** ```bash git add pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py pipeline/transform/macros/parse_report_card_grade.sql pipeline/transform/models/staging/stg_ofsted_inspections.sql git commit -m "feat(pipeline): extract Ofsted report-card judgements (rc_* columns)" ``` --- ### Task 8: PR + post-merge verification **Files:** none new - [ ] **Step 1: Push and open the PR** (Gitea — use the git credential helper + basic-auth API pattern; token-header auth 401s): ```bash git push -u origin feat/compare-data-foundation # then create the PR via the Gitea API with basic auth from `git credential fill` ``` PR body: link spec §5/§8, list the new mart columns, note the Task 6 human dependency (filebrowser upload) and the Task 1 findings file. - [ ] **Step 2: After merge, verify the DAG run picked everything up** The daily/monthly DAGs rebuild the affected models (`pipeline/dags/school_data_pipeline.py`). Spot-check via the public API (production after promotion, staging first at stx.schoolcompare.co.uk — note external /api is broken at the staging proxy, so check staging from the host): ```bash # 2015/16 national row exists curl -sL "https://www.schoolcompare.co.uk/api/national-averages" | python3 -c "import json,sys; d=json.load(sys.stdin); assert any(r['year']==201516 and r['primary'] for r in d['by_year']), '2015/16 missing'; print('201516 ok')" # 2021/22 school rows exist (Barclay) curl -sL "https://www.schoolcompare.co.uk/api/schools/138690" | python3 -c "import json,sys; d=json.load(sys.stdin); ys=[r['year'] for r in d['yearly_data']]; assert 202122 in [int(y) for y in ys], ys; print('202122 ok')" ``` (The admissions/CI/KS4/rc_* columns aren't API-visible until the backend PR maps them — verify those directly in Postgres from the pipeline host: `select count(*) from marts.fact_admissions where second_preference_offers is not null;` etc.) - [ ] **Step 3: Update the spec** — tick off the §5 promotions this PR delivered (edit the spec's promotion list to note "landed in PR #NN") and commit to main via a docs PR or alongside the backend PR. --- ## Out of scope (next plans) 1. **Backend PR:** map new columns in `backend/models.py`, extend `/api/compare` with supplementary blocks + `national_averages`, computed benchmarks (FSM/EAL/SEN/size medians, disadvantaged national average), CI-based progress banding, report-card label translation (verify against Ofsted toolkit), Ofsted provider-page URLs, graded-vs-ungraded surfacing. 2. **Frontend PR:** rebuild `/compare` per the mockups + e2e journeys (promotion gate). 3. **Separate bug fix:** third school's series not rendering on the current production chart. 4. **Post-v1 (spec):** census ethnicity/young-carer promotion, IDACI display, attendance section, gender-split/absence tier-2 measures. 5. **Already in marts, no work needed:** KS4 EBacc entry/APS, grade 5+ English & maths, Progress 8 CIs — `fact_ks4_performance` carries them today; only the backend needs to expose them.