feat(pipeline): compare-screen data foundation — raw→marts promotions, national averages, Ofsted report cards #32
@@ -0,0 +1,591 @@
|
||||
# Compare-Screen Data Foundation (Pipeline PR) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Land every pipeline/dbt change the compare-screen redesign needs (spec §5 + §8 of `docs/superpowers/specs/2026-07-11-compare-screen-redesign-design.md`): promote raw-but-unstored fields to marts, close the national-averages gaps, and wire the Ofsted report-card columns.
|
||||
|
||||
**Architecture:** Meltano Singer taps load `raw.*` tables; dbt builds `staging` → `marts` (read-only for the backend). All changes here are additive columns/rows — no breaking changes to existing marts. The full `dbt build` runs on the server via the Airflow DAGs; locally we gate with `dbt parse` (no DB needed) plus network-only diagnostic scripts.
|
||||
|
||||
**Tech Stack:** Python (Singer SDK taps), dbt-postgres ~1.10 (invoked as `python -m dbt.cli.main`), Meltano, PostgreSQL.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **No new external sources** (spec §5): only fields already in the `raw` schema or in files the taps already download. The one sanctioned tap change is the Ofsted MI report-card columns (spec §5, §8.4) and the legacy-KS2 year addition (same DfE performance-tables source).
|
||||
- **Additive only:** never rename or drop existing mart columns; the backend maps them 1:1 in `backend/models.py`.
|
||||
- **Never push to `main`.** Branch: `feat/compare-data-foundation`; PR checks must pass.
|
||||
- Backend `models.py` changes belong to the follow-up backend PR, not this one.
|
||||
- dbt invocation is always `python -m dbt.cli.main` (a bare `dbt` resolves to the wrong binary — see `pipeline/dags/school_data_pipeline.py:27`).
|
||||
- EES suppression codes `z`/`c`/`x` must go through the `safe_numeric` macro.
|
||||
- Computed benchmarks (FSM/EAL/SEN medians, disadvantaged national average) are **backend work** (spec §5) — explicitly out of scope here.
|
||||
|
||||
---
|
||||
|
||||
### Task 0: Create the branch
|
||||
|
||||
**Files:** none
|
||||
|
||||
- [ ] **Step 1:** `git checkout main && git pull && git checkout -b feat/compare-data-foundation`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Diagnostics — pin the three unknowns
|
||||
|
||||
The spec flags three facts we must confirm from the actual files before wiring code: (a) why `gps_expected_pct`/`science_expected_pct` are NULL in `marts.fact_ks2_national_averages` despite being mapped end-to-end; (b) what the KS2 attainment long file calls its subjects/years for 2021/22 and 2022/23 (subject-level 2022/23 is NULL in prod; school-level 2021/22 is absent); (c) the exact report-card column headers in the current Ofsted MI CSV.
|
||||
|
||||
**Files:**
|
||||
- Create: `pipeline/scripts/diagnose_compare_gaps.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: a printed findings report; Tasks 5, 6, 7 consume the confirmed column/label names. Precedent: `pipeline/scripts/diagnose_ees_ks4.py`.
|
||||
|
||||
- [ ] **Step 1: Write the diagnostic script**
|
||||
|
||||
```python
|
||||
"""Diagnose the three data gaps blocking the compare-screen redesign.
|
||||
|
||||
Run from repo root (network access required, no DB needed):
|
||||
python pipeline/scripts/diagnose_compare_gaps.py
|
||||
"""
|
||||
import io
|
||||
import re
|
||||
import sys
|
||||
import zipfile
|
||||
|
||||
import pandas as pd
|
||||
import requests
|
||||
|
||||
sys.path.insert(0, "pipeline/plugins/extractors/tap-uk-ees")
|
||||
sys.path.insert(0, "pipeline/plugins/extractors/tap-uk-ofsted")
|
||||
from tap_uk_ees.tap import ( # noqa: E402
|
||||
_KS2_NATIONAL_COL_MAP,
|
||||
_KS2_NATIONAL_CSV_URL,
|
||||
download_release_zip,
|
||||
get_all_releases,
|
||||
)
|
||||
from tap_uk_ofsted.tap import discover_csv_url # noqa: E402
|
||||
|
||||
TIMEOUT = 120
|
||||
|
||||
|
||||
def check_national_gps_science():
|
||||
print("\n=== (a) National catalogue CSV: GPS/science columns ===")
|
||||
resp = requests.get(_KS2_NATIONAL_CSV_URL, timeout=TIMEOUT)
|
||||
resp.raise_for_status()
|
||||
df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False)
|
||||
df.columns = [c.strip().lower() for c in df.columns]
|
||||
for csv_col in ("pt_gps_exp", "pt_scita_exp", "avg_readscore", "avg_matscore", "avg_gpsscore"):
|
||||
status = "PRESENT" if csv_col in df.columns else "MISSING"
|
||||
print(f" {csv_col}: {status}")
|
||||
gps_like = [c for c in df.columns if "gps" in c or "scita" in c or "sci" in c]
|
||||
print(f" all gps/science-ish columns: {gps_like}")
|
||||
nat = df[df.get("geographic_level", "").str.strip().str.lower() == "national"]
|
||||
print(f" national rows time_periods: {sorted(nat['time_period'].unique())}")
|
||||
# Sample the values our map would read for the latest year
|
||||
latest = nat[nat["time_period"] == nat["time_period"].max()]
|
||||
for csv_col, field in _KS2_NATIONAL_COL_MAP.items():
|
||||
val = latest.iloc[0].get(csv_col, "<col missing>") if len(latest) else "<no row>"
|
||||
print(f" {field} <- {csv_col} = {val!r}")
|
||||
|
||||
|
||||
def check_ks2_attainment_years_subjects():
|
||||
print("\n=== (b) EES KS2 attainment: years & subject labels ===")
|
||||
releases = get_all_releases("key-stage-2-attainment")
|
||||
print(f" releases found: {[r['time_period'] for r in releases]}")
|
||||
for release in releases:
|
||||
zf = download_release_zip(release["id"])
|
||||
name = next((n for n in zf.namelist()
|
||||
if "ks2_school_attainment_data" in n and n.endswith(".csv")), None)
|
||||
if not name:
|
||||
print(f" {release['time_period']}: NO school attainment CSV in ZIP")
|
||||
continue
|
||||
with zf.open(name) as f:
|
||||
df = pd.read_csv(f, dtype=str, keep_default_na=False, nrows=200000)
|
||||
years = sorted(df["time_period"].unique())
|
||||
subjects = sorted(df["subject"].unique())
|
||||
print(f" release {release['time_period']}: time_periods={years}")
|
||||
print(f" subjects={subjects}")
|
||||
|
||||
|
||||
def check_ofsted_report_card_columns():
|
||||
print("\n=== (c) Ofsted MI CSV: report-card columns ===")
|
||||
url = discover_csv_url()
|
||||
print(f" MI file: {url}")
|
||||
resp = requests.get(url, timeout=TIMEOUT)
|
||||
resp.raise_for_status()
|
||||
df = pd.read_csv(io.BytesIO(resp.content), dtype=str, keep_default_na=False, nrows=5)
|
||||
rc_like = [c for c in df.columns
|
||||
if re.search(r"report card|inclusion|curriculum|achievement|safeguard|well.?being|governance", c, re.I)]
|
||||
print(f" candidate report-card columns ({len(rc_like)}):")
|
||||
for c in rc_like:
|
||||
print(f" - {c!r}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
check_national_gps_science()
|
||||
check_ks2_attainment_years_subjects()
|
||||
check_ofsted_report_card_columns()
|
||||
```
|
||||
|
||||
Note: if `_KS2_NATIONAL_CSV_URL` is named differently in `tap_uk_ees/tap.py` (it is defined near the `_KS2_NATIONAL_COL_MAP` around line ~490), import whatever constant holds the catalogue CSV URL.
|
||||
|
||||
- [ ] **Step 2: Run it and record findings**
|
||||
|
||||
Run: `python pipeline/scripts/diagnose_compare_gaps.py 2>&1 | tee /tmp/compare-gaps-findings.txt`
|
||||
Expected: three sections printed. Paste the findings as a comment block at the bottom of the script (so they're committed evidence), e.g. `# FINDINGS 2026-07-12: pt_gps_exp MISSING (actual col: ...), 202122 present in release X, rc columns: [...]`.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add pipeline/scripts/diagnose_compare_gaps.py
|
||||
git commit -m "chore(pipeline): diagnostic for compare-screen data gaps"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Admissions preference detail → mart
|
||||
|
||||
Staging already extracts `second_preference_offers`, `third_preference_offers`, `total_offers` (`stg_ees_admissions.sql:26-29`) — the mart drops them. The cross-LA fields are declared in the tap (`all_applications_from_another_LA`, `offers_to_applicants_from_another_LA`) but not selected in staging.
|
||||
|
||||
**Files:**
|
||||
- Modify: `pipeline/transform/models/staging/stg_ees_admissions.sql` (after line 33, in `renamed`)
|
||||
- Modify: `pipeline/transform/models/marts/fact_admissions.sql`
|
||||
- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_admissions block, ~line 120)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces mart columns: `total_offers int`, `second_preference_offers int`, `third_preference_offers int`, `cross_la_applications int`, `cross_la_offers int`. The backend PR will map these in `FactAdmissions`.
|
||||
|
||||
- [ ] **Step 1: Add cross-LA columns to staging**
|
||||
|
||||
In `stg_ees_admissions.sql`, after the `first_preference_applications` line (line 33):
|
||||
|
||||
```sql
|
||||
-- Cross-borough demand: applications naming this school from families
|
||||
-- living in another local authority, and offers made to them.
|
||||
{{ safe_numeric('"all_applications_from_another_LA"') }}::integer as cross_la_applications,
|
||||
{{ safe_numeric('"offers_to_applicants_from_another_LA"') }}::integer as cross_la_offers,
|
||||
```
|
||||
|
||||
(Quote the identifiers — the tap emits them with mixed case, same trap as `FSM_eligible_percent`, see the header comment in that file. If `dbt parse` or the DAG run later shows the raw columns are lower-cased in Postgres, drop the double quotes.)
|
||||
|
||||
- [ ] **Step 2: Pass everything through the mart**
|
||||
|
||||
Replace the full select list in `fact_admissions.sql`:
|
||||
|
||||
```sql
|
||||
-- Mart: School admissions — one row per URN per year
|
||||
|
||||
select
|
||||
urn,
|
||||
year,
|
||||
school_phase,
|
||||
places_offered,
|
||||
total_offers,
|
||||
total_applications,
|
||||
first_preference_applications,
|
||||
first_preference_offers,
|
||||
second_preference_offers,
|
||||
third_preference_offers,
|
||||
cross_la_applications,
|
||||
cross_la_offers,
|
||||
first_preference_offer_pct,
|
||||
oversubscription_ratio,
|
||||
oversubscribed,
|
||||
admissions_policy
|
||||
from {{ ref('stg_ees_admissions') }}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add schema tests**
|
||||
|
||||
In `_marts_schema.yml` under `fact_admissions.columns`, append:
|
||||
|
||||
```yaml
|
||||
- name: second_preference_offers
|
||||
- name: third_preference_offers
|
||||
- name: cross_la_applications
|
||||
- name: cross_la_offers
|
||||
- name: total_offers
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Parse gate**
|
||||
|
||||
Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .`
|
||||
Expected: `Done.` with no compilation errors.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add pipeline/transform/models/staging/stg_ees_admissions.sql pipeline/transform/models/marts/fact_admissions.sql pipeline/transform/models/marts/_marts_schema.yml
|
||||
git commit -m "feat(pipeline): admissions preference breakdown and cross-LA demand in marts"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: KS2 progress confidence intervals + writing working-towards
|
||||
|
||||
The tap already emits `progress_measure_lower_conf_interval`, `progress_measure_upper_conf_interval`, `working_towards_expected_standard_pupil_percent` (tap.py:203-206). The staging pivot drops them. These power the CI-based Above/Average/Below progress chips (spec §8, first-review item on statistical honesty).
|
||||
|
||||
**Files:**
|
||||
- Modify: `pipeline/transform/models/staging/stg_ees_ks2.sql` (inside the `pivoted` CTE, next to each subject's `progress_measure_score` case, lines ~41/55/72, and in the final select ~lines 145-152)
|
||||
- Modify: `pipeline/transform/models/marts/fact_ks2_performance.sql`
|
||||
- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_ks2_performance block, ~line 82)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces mart columns: `reading_progress_lower_ci`, `reading_progress_upper_ci`, `writing_progress_lower_ci`, `writing_progress_upper_ci`, `maths_progress_lower_ci`, `maths_progress_upper_ci` (float), `writing_working_towards_pct` (float).
|
||||
|
||||
- [ ] **Step 1: Add pivot cases in staging**
|
||||
|
||||
After the `reading_progress` case (line ~41), add:
|
||||
|
||||
```sql
|
||||
max(case when subject = 'Reading'
|
||||
and breakdown_topic = 'All pupils' and breakdown = 'Total'
|
||||
then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as reading_progress_lower_ci,
|
||||
max(case when subject = 'Reading'
|
||||
and breakdown_topic = 'All pupils' and breakdown = 'Total'
|
||||
then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as reading_progress_upper_ci,
|
||||
```
|
||||
|
||||
After the `writing_progress` case (line ~55), add:
|
||||
|
||||
```sql
|
||||
max(case when subject = 'Writing'
|
||||
and breakdown_topic = 'All pupils' and breakdown = 'Total'
|
||||
then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as writing_progress_lower_ci,
|
||||
max(case when subject = 'Writing'
|
||||
and breakdown_topic = 'All pupils' and breakdown = 'Total'
|
||||
then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as writing_progress_upper_ci,
|
||||
max(case when subject = 'Writing'
|
||||
and breakdown_topic = 'All pupils' and breakdown = 'Total'
|
||||
then {{ safe_numeric('working_towards_expected_standard_pupil_percent') }} end) as writing_working_towards_pct,
|
||||
```
|
||||
|
||||
After the `maths_progress` case (line ~72), add:
|
||||
|
||||
```sql
|
||||
max(case when subject = 'Maths'
|
||||
and breakdown_topic = 'All pupils' and breakdown = 'Total'
|
||||
then {{ safe_numeric('progress_measure_lower_conf_interval') }} end) as maths_progress_lower_ci,
|
||||
max(case when subject = 'Maths'
|
||||
and breakdown_topic = 'All pupils' and breakdown = 'Total'
|
||||
then {{ safe_numeric('progress_measure_upper_conf_interval') }} end) as maths_progress_upper_ci,
|
||||
```
|
||||
|
||||
Then add the seven new columns to the model's final select (next to the existing `p.reading_progress` / `p.writing_progress` / `p.maths_progress` lines ~145-152):
|
||||
|
||||
```sql
|
||||
p.reading_progress_lower_ci,
|
||||
p.reading_progress_upper_ci,
|
||||
p.writing_progress_lower_ci,
|
||||
p.writing_progress_upper_ci,
|
||||
p.writing_working_towards_pct,
|
||||
p.maths_progress_lower_ci,
|
||||
p.maths_progress_upper_ci,
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Pass through the mart**
|
||||
|
||||
In `fact_ks2_performance.sql`, add the same seven column names to the select list immediately after the existing `maths_progress` line (this mart selects staging columns by name; match the file's existing alias style — if columns are selected bare, add them bare).
|
||||
|
||||
- [ ] **Step 3: Schema tests**
|
||||
|
||||
In `_marts_schema.yml` under `fact_ks2_performance.columns`, append the seven names (no tests beyond presence — values are legitimately NULL for 2023/24+ since progress measures ended with 2022/23, spec §4.3):
|
||||
|
||||
```yaml
|
||||
- name: reading_progress_lower_ci
|
||||
- name: reading_progress_upper_ci
|
||||
- name: writing_progress_lower_ci
|
||||
- name: writing_progress_upper_ci
|
||||
- name: writing_working_towards_pct
|
||||
- name: maths_progress_lower_ci
|
||||
- name: maths_progress_upper_ci
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Parse gate**
|
||||
|
||||
Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .`
|
||||
Expected: `Done.`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add pipeline/transform/models/staging/stg_ees_ks2.sql pipeline/transform/models/marts/fact_ks2_performance.sql pipeline/transform/models/marts/_marts_schema.yml
|
||||
git commit -m "feat(pipeline): KS2 progress confidence intervals and writing working-towards"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: KS4 — Progress 8 banding and disadvantage gaps
|
||||
|
||||
The tap's `ees_ks4_info` stream already declares `progress8_banding` (DfE's own "well above average … well below average" label — the ready-made secondary chip), `attainment8_diffn` and `progress8_diffn` (tap.py:338-340). Wire them through staging into the mart.
|
||||
|
||||
**Files:**
|
||||
- Modify: `pipeline/transform/models/staging/stg_ees_ks4.sql` (the CTE that reads `ees_ks4_info` — the same one that already surfaces `sen_pct`; add three columns to its select and to the final joined select)
|
||||
- Modify: `pipeline/transform/models/marts/fact_ks4_performance.sql` (add after `progress_8_upper_ci`)
|
||||
- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (fact_ks4_performance block, ~line 93)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces mart columns: `progress_8_banding text`, `attainment_8_disadvantage_gap float`, `progress_8_disadvantage_gap float`.
|
||||
|
||||
- [ ] **Step 1: Staging — select from the info source**
|
||||
|
||||
In the info CTE of `stg_ees_ks4.sql` add:
|
||||
|
||||
```sql
|
||||
nullif(trim(progress8_banding), '') as progress_8_banding,
|
||||
{{ safe_numeric('attainment8_diffn') }} as attainment_8_disadvantage_gap,
|
||||
{{ safe_numeric('progress8_diffn') }} as progress_8_disadvantage_gap,
|
||||
```
|
||||
|
||||
and add the three names to the model's final select (aliased the same way the CTE's other columns are).
|
||||
|
||||
- [ ] **Step 2: Mart passthrough**
|
||||
|
||||
In `fact_ks4_performance.sql`, after the `progress_8_upper_ci,` line:
|
||||
|
||||
```sql
|
||||
progress_8_banding,
|
||||
attainment_8_disadvantage_gap,
|
||||
progress_8_disadvantage_gap,
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Schema tests** — append the three names under `fact_ks4_performance.columns`, plus an accepted-values guard that tolerates NULL:
|
||||
|
||||
```yaml
|
||||
- name: progress_8_banding
|
||||
tests:
|
||||
- accepted_values:
|
||||
values: ['Well above average', 'Above average', 'Average', 'Below average', 'Well below average']
|
||||
config:
|
||||
where: "progress_8_banding is not null"
|
||||
- name: attainment_8_disadvantage_gap
|
||||
- name: progress_8_disadvantage_gap
|
||||
```
|
||||
|
||||
(If the DAG run later shows different capitalisation in the data, fix the accepted values to match the data, not vice versa.)
|
||||
|
||||
- [ ] **Step 4: Parse gate** — `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add pipeline/transform/models/staging/stg_ees_ks4.sql pipeline/transform/models/marts/fact_ks4_performance.sql pipeline/transform/models/marts/_marts_schema.yml
|
||||
git commit -m "feat(pipeline): Progress 8 banding and KS4 disadvantage gaps in marts"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: National averages — 2015/16 row and GPS/science/scaled-score fix
|
||||
|
||||
Two changes. (1) `stg_ees_ks2_national.sql:34` filters `>= 201617`, which is exactly why the England line starts a year late (2015/16 RWM = 53% exists in the catalogue). (2) GPS/science expected are NULL in prod despite full end-to-end mapping — Task 1's findings say whether the catalogue CSV column names differ from `_KS2_NATIONAL_COL_MAP` (`pt_gps_exp`, `pt_scita_exp`) or whether values are suppressed at source.
|
||||
|
||||
**Files:**
|
||||
- Modify: `pipeline/transform/models/staging/stg_ees_ks2_national.sql:34`
|
||||
- Modify (conditional on Task 1 findings): `pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py` (`_KS2_NATIONAL_COL_MAP`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: a 201516 row in `marts.fact_ks2_national_averages`; non-NULL `gps_expected_pct`, `science_expected_pct`, `reading_avg_score`, `maths_avg_score`, `gps_avg_score` for years the DfE publishes them. Backend/frontend consume via `/api/national-averages` unchanged (additive year + newly non-NULL fields).
|
||||
|
||||
- [ ] **Step 1: Widen the year filter**
|
||||
|
||||
In `stg_ees_ks2_national.sql`, change line 34:
|
||||
|
||||
```sql
|
||||
and cast(trim(time_period) as integer) >= 201516
|
||||
```
|
||||
|
||||
(2015/16 was the first year of the current expected-standard tests; nothing earlier is comparable, so keep a floor.)
|
||||
|
||||
- [ ] **Step 2: Fix the column map per Task 1 findings**
|
||||
|
||||
If Task 1 reported the actual CSV column names for GPS/science/scaled scores differ, update `_KS2_NATIONAL_COL_MAP` in `tap.py` accordingly, e.g. (illustrative — use the diagnosed names):
|
||||
|
||||
```python
|
||||
_KS2_NATIONAL_COL_MAP = {
|
||||
# ... existing entries ...
|
||||
"pt_gps_exp": "gps_expected_pct", # replace key with diagnosed name
|
||||
"pt_scita_exp": "science_expected_pct", # replace key with diagnosed name
|
||||
}
|
||||
```
|
||||
|
||||
If Task 1 showed the columns are present but suppressed (`x`) at national level for all years, instead delete the two entries from the map, delete the corresponding lines from `stg_ees_ks2_national.sql` and `fact_ks2_national_averages.sql`, and record in the PR description that GPS/science England ticks stay "not in dataset" (the mockups already carry that caveat).
|
||||
|
||||
- [ ] **Step 3: Parse gate** — `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.`
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add pipeline/transform/models/staging/stg_ees_ks2_national.sql pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py
|
||||
git commit -m "fix(pipeline): include 2015/16 national averages; fix GPS/science national mapping"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Legacy KS2 — load the 2021/22 school-level year
|
||||
|
||||
School-level 2021/22 exists in DfE performance-tables archives (same source as the four legacy years already loaded) but in neither our legacy config (stops at 201819, `pipeline/meltano.yml:33-37`) nor EES (starts 2022/23) — unless Task 1's finding (b) showed an EES release carrying 202122, in which case skip this task and note why in the PR.
|
||||
|
||||
The legacy URLs point at the self-hosted filebrowser (`10.0.1.224:8081`) — **the 2021/22 DfE archive must be uploaded there first; this is the one human dependency in this plan.**
|
||||
|
||||
**Files:**
|
||||
- Modify: `pipeline/meltano.yml` (legacy_ks2_urls block, line ~33)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `raw.legacy_ks2` rows with `year = '202122'`, flowing through `stg_legacy_ks2` → `fact_ks2_performance` unchanged (the stream maps old column names already; 2021/22 CSVs use the same `PTRWM_EXP`-style headers as 2018/19).
|
||||
|
||||
- [ ] **Step 1: Verify the 2021/22 CSV headers match `_LEGACY_KS2_COLUMN_MAP`**
|
||||
|
||||
Download the DfE 2021/22 KS2 revised archive (gov.uk "Compare School Performance data download": 2021-2022 all-schools ZIP), then:
|
||||
|
||||
Run: `python -c "import zipfile,io,pandas as pd; zf=zipfile.ZipFile('/path/to/2021-2022.zip'); n=[x for x in zf.namelist() if 'ks2final' in x.lower() and x.endswith('.csv')][0]; df=pd.read_csv(zf.open(n), dtype=str, nrows=5); import sys; sys.path.insert(0,'pipeline/plugins/extractors/tap-uk-ees'); from tap_uk_ees.tap import _LEGACY_KS2_COLUMN_MAP as m; missing=[c for c in m if c not in df.columns]; print('missing legacy columns:', missing)"`
|
||||
Expected: `missing legacy columns: []` (progress columns `READPROG` etc. may legitimately be missing/blank in 2021/22 — acceptable, they load as NULL).
|
||||
|
||||
- [ ] **Step 2: Upload the archive to the filebrowser and add the config entry**
|
||||
|
||||
In `pipeline/meltano.yml` under `legacy_ks2_urls`, add (with the real share URL from the filebrowser upload):
|
||||
|
||||
```yaml
|
||||
"202122": "http://10.0.1.224:8081/filebrowser/api/public/dl/<SHARE_ID>?inline=true"
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add pipeline/meltano.yml
|
||||
git commit -m "feat(pipeline): load 2021/22 school-level KS2 from legacy performance tables"
|
||||
```
|
||||
|
||||
- [ ] **Step 4 (only if Task 1(b) showed 2022/23 subject labels differ):** widen the subject matchers in `stg_ees_ks2.sql` the same way GPS already is (`subject ilike '%grammar%' or subject = 'GPS'`), e.g. `subject in ('Reading', 'reading')` → use the diagnosed labels. Parse-gate and commit as `fix(pipeline): match 2022/23 KS2 subject labels`.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Ofsted report-card columns (rc_*)
|
||||
|
||||
Resolves the tap TODO (`stg_ofsted_inspections.sql:37`). The marts/backed columns already exist as stubs; this wires real values. Uses Task 1(c)'s confirmed MI column names — the candidates below follow the MI file's existing naming style and must be corrected against the diagnostic output.
|
||||
|
||||
**Files:**
|
||||
- Modify: `pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py` (COLUMN_PRIORITY ~line 19-72, schema ~line 100-114)
|
||||
- Create: `pipeline/transform/macros/parse_report_card_grade.sql`
|
||||
- Modify: `pipeline/transform/models/staging/stg_ofsted_inspections.sql:36-46`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces mart columns (already declared in `fact_ofsted_inspection`): `rc_safeguarding_met boolean`, and `rc_inclusion` … `rc_sixth_form` as integers on the 5-point scale `1=Exceptional, 2=Strong standard, 3=Expected standard, 4=Needs attention/Attention needed, 5=Urgent improvement`. The backend translates codes to labels (same pattern as `gias_codes.py`), verifying wording against Ofsted's published toolkit (spec §8.4).
|
||||
|
||||
- [ ] **Step 1: Add tap column mappings**
|
||||
|
||||
In `COLUMN_PRIORITY` add (replace candidate strings with Task 1(c)'s exact headers — keep them as priority lists so older files degrade to blank):
|
||||
|
||||
```python
|
||||
"rc_safeguarding_met": ["Report card safeguarding", "Safeguarding"],
|
||||
"rc_inclusion": ["Report card inclusion", "Inclusion"],
|
||||
"rc_curriculum_teaching": ["Report card curriculum and teaching", "Curriculum and teaching"],
|
||||
"rc_achievement": ["Report card achievement", "Achievement"],
|
||||
"rc_attendance_behaviour": ["Report card attendance and behaviour", "Attendance and behaviour"],
|
||||
"rc_personal_development": ["Report card personal development and well-being", "Personal development and well-being"],
|
||||
"rc_leadership_governance": ["Report card leadership and governance", "Leadership and governance"],
|
||||
"rc_early_years": ["Report card early years", "Early years"],
|
||||
"rc_sixth_form": ["Report card sixth form", "Sixth form"],
|
||||
```
|
||||
|
||||
And in the stream schema (next to `report_url`, ~line 114):
|
||||
|
||||
```python
|
||||
th.Property("rc_safeguarding_met", th.StringType),
|
||||
th.Property("rc_inclusion", th.StringType),
|
||||
th.Property("rc_curriculum_teaching", th.StringType),
|
||||
th.Property("rc_achievement", th.StringType),
|
||||
th.Property("rc_attendance_behaviour", th.StringType),
|
||||
th.Property("rc_personal_development", th.StringType),
|
||||
th.Property("rc_leadership_governance", th.StringType),
|
||||
th.Property("rc_early_years", th.StringType),
|
||||
th.Property("rc_sixth_form", th.StringType),
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write the grade-parsing macro**
|
||||
|
||||
`pipeline/transform/macros/parse_report_card_grade.sql`:
|
||||
|
||||
```sql
|
||||
{% macro parse_report_card_grade(column_name) %}
|
||||
case lower(trim(nullif({{ column_name }}, 'NULL')))
|
||||
when 'exceptional' then 1
|
||||
when 'strong standard' then 2
|
||||
when 'expected standard' then 3
|
||||
when 'needs attention' then 4
|
||||
when 'attention needed' then 4
|
||||
when 'urgent improvement' then 5
|
||||
end
|
||||
{% endmacro %}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Wire staging**
|
||||
|
||||
Replace `stg_ofsted_inspections.sql` lines 36-46 (the NULL stubs) with:
|
||||
|
||||
```sql
|
||||
-- Report Card fields (post-Nov 2025 framework), 5-point scale:
|
||||
-- 1 Exceptional · 2 Strong standard · 3 Expected standard
|
||||
-- · 4 Needs attention · 5 Urgent improvement
|
||||
(lower(trim(nullif(rc_safeguarding_met, 'NULL'))) = 'met') as rc_safeguarding_met,
|
||||
{{ parse_report_card_grade('rc_inclusion') }}::integer as rc_inclusion,
|
||||
{{ parse_report_card_grade('rc_curriculum_teaching') }}::integer as rc_curriculum_teaching,
|
||||
{{ parse_report_card_grade('rc_achievement') }}::integer as rc_achievement,
|
||||
{{ parse_report_card_grade('rc_attendance_behaviour') }}::integer as rc_attendance_behaviour,
|
||||
{{ parse_report_card_grade('rc_personal_development') }}::integer as rc_personal_development,
|
||||
{{ parse_report_card_grade('rc_leadership_governance') }}::integer as rc_leadership_governance,
|
||||
{{ parse_report_card_grade('rc_early_years') }}::integer as rc_early_years,
|
||||
{{ parse_report_card_grade('rc_sixth_form') }}::integer as rc_sixth_form,
|
||||
```
|
||||
|
||||
Note `rc_safeguarding_met` becomes boolean (NULL when blank) — matching `fact_ofsted_inspection`'s `rc_safeguarding_met` Boolean column. If `fact_ofsted_inspection.sql` casts these columns, align its casts too (inspect that model; it currently passes the text stubs through).
|
||||
|
||||
- [ ] **Step 4: Parse gate + tap smoke test**
|
||||
|
||||
Run: `cd pipeline/transform && python -m dbt.cli.main parse --profiles-dir .` → `Done.`
|
||||
Run: `python -c "import sys; sys.path.insert(0,'pipeline/plugins/extractors/tap-uk-ofsted'); from tap_uk_ofsted.tap import COLUMN_PRIORITY; assert 'rc_inclusion' in COLUMN_PRIORITY; print('ok')"` → `ok`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add pipeline/plugins/extractors/tap-uk-ofsted/tap_uk_ofsted/tap.py pipeline/transform/macros/parse_report_card_grade.sql pipeline/transform/models/staging/stg_ofsted_inspections.sql
|
||||
git commit -m "feat(pipeline): extract Ofsted report-card judgements (rc_* columns)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: PR + post-merge verification
|
||||
|
||||
**Files:** none new
|
||||
|
||||
- [ ] **Step 1: Push and open the PR** (Gitea — use the git credential helper + basic-auth API pattern; token-header auth 401s):
|
||||
|
||||
```bash
|
||||
git push -u origin feat/compare-data-foundation
|
||||
# then create the PR via the Gitea API with basic auth from `git credential fill`
|
||||
```
|
||||
|
||||
PR body: link spec §5/§8, list the new mart columns, note the Task 6 human dependency (filebrowser upload) and the Task 1 findings file.
|
||||
|
||||
- [ ] **Step 2: After merge, verify the DAG run picked everything up**
|
||||
|
||||
The daily/monthly DAGs rebuild the affected models (`pipeline/dags/school_data_pipeline.py`). Spot-check via the public API (production after promotion, staging first at stx.schoolcompare.co.uk — note external /api is broken at the staging proxy, so check staging from the host):
|
||||
|
||||
```bash
|
||||
# 2015/16 national row exists
|
||||
curl -sL "https://www.schoolcompare.co.uk/api/national-averages" | python3 -c "import json,sys; d=json.load(sys.stdin); assert any(r['year']==201516 and r['primary'] for r in d['by_year']), '2015/16 missing'; print('201516 ok')"
|
||||
# 2021/22 school rows exist (Barclay)
|
||||
curl -sL "https://www.schoolcompare.co.uk/api/schools/138690" | python3 -c "import json,sys; d=json.load(sys.stdin); ys=[r['year'] for r in d['yearly_data']]; assert 202122 in [int(y) for y in ys], ys; print('202122 ok')"
|
||||
```
|
||||
|
||||
(The admissions/CI/KS4/rc_* columns aren't API-visible until the backend PR maps them — verify those directly in Postgres from the pipeline host: `select count(*) from marts.fact_admissions where second_preference_offers is not null;` etc.)
|
||||
|
||||
- [ ] **Step 3: Update the spec** — tick off the §5 promotions this PR delivered (edit the spec's promotion list to note "landed in PR #NN") and commit to main via a docs PR or alongside the backend PR.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope (next plans)
|
||||
|
||||
1. **Backend PR:** map new columns in `backend/models.py`, extend `/api/compare` with supplementary blocks + `national_averages`, computed benchmarks (FSM/EAL/SEN/size medians, disadvantaged national average), CI-based progress banding, report-card label translation (verify against Ofsted toolkit), Ofsted provider-page URLs, graded-vs-ungraded surfacing.
|
||||
2. **Frontend PR:** rebuild `/compare` per the mockups + e2e journeys (promotion gate).
|
||||
3. **Separate bug fix:** third school's series not rendering on the current production chart.
|
||||
4. **Post-v1 (spec):** census ethnicity/young-carer promotion, IDACI display, attendance section, gender-split/absence tier-2 measures.
|
||||
5. **Already in marts, no work needed:** KS4 EBacc entry/APS, grade 5+ English & maths, Progress 8 CIs — `fact_ks4_performance` carries them today; only the backend needs to expose them.
|
||||
@@ -0,0 +1,177 @@
|
||||
# Compare Screen Redesign — Expert Data Review
|
||||
|
||||
**Date:** 2026-07-11
|
||||
**Reviewer:** subagent briefed as an English education-standards / DfE-Ofsted data expert
|
||||
**Subject:** desktop + mobile compare mockups and the redesign spec
|
||||
(`2026-07-11-compare-screen-redesign-design.md`)
|
||||
**Status:** first-pass must-fixes applied 2026-07-12; second-pass
|
||||
findings (below) applied 2026-07-12 — mockups + spec §4/§8 updated
|
||||
|
||||
## Must-fix
|
||||
|
||||
1. **COVID gap is wrong and drops a real results year.** KS2 tests were
|
||||
cancelled 2019/20 and 2020/21 only; they resumed in 2021/22 with
|
||||
published school-level results (England RWM ≈ 59%). The mockup charts
|
||||
omit 2021/22 entirely and the tooltip claims no tests were held
|
||||
2019/20–2021/22. Fix: add 2021/22 to axis and all series; shrink the
|
||||
gap band; optionally annotate 2021/22 with DfE's post-pandemic
|
||||
comparability caution.
|
||||
2. **Report-card at-a-glance summary miscounts areas.** Detail list has
|
||||
4 Strong / 2 Expected / 1 Attention needed + Safeguarding met, but
|
||||
the summary says "3 areas Expected standard" — it counts safeguarding
|
||||
as a graded area. Safeguarding is a separate binary judgement and
|
||||
must be excluded from rating counts.
|
||||
3. **"Where the offers went" derivation is unsound.** Places − 1st-pref
|
||||
offers ≠ "second or third choices": the residual can include 4th–6th
|
||||
preference offers (pan-London scheme) and LA-allocated children who
|
||||
didn't choose the school; and offers don't necessarily equal PAN.
|
||||
Use the real 2nd/3rd-preference fields being promoted from
|
||||
`raw.ees_admissions`; until then drop the row.
|
||||
4. **Ofsted timeline in the copy is wrong.** Overall grades were
|
||||
abolished September 2024, not November 2025; Sept 2024–Nov 2025
|
||||
inspections kept the four key judgements without an overall grade
|
||||
(ungraded inspections carried grades forward). Neither mockup shows
|
||||
the interim regime, which will dominate real comparisons. Fix copy
|
||||
and add an interim example.
|
||||
5. **Barclay's "published an overall grade only — no area-by-area
|
||||
detail" misdescribes inspections.** No inspection type does that; a
|
||||
2021 graded inspection necessarily had subgrades — the gap is in our
|
||||
dataset. If it was an ungraded (s8) inspection, "Outstanding" is a
|
||||
carried-forward grade and should say so. Fix: "We don't hold
|
||||
area-by-area detail for this inspection", and distinguish graded vs
|
||||
ungraded in the data model.
|
||||
|
||||
## Should-fix
|
||||
|
||||
6. Writing is teacher assessment, not a test — "national tests and
|
||||
teacher assessments"; note TA caveat on the Writing strip.
|
||||
7. Verify renewed-framework wording against Ofsted's final toolkit:
|
||||
likely "Needs attention" (not "Attention needed") and "Personal
|
||||
development and well-being" (which otherwise collides with the
|
||||
identically-named legacy judgement). Pin every label to the
|
||||
published toolkit.
|
||||
8. "Expected standard" now means two things on one page (Ofsted area
|
||||
rating vs KS2 measure) — disambiguate in tooltips.
|
||||
9. Disadvantaged row: DfE definition includes looked-after / previously
|
||||
looked-after children, not just FSM6; benchmark labels inconsistent
|
||||
across desktop/mobile; subgroup percentages need cohort sizes or a
|
||||
volatility threshold before chips are attached.
|
||||
10. "Trend, last 7 years" spans ten years; sparklines render the COVID
|
||||
gap as equal spacing (the exact defect the audit criticises) and
|
||||
"Improved: 52% → 87%" endpoint-cherry-picks a volatile series.
|
||||
11. At-a-glance "Getting a place" uses different metrics per school
|
||||
(Barclay is also oversubscribed on total preferences but shows a
|
||||
green chip). Standardise on first-preference success %. Explain the
|
||||
equal-preference rule; condition "living close by matters" on the
|
||||
school's actual oversubscription criteria.
|
||||
12. "457 applications for 180 places" = total preferences at any rank,
|
||||
not head-to-head applicants; lead with first preferences vs places.
|
||||
Add offers-vs-final-intake (waiting lists/appeals) caveat.
|
||||
13. Elmhurst's subgrade list is likely missing Early years provision
|
||||
(school has a nursery) — possible pipeline gap.
|
||||
14. "Ofsted rating" label is obsolete post-Sept-2024 — use "Latest
|
||||
Ofsted inspection"; check whether Oct 2021 is the latest inspection
|
||||
or merely the latest graded one.
|
||||
15. SEN: "EHCP plans" is redundant; 28% SEN support often indicates
|
||||
resourced provision — add a note; England SEN-support ≈ 14%, not 13%.
|
||||
|
||||
## Nice-to-have
|
||||
|
||||
16. Consistent labelling of official DfE vs dataset-computed benchmarks
|
||||
(and medians shouldn't be called averages inconsistently).
|
||||
17. England 2015/16 RWM (53%) exists in DfE publications — the null is
|
||||
a dataset gap; source it or the England line looks broken.
|
||||
18. "1 in 4 first choices missed out" — actually more than 1 in 4.
|
||||
19. "1,273 of 1,260 places (full)" is over capacity; capacity figures
|
||||
are often stale — say "at or above capacity".
|
||||
20. State the actual suppression rule (DfE: ≤5 pupils suppressed,
|
||||
small numbers rounded) instead of "a handful".
|
||||
21. Spec §4.3 progress chips can't exist for displayed years: KS2
|
||||
progress ended with 2022/23 (no KS1 baseline) and returns
|
||||
~2027/28 with the reception baseline. Make explicit in the spec.
|
||||
IDACI (spec §4.5) is absent from mockups; if shipped, caveat it
|
||||
describes pupils' neighbourhoods, not the school.
|
||||
22. Tooltips should give the official term "first preference" alongside
|
||||
the plain-English "first choice".
|
||||
|
||||
## Overall assessment (verbatim gist)
|
||||
|
||||
The bones are genuinely good by education-data standards —
|
||||
England-average anchoring, explicit non-comparability messaging across
|
||||
Ofsted regimes, refusal to synthesise an overall grade, time-true
|
||||
x-axis, neutral FSM/EAL framing — better than most commercial
|
||||
school-comparison sites. But items 1–5 are outright factual errors or
|
||||
misdescriptions that a well-informed parent or Ofsted would catch;
|
||||
the admissions section needs the most conceptual work (equal
|
||||
preference, preferences-vs-applicants, offers-vs-intake). Fix 1–5
|
||||
before user testing; the rest fold into the planned PRs.
|
||||
|
||||
---
|
||||
|
||||
# Second-pass review (2026-07-12)
|
||||
|
||||
Same reviewer, after the must-fixes and the new three-tier metric
|
||||
exposure model were applied.
|
||||
|
||||
## Verification of first-pass must-fixes
|
||||
|
||||
- **1 (COVID/2021/22): resolved.** Time-true axis, band covers only the
|
||||
cancelled years, England 58.7% consistent with official figures,
|
||||
dataset gaps break lines honestly; reading/maths England series all
|
||||
match published figures; RWM ≤ min(subject) checks pass.
|
||||
- **2 (report-card count): resolved** — safeguarding excluded, spec §8.2.
|
||||
- **3 (offers derivation): resolved** — row removed, spec §8.3 bans it.
|
||||
- **4 (Ofsted timeline): resolved on desktop; mobile omits the interim
|
||||
regime clause** (see finding 6).
|
||||
- **5 (Barclay explanation): resolved.**
|
||||
|
||||
## New findings
|
||||
|
||||
1. **Should-fix — scaled-score strip domain contradicts caption.**
|
||||
Caption says "scaled scores run 80–120", strips render 100–120;
|
||||
truncated domain exaggerates small gaps and below-100 averages
|
||||
would fall off the edge. Render 80–120, or caption the 100–120
|
||||
window honestly and define below-100 behaviour.
|
||||
2. **Should-fix — scaled-score England ticks (106/105/105) unsourced.**
|
||||
Plausible but hand-entered; verify against DfE 2024/25 tables and
|
||||
add loading official England scaled scores to the pipeline list
|
||||
(absent from §8.1/§8.6).
|
||||
3. **Should-fix — "Writing" listed under "Higher standard" in the
|
||||
picker.** Writing TA outcome is "greater depth" (GDS), never
|
||||
"higher standard". Label "Writing — greater depth (teacher
|
||||
assessment)"; tooltip the combined higher-standard composition.
|
||||
4. Nice — "grammar & punctuation" summary line drops "spelling" (GPS).
|
||||
5. Nice — science is teacher-assessed (no KS2 test since 2009) and
|
||||
coarse; tooltip it like writing; reconsider its tier-2 slot.
|
||||
6. **Should-fix — mobile Ofsted copy skips the interim regime**
|
||||
(Sept 2024–Nov 2025) that desktop explains. One clause fixes it.
|
||||
7. **Should-fix — benchmark provenance still inconsistent** (EAL
|
||||
tooltip unsourced; FSM/disadvantaged chips vs tooltips use three
|
||||
vocabularies; header note says all England averages are official).
|
||||
Adopt one house style: official = "England average", computed =
|
||||
"benchmark / typical state school (our dataset)". Also tighten EAL
|
||||
definition to census wording ("first language known or believed to
|
||||
be other than English").
|
||||
8. Nice — "community primaries" distance note attached to an academy
|
||||
(Elmhurst); say "non-faith primaries" or condition on policy field.
|
||||
9. Nice — "Improving since 2022" → "since 2022/23".
|
||||
10. Nice — England chart tooltips show decimals; §7 mandates whole
|
||||
percents.
|
||||
|
||||
## Residual gaps not covered by spec §8
|
||||
|
||||
11. Spec promises IDACI-in-words, Attendance section, and tier-2
|
||||
gender/absence that the mockups never show — mark post-v1 or
|
||||
demonstrate, so implementation scope is unambiguous.
|
||||
12. Add official England scaled-score averages to the pipeline task
|
||||
list.
|
||||
13. Add the writing/greater-depth terminology rule to §8.7.
|
||||
|
||||
## Verdict
|
||||
|
||||
All must-fixes genuinely resolved; the tier model is conceptually
|
||||
sound ("no measure is lost", honest dataset-gap breaks, grouped
|
||||
picker). Remaining issues are contained: one internal contradiction
|
||||
(80–120 vs 100–120), one provenance inconsistency, one terminology
|
||||
error (writing/GDS). With findings 1–3 and 6–7 addressed, the data
|
||||
framing is fit to put in front of parents.
|
||||
@@ -0,0 +1,318 @@
|
||||
# Compare Screen Redesign — Audit & Design
|
||||
|
||||
**Date:** 2026-07-11
|
||||
**Status:** Draft — awaiting review
|
||||
**Scope:** `/compare` page (nextjs-app), `/api/compare` endpoint (backend)
|
||||
|
||||
## 1. Audit of the current screen
|
||||
|
||||
The current compare page (`nextjs-app/components/ComparisonView.tsx`) is a
|
||||
single-metric analyst tool: a `<select>` with ~40 KS2/GCSE metrics, one
|
||||
line chart over time, and a year-by-year table — all for the one selected
|
||||
metric. Observed on production with 3 primary schools:
|
||||
|
||||
**What works**
|
||||
|
||||
- URL-shareable state (`?urns=…&metric=…`), native share sheet.
|
||||
- Phase tabs (primary/secondary) with sensible auto-detection.
|
||||
- Colour-coded school cards tied to chart series.
|
||||
- Metric descriptions from `/api/metrics` (single source of truth).
|
||||
|
||||
**What doesn't**
|
||||
|
||||
1. **Performance-only.** The database already holds Ofsted inspections,
|
||||
admissions/oversubscription history, pupil characteristics (FSM/EAL),
|
||||
SEN, deprivation (IDACI), finance, capacity, faith, gender, trust —
|
||||
none of it reaches the compare screen. `/api/compare` returns only
|
||||
`yearly_data` + minimal `school_info`, while `/api/schools/{urn}`
|
||||
already returns all supplementary blocks.
|
||||
2. **One metric at a time.** A parent must know which of ~40 metrics
|
||||
matters, select each in turn, and hold results in their head. There is
|
||||
no side-by-side overview and no way to see two dimensions at once.
|
||||
3. **No benchmarks.** Numbers float without anchors: is 79% RWM good?
|
||||
The DB has official national averages (`fact_ks2_national_averages`)
|
||||
but the page never shows them.
|
||||
4. **Domain jargon untranslated.** "GPS Expected %", "Progress scores",
|
||||
"RWM Combined" assume DfE literacy. The only plain-English help is one
|
||||
note for progress scores.
|
||||
5. **Raw numbers, no judgement support.** 87.0% vs 92.0% vs 79.0% — the
|
||||
page never says "all three are well above the England average of 62%",
|
||||
which is the fact a parent actually needs.
|
||||
6. **Bugs/paper cuts observed:** the third school's series did not render
|
||||
on the production chart despite table data (worth a separate fix);
|
||||
the COVID gap (2018/19 → 2022/23) renders as equal spacing with no
|
||||
annotation; table shows "87.0%" precision that implies false accuracy.
|
||||
|
||||
## 2. Data inventory (available vs shown)
|
||||
|
||||
| Domain | Source table | On detail page | On compare |
|
||||
|---|---|---|---|
|
||||
| KS2 attainment/progress | fact_ks2_performance | yes | **yes** (only thing shown) |
|
||||
| National averages | fact_ks2_national_averages | partial | no |
|
||||
| Ofsted (latest + subgrades + report-card fields) | fact_ofsted_inspection, dim_school | yes | no |
|
||||
| Admissions & oversubscription (multi-year) | fact_admissions | yes | no |
|
||||
| Pupil characteristics (FSM, EAL, gender split) | fact_pupil_characteristics | yes | no |
|
||||
| Context (SEN, disadvantaged, stability, absence) | fact_ks2_performance | via metric picker | buried in picker |
|
||||
| Deprivation (IDACI) | fact_deprivation | yes | no |
|
||||
| Finance (per-pupil spend) | fact_finance | yes | no |
|
||||
| School facts (capacity, faith, ages, trust, nursery, gender) | dim_school | yes | no |
|
||||
| Location/distance | dim_location | map | no |
|
||||
|
||||
## 3. Design goals
|
||||
|
||||
1. **Answer parent questions, in order:** Is it a good school (Ofsted)?
|
||||
Do children do well there (academics vs England)? Will my child get a
|
||||
place (admissions)? What is the school like (size, community, faith)?
|
||||
2. **Every number gets an anchor** — the England average, rendered as a
|
||||
consistent visual tick, plus a plain-English chip
|
||||
(Above / Close to / Below England average).
|
||||
3. **Plain English first, jargon on demand.** Labels are questions or
|
||||
sentences ("Children reaching the expected standard in reading,
|
||||
writing and maths"), codes/acronyms live in tooltips.
|
||||
4. **Scan whole-picture first, drill down second.** The single-metric
|
||||
trend explorer survives, demoted to an "Explore trends" section at the
|
||||
bottom rather than being the entire page.
|
||||
|
||||
## 4. Proposed structure
|
||||
|
||||
Columns = schools (max 4 visible on desktop, horizontal scroll beyond),
|
||||
rows = dimensions. Sticky compact school header keeps column identity
|
||||
while scrolling. Sections, in order:
|
||||
|
||||
1. **At a glance** — verdict row per school: Ofsted badge, headline
|
||||
attainment vs England (dot strip + chip), oversubscription chip,
|
||||
size, distance (when a location is set).
|
||||
2. **Ofsted inspection** — must handle all three inspection regimes,
|
||||
which will coexist in comparisons for years:
|
||||
- **Legacy graded (pre-Sept 2024):** overall grade badge
|
||||
(Outstanding/Good/Requires improvement/Inadequate). Subgrades,
|
||||
where published, are rendered in the **same area-by-rating chip
|
||||
list UX as report cards** (one row per judgement area, rating as
|
||||
a chip) — one visual grammar for inspection detail across both
|
||||
regimes. Where our dataset has no subgrades for an inspection,
|
||||
say so honestly ("We don't hold area-by-area detail for this
|
||||
inspection") and point to the school's Ofsted page — never claim
|
||||
the inspection itself published no detail (graded inspections
|
||||
always have subgrades; if it was ungraded, the grade is
|
||||
carried forward and must be labelled as such).
|
||||
- **Interim ungraded (Sept 2024 – Nov 2025):** parsed outcome
|
||||
("remains Good") shown as the effective grade, marked as such.
|
||||
- **Renewed framework report card (from Nov 2025):** no overall
|
||||
grade exists. Render the report card as an area-by-rating list
|
||||
using Ofsted's 5-point scale (Exceptional / Strong standard /
|
||||
Expected standard / Attention needed / Urgent improvement) across
|
||||
the evaluation areas we model (`rc_inclusion`,
|
||||
`rc_curriculum_teaching`, `rc_achievement`,
|
||||
`rc_attendance_behaviour`, `rc_personal_development`,
|
||||
`rc_leadership_governance`, `rc_early_years`, `rc_sixth_form`)
|
||||
plus the separate safeguarding met/not-met flag. **At-a-glance
|
||||
summary rule:** never an unlabelled colour strip — summarise by
|
||||
counting areas per rating, best first ("5 areas Strong standard ·
|
||||
3 areas Expected standard"), and always name any area rated
|
||||
Attention needed or Urgent improvement explicitly (never fold
|
||||
problems into a count), plus "Safeguarding not met" whenever that
|
||||
flag is false. When everything is Expected standard or better,
|
||||
add the reassurance line "No areas need attention".
|
||||
When a comparison mixes regimes, show a one-line comparability note
|
||||
("Ofsted changed how it reports in Nov 2025 — a report card and an
|
||||
older overall grade aren't directly comparable"). Never derive a
|
||||
fake overall grade from report-card areas.
|
||||
3. **Academics (KS2)** — one dot-strip row per headline measure (RWM
|
||||
expected, RWM higher, reading/writing/maths expected), each with the
|
||||
England-average tick and per-school dots; copy must say "tests and
|
||||
teacher assessments" (writing is TA, not a test). Progress scores
|
||||
translated to Above/Average/Below chips (CI-based) — **but note KS2
|
||||
progress measures ended with 2022/23** (no KS1 baseline afterwards)
|
||||
and return only when the reception-baseline cohort reaches Y6
|
||||
(~2027/28), so progress chips apply to historical years in the
|
||||
trends explorer, not the headline view. Sparkline per school over
|
||||
the full published period, with an honest gap for the cancelled
|
||||
test years (2019/20–2020/21). Disadvantaged-pupils row under an
|
||||
"Equity" subheading, always with cohort size shown and DfE's full
|
||||
definition (FSM6 **or** looked-after/previously looked-after).
|
||||
4. **Getting a place** — oversubscription ratio as plain sentence
|
||||
("184 applications for 80 places"), first-preference success %, trend
|
||||
vs last year, admissions policy.
|
||||
5. **Who goes there** — pupils on roll (vs capacity), boys/girls, FSM %,
|
||||
EAL %, SEN support %, faith, ages, nursery, trust. *Post-v1:* IDACI
|
||||
decile in words (needs a coverage check of `fact_deprivation` and
|
||||
the neighbourhood-not-school caveat, §8.7).
|
||||
6. **Attendance** — *post-v1.* The KS2 test-day absence fields are the
|
||||
only per-school absence data we hold; they're near-zero for most
|
||||
schools and easy to misread as general attendance. Ship only if a
|
||||
general-absence source lands.
|
||||
7. **Explore trends** (existing feature, collapsed) — metric picker +
|
||||
multi-year line chart + table, with an added England-average
|
||||
reference line and a COVID-gap annotation.
|
||||
|
||||
**Metric exposure model (three tiers).** No measure from the current
|
||||
page is lost; they surface at three levels of prominence:
|
||||
- **Tier 1 — headline strips (always visible):** RWM expected,
|
||||
reading/writing/maths expected, RWM higher standard.
|
||||
- **Tier 2 — "More measures" expansion inside Academics:** GPS and
|
||||
science expected % (science labelled teacher-assessed), average
|
||||
scaled scores (reading/maths/GPS, same dot-strip grammar showing
|
||||
the 100–120 window of the 80–120 scale, widening below 100, with
|
||||
the England tick) — one tap/click away, same visual language.
|
||||
*Post-v1:* gender split and absence (see §4.6).
|
||||
- **Tier 3 — Explore trends:** the full grouped catalogue (the
|
||||
current page's ~40 metrics, including equity and school-context
|
||||
measures, and the GCSE set for secondary phase) drives the
|
||||
year-by-year chart and table via the grouped metric picker.
|
||||
The tier assignment is a content decision per phase (secondary:
|
||||
Attainment 8, Progress 8 banding, grade 5+ English & maths as tier 1;
|
||||
EBacc and subject entries as tier 2).
|
||||
|
||||
Finance (per-pupil spend) is deliberately deferred: low parent value,
|
||||
risk of misreading. Revisit later.
|
||||
|
||||
**Mobile (design target — mobile first):** the desktop grid is the
|
||||
adaptation, not the other way round. On mobile the layout goes
|
||||
*measure-first*: each row is one measure with all schools listed under
|
||||
it (colour dot + short name + value + chip), so comparison never
|
||||
requires horizontal swiping between school cards. A sticky horizontal
|
||||
school-chip bar keeps identity and add/remove available while
|
||||
scrolling. Dot strips already read measure-first and carry over
|
||||
unchanged. The trend chart scrolls horizontally inside its container.
|
||||
|
||||
## 5. Data strategy — existing dataset only
|
||||
|
||||
Constraint (agreed 2026-07-11): use only data already in marts plus
|
||||
fields already present in the `raw` schema extracts we pull today.
|
||||
No new external sources.
|
||||
|
||||
**Gaps in the mockup, resolved within this constraint:**
|
||||
|
||||
| Mockup element | Resolution |
|
||||
|---|---|
|
||||
| England average for disadvantaged pupils | Compute from our own data: `stg_ees_ks2` already pivots the Disadvantaged breakdown per school; aggregate it (weighted by eligible pupils) into `fact_ks2_national_averages` or compute in the API. Label it "England average (state schools)". |
|
||||
| England context for FSM / EAL / SEN chips | Compute dataset-wide medians per phase, same pattern as `/api/national-averages` does for KS4. |
|
||||
| "Much larger than average" size label | Dataset median pupils-on-roll per phase. |
|
||||
| Ofsted link | We don't have deep links to the latest report, so always link to the school's Ofsted provider page, `https://reports.ofsted.gov.uk/provider/21/{urn}`, derived from URN (label it "the school's Ofsted page", not "the report"). |
|
||||
|
||||
**Raw fields we already pull but don't store — promote to marts (one
|
||||
dbt/pipeline PR, no tap changes):**
|
||||
|
||||
- `raw.ees_admissions`: 2nd/3rd preference applications and offers,
|
||||
total-preference counts, cross-LA applications and offers → richer
|
||||
"Getting a place" (e.g. "offers reached 2nd-choice families",
|
||||
competition from outside the borough).
|
||||
- `raw.ees_ks2_attainment`: progress-measure confidence intervals and
|
||||
"working towards" % → lets the Above/Average/Below progress chips be
|
||||
statistically honest (band by CI overlap with 0, mirroring DfE
|
||||
methodology) instead of thresholding the point estimate.
|
||||
- `raw.ees_ks4_performance` / `ees_ks4_info`: `progress8_banding`
|
||||
(DfE's own plain-English "well above average … well below average"
|
||||
label — exactly the chip we want for secondary), EBacc entry/APS,
|
||||
grade-5+ English & maths, `attainment8_diffn`/`progress8_diffn`
|
||||
(disadvantage gaps) → the secondary-phase version of the Academics
|
||||
section.
|
||||
- `raw.ees_census`: young-carer % and the ethnicity breakdown →
|
||||
optional "Who goes there" enrichment; hold for a later iteration
|
||||
(presentation needs care), but the data requires no new extract.
|
||||
- `raw.ofsted_inspections` / tap-uk-ofsted: the `rc_*` report-card
|
||||
columns exist in staging/marts but are stubbed `null` — the tap has a
|
||||
TODO to map the report-card column names from the Ofsted MI file
|
||||
(same monthly extract we already download; inspections from Nov 2025
|
||||
onward carry them). This is the one promotion that needs a small tap
|
||||
schema addition, and it's a prerequisite for the new-framework Ofsted
|
||||
display above.
|
||||
|
||||
Explicitly out (not in any current extract): school-level phonics,
|
||||
workforce/teacher data, per-school attendance beyond the KS2 test-day
|
||||
absence fields, Ofsted report-card documents themselves.
|
||||
|
||||
## 6. API changes
|
||||
|
||||
Extend `GET /api/compare` response per URN with the same supplementary
|
||||
blocks the detail endpoint already builds (`get_supplementary_data`):
|
||||
`ofsted`, `census`, `admissions` (+ `admissions_history`), `deprivation`,
|
||||
plus a top-level `national_averages` block for the latest year. Reuse the
|
||||
existing function; no new tables. Response stays backward-compatible
|
||||
(additive fields only). Add derived helper fields server-side or compute
|
||||
chips client-side from `national_averages` (client-side preferred — no
|
||||
schema churn).
|
||||
|
||||
## 7. Accessibility & comprehension devices
|
||||
|
||||
- Verdict chips are text + colour + position (never colour alone).
|
||||
- Every acronym has a tooltip using existing `MetricTooltip`.
|
||||
- "How to read this" one-liner at the top of each section.
|
||||
- Chart palette: coral `#e07256`, teal `#00949b`, purple `#8664c9`
|
||||
(validated: lightness band, chroma, CVD separation, contrast — the
|
||||
current `--chart-2/-4` tokens fail chroma/contrast checks and should
|
||||
be nudged to these).
|
||||
- Numbers rounded to whole percents; England tick labelled on first use.
|
||||
|
||||
## 8. Expert-review requirements
|
||||
|
||||
An adversarial review by an education-data expert (full findings in
|
||||
`2026-07-11-compare-screen-expert-review.md`) was applied to the
|
||||
mockups on 2026-07-12. The following are binding requirements for
|
||||
implementation, beyond what the mockups can show:
|
||||
|
||||
1. **Chart truthfulness:** KS2 tests were cancelled 2019/20–2020/21
|
||||
only; 2021/22 results exist and must be loaded (school-level rows
|
||||
and the 2015/16 national figure, 53%, are currently missing from
|
||||
our dataset — pipeline task). Load official England averages for
|
||||
GPS/science expected % and the scaled scores (reading/maths/GPS) —
|
||||
the mockups' scaled-score ticks are indicative until then. Never
|
||||
render missing years as if time were continuous.
|
||||
2. **Report-card summaries** count graded areas only — safeguarding is
|
||||
a separate binary flag, never included in rating counts.
|
||||
3. **Admissions:** use the real preference-breakdown fields from
|
||||
`raw.ees_admissions`; never derive "lower-preference offers" as
|
||||
places − first-preference offers. Frame total applications as
|
||||
"named on N forms" (any rank), lead with first-preference success,
|
||||
and standardise at-a-glance chips on that one metric. Explain the
|
||||
equal-preference rule; caveat offers vs final intake (waiting
|
||||
lists/appeals); condition "distance decides" on the school's actual
|
||||
oversubscription criteria where we have the admissions-policy field.
|
||||
4. **Ofsted:** overall grades ended September 2024 (report cards from
|
||||
November 2025); the interim regime must be renderable. Distinguish
|
||||
graded (s5) vs ungraded (s8) inspections and surface carried-forward
|
||||
grades as such; "we don't hold the detail" is a statement about our
|
||||
dataset, never about the inspection. Verify every scale/area label
|
||||
against Ofsted's final published toolkit before launch (e.g. "Needs
|
||||
attention" vs "Attention needed"; "Personal development and
|
||||
well-being" vs the identically-named legacy judgement). Check
|
||||
whether a school's latest inspection is merely its latest *graded*
|
||||
one. Confirm Early years provision subgrades flow through the
|
||||
pipeline for schools with nurseries.
|
||||
5. **Subgroup honesty:** disadvantaged-pupil percentages carry cohort
|
||||
sizes and follow the DfE suppression rule (≤5 pupils suppressed);
|
||||
state the rule verbatim in the footer.
|
||||
6. **Benchmark provenance:** official DfE figures and
|
||||
dataset-computed benchmarks must be labelled distinctly and
|
||||
consistently everywhere (a computed median is a "benchmark",
|
||||
not an "England average").
|
||||
7. **Copy details:** "Latest Ofsted inspection" (not "Ofsted rating");
|
||||
"EHC plans"; SEN-support benchmark ≈14%; high SEN share may
|
||||
indicate resourced provision (say so neutrally); "at or above
|
||||
capacity" rather than "full" (capacity data is often stale);
|
||||
disambiguate Ofsted's "Expected standard" from the KS2 measure;
|
||||
give official terms ("first preference") alongside plain English.
|
||||
Writing has no "higher standard" — its TA outcome is "greater
|
||||
depth (GDS)"; never list writing under a higher-standard group.
|
||||
Science and writing are teacher-assessed and must be labelled as
|
||||
such (no KS2 science test since 2009). House style for benchmark
|
||||
provenance: official DfE figures say "England average"; computed
|
||||
figures say "state-school average (computed from our dataset)" —
|
||||
applied to every chip, tooltip, header note and section intro.
|
||||
EAL uses the census wording: first language known or believed to
|
||||
be other than English. If IDACI ships, caveat that it describes
|
||||
pupils' home neighbourhoods, not the school.
|
||||
|
||||
## 9. Rollout
|
||||
|
||||
1. **PR 1 (backend):** extend `/api/compare` + tests.
|
||||
2. **PR 2 (frontend):** new compare layout behind the existing route;
|
||||
e2e journey updated in the same PR (promotion gate).
|
||||
3. **Fix separately:** missing third series on the current chart.
|
||||
|
||||
## 10. Open questions for review
|
||||
|
||||
- Max schools: keep 10 in API but cap visible columns at 4 with scroll?
|
||||
- Should distance-from-home appear when the user searched by postcode
|
||||
(data exists via `dim_location`)?
|
||||
- Keep finance out of v1? (Recommended: yes, out.)
|
||||
Reference in New Issue
Block a user