docs: compare-screen redesign spec, expert review, and data-foundation plan

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
This commit is contained in:
Tudor
2026-07-12 21:19:53 +01:00
co-authored by Claude Fable 5
parent 58e90fef61
commit 297bdbd12e
3 changed files with 1086 additions and 0 deletions
@@ -0,0 +1,177 @@
# Compare Screen Redesign — Expert Data Review
**Date:** 2026-07-11
**Reviewer:** subagent briefed as an English education-standards / DfE-Ofsted data expert
**Subject:** desktop + mobile compare mockups and the redesign spec
(`2026-07-11-compare-screen-redesign-design.md`)
**Status:** first-pass must-fixes applied 2026-07-12; second-pass
findings (below) applied 2026-07-12 — mockups + spec §4/§8 updated
## Must-fix
1. **COVID gap is wrong and drops a real results year.** KS2 tests were
cancelled 2019/20 and 2020/21 only; they resumed in 2021/22 with
published school-level results (England RWM ≈ 59%). The mockup charts
omit 2021/22 entirely and the tooltip claims no tests were held
2019/202021/22. Fix: add 2021/22 to axis and all series; shrink the
gap band; optionally annotate 2021/22 with DfE's post-pandemic
comparability caution.
2. **Report-card at-a-glance summary miscounts areas.** Detail list has
4 Strong / 2 Expected / 1 Attention needed + Safeguarding met, but
the summary says "3 areas Expected standard" — it counts safeguarding
as a graded area. Safeguarding is a separate binary judgement and
must be excluded from rating counts.
3. **"Where the offers went" derivation is unsound.** Places 1st-pref
offers ≠ "second or third choices": the residual can include 4th6th
preference offers (pan-London scheme) and LA-allocated children who
didn't choose the school; and offers don't necessarily equal PAN.
Use the real 2nd/3rd-preference fields being promoted from
`raw.ees_admissions`; until then drop the row.
4. **Ofsted timeline in the copy is wrong.** Overall grades were
abolished September 2024, not November 2025; Sept 2024Nov 2025
inspections kept the four key judgements without an overall grade
(ungraded inspections carried grades forward). Neither mockup shows
the interim regime, which will dominate real comparisons. Fix copy
and add an interim example.
5. **Barclay's "published an overall grade only — no area-by-area
detail" misdescribes inspections.** No inspection type does that; a
2021 graded inspection necessarily had subgrades — the gap is in our
dataset. If it was an ungraded (s8) inspection, "Outstanding" is a
carried-forward grade and should say so. Fix: "We don't hold
area-by-area detail for this inspection", and distinguish graded vs
ungraded in the data model.
## Should-fix
6. Writing is teacher assessment, not a test — "national tests and
teacher assessments"; note TA caveat on the Writing strip.
7. Verify renewed-framework wording against Ofsted's final toolkit:
likely "Needs attention" (not "Attention needed") and "Personal
development and well-being" (which otherwise collides with the
identically-named legacy judgement). Pin every label to the
published toolkit.
8. "Expected standard" now means two things on one page (Ofsted area
rating vs KS2 measure) — disambiguate in tooltips.
9. Disadvantaged row: DfE definition includes looked-after / previously
looked-after children, not just FSM6; benchmark labels inconsistent
across desktop/mobile; subgroup percentages need cohort sizes or a
volatility threshold before chips are attached.
10. "Trend, last 7 years" spans ten years; sparklines render the COVID
gap as equal spacing (the exact defect the audit criticises) and
"Improved: 52% → 87%" endpoint-cherry-picks a volatile series.
11. At-a-glance "Getting a place" uses different metrics per school
(Barclay is also oversubscribed on total preferences but shows a
green chip). Standardise on first-preference success %. Explain the
equal-preference rule; condition "living close by matters" on the
school's actual oversubscription criteria.
12. "457 applications for 180 places" = total preferences at any rank,
not head-to-head applicants; lead with first preferences vs places.
Add offers-vs-final-intake (waiting lists/appeals) caveat.
13. Elmhurst's subgrade list is likely missing Early years provision
(school has a nursery) — possible pipeline gap.
14. "Ofsted rating" label is obsolete post-Sept-2024 — use "Latest
Ofsted inspection"; check whether Oct 2021 is the latest inspection
or merely the latest graded one.
15. SEN: "EHCP plans" is redundant; 28% SEN support often indicates
resourced provision — add a note; England SEN-support ≈ 14%, not 13%.
## Nice-to-have
16. Consistent labelling of official DfE vs dataset-computed benchmarks
(and medians shouldn't be called averages inconsistently).
17. England 2015/16 RWM (53%) exists in DfE publications — the null is
a dataset gap; source it or the England line looks broken.
18. "1 in 4 first choices missed out" — actually more than 1 in 4.
19. "1,273 of 1,260 places (full)" is over capacity; capacity figures
are often stale — say "at or above capacity".
20. State the actual suppression rule (DfE: ≤5 pupils suppressed,
small numbers rounded) instead of "a handful".
21. Spec §4.3 progress chips can't exist for displayed years: KS2
progress ended with 2022/23 (no KS1 baseline) and returns
~2027/28 with the reception baseline. Make explicit in the spec.
IDACI (spec §4.5) is absent from mockups; if shipped, caveat it
describes pupils' neighbourhoods, not the school.
22. Tooltips should give the official term "first preference" alongside
the plain-English "first choice".
## Overall assessment (verbatim gist)
The bones are genuinely good by education-data standards —
England-average anchoring, explicit non-comparability messaging across
Ofsted regimes, refusal to synthesise an overall grade, time-true
x-axis, neutral FSM/EAL framing — better than most commercial
school-comparison sites. But items 15 are outright factual errors or
misdescriptions that a well-informed parent or Ofsted would catch;
the admissions section needs the most conceptual work (equal
preference, preferences-vs-applicants, offers-vs-intake). Fix 15
before user testing; the rest fold into the planned PRs.
---
# Second-pass review (2026-07-12)
Same reviewer, after the must-fixes and the new three-tier metric
exposure model were applied.
## Verification of first-pass must-fixes
- **1 (COVID/2021/22): resolved.** Time-true axis, band covers only the
cancelled years, England 58.7% consistent with official figures,
dataset gaps break lines honestly; reading/maths England series all
match published figures; RWM ≤ min(subject) checks pass.
- **2 (report-card count): resolved** — safeguarding excluded, spec §8.2.
- **3 (offers derivation): resolved** — row removed, spec §8.3 bans it.
- **4 (Ofsted timeline): resolved on desktop; mobile omits the interim
regime clause** (see finding 6).
- **5 (Barclay explanation): resolved.**
## New findings
1. **Should-fix — scaled-score strip domain contradicts caption.**
Caption says "scaled scores run 80120", strips render 100120;
truncated domain exaggerates small gaps and below-100 averages
would fall off the edge. Render 80120, or caption the 100120
window honestly and define below-100 behaviour.
2. **Should-fix — scaled-score England ticks (106/105/105) unsourced.**
Plausible but hand-entered; verify against DfE 2024/25 tables and
add loading official England scaled scores to the pipeline list
(absent from §8.1/§8.6).
3. **Should-fix — "Writing" listed under "Higher standard" in the
picker.** Writing TA outcome is "greater depth" (GDS), never
"higher standard". Label "Writing — greater depth (teacher
assessment)"; tooltip the combined higher-standard composition.
4. Nice — "grammar & punctuation" summary line drops "spelling" (GPS).
5. Nice — science is teacher-assessed (no KS2 test since 2009) and
coarse; tooltip it like writing; reconsider its tier-2 slot.
6. **Should-fix — mobile Ofsted copy skips the interim regime**
(Sept 2024Nov 2025) that desktop explains. One clause fixes it.
7. **Should-fix — benchmark provenance still inconsistent** (EAL
tooltip unsourced; FSM/disadvantaged chips vs tooltips use three
vocabularies; header note says all England averages are official).
Adopt one house style: official = "England average", computed =
"benchmark / typical state school (our dataset)". Also tighten EAL
definition to census wording ("first language known or believed to
be other than English").
8. Nice — "community primaries" distance note attached to an academy
(Elmhurst); say "non-faith primaries" or condition on policy field.
9. Nice — "Improving since 2022" → "since 2022/23".
10. Nice — England chart tooltips show decimals; §7 mandates whole
percents.
## Residual gaps not covered by spec §8
11. Spec promises IDACI-in-words, Attendance section, and tier-2
gender/absence that the mockups never show — mark post-v1 or
demonstrate, so implementation scope is unambiguous.
12. Add official England scaled-score averages to the pipeline task
list.
13. Add the writing/greater-depth terminology rule to §8.7.
## Verdict
All must-fixes genuinely resolved; the tier model is conceptually
sound ("no measure is lost", honest dataset-gap breaks, grouped
picker). Remaining issues are contained: one internal contradiction
(80120 vs 100120), one provenance inconsistency, one terminology
error (writing/GDS). With findings 13 and 67 addressed, the data
framing is fit to put in front of parents.
@@ -0,0 +1,318 @@
# Compare Screen Redesign — Audit & Design
**Date:** 2026-07-11
**Status:** Draft — awaiting review
**Scope:** `/compare` page (nextjs-app), `/api/compare` endpoint (backend)
## 1. Audit of the current screen
The current compare page (`nextjs-app/components/ComparisonView.tsx`) is a
single-metric analyst tool: a `<select>` with ~40 KS2/GCSE metrics, one
line chart over time, and a year-by-year table — all for the one selected
metric. Observed on production with 3 primary schools:
**What works**
- URL-shareable state (`?urns=…&metric=…`), native share sheet.
- Phase tabs (primary/secondary) with sensible auto-detection.
- Colour-coded school cards tied to chart series.
- Metric descriptions from `/api/metrics` (single source of truth).
**What doesn't**
1. **Performance-only.** The database already holds Ofsted inspections,
admissions/oversubscription history, pupil characteristics (FSM/EAL),
SEN, deprivation (IDACI), finance, capacity, faith, gender, trust —
none of it reaches the compare screen. `/api/compare` returns only
`yearly_data` + minimal `school_info`, while `/api/schools/{urn}`
already returns all supplementary blocks.
2. **One metric at a time.** A parent must know which of ~40 metrics
matters, select each in turn, and hold results in their head. There is
no side-by-side overview and no way to see two dimensions at once.
3. **No benchmarks.** Numbers float without anchors: is 79% RWM good?
The DB has official national averages (`fact_ks2_national_averages`)
but the page never shows them.
4. **Domain jargon untranslated.** "GPS Expected %", "Progress scores",
"RWM Combined" assume DfE literacy. The only plain-English help is one
note for progress scores.
5. **Raw numbers, no judgement support.** 87.0% vs 92.0% vs 79.0% — the
page never says "all three are well above the England average of 62%",
which is the fact a parent actually needs.
6. **Bugs/paper cuts observed:** the third school's series did not render
on the production chart despite table data (worth a separate fix);
the COVID gap (2018/19 → 2022/23) renders as equal spacing with no
annotation; table shows "87.0%" precision that implies false accuracy.
## 2. Data inventory (available vs shown)
| Domain | Source table | On detail page | On compare |
|---|---|---|---|
| KS2 attainment/progress | fact_ks2_performance | yes | **yes** (only thing shown) |
| National averages | fact_ks2_national_averages | partial | no |
| Ofsted (latest + subgrades + report-card fields) | fact_ofsted_inspection, dim_school | yes | no |
| Admissions & oversubscription (multi-year) | fact_admissions | yes | no |
| Pupil characteristics (FSM, EAL, gender split) | fact_pupil_characteristics | yes | no |
| Context (SEN, disadvantaged, stability, absence) | fact_ks2_performance | via metric picker | buried in picker |
| Deprivation (IDACI) | fact_deprivation | yes | no |
| Finance (per-pupil spend) | fact_finance | yes | no |
| School facts (capacity, faith, ages, trust, nursery, gender) | dim_school | yes | no |
| Location/distance | dim_location | map | no |
## 3. Design goals
1. **Answer parent questions, in order:** Is it a good school (Ofsted)?
Do children do well there (academics vs England)? Will my child get a
place (admissions)? What is the school like (size, community, faith)?
2. **Every number gets an anchor** — the England average, rendered as a
consistent visual tick, plus a plain-English chip
(Above / Close to / Below England average).
3. **Plain English first, jargon on demand.** Labels are questions or
sentences ("Children reaching the expected standard in reading,
writing and maths"), codes/acronyms live in tooltips.
4. **Scan whole-picture first, drill down second.** The single-metric
trend explorer survives, demoted to an "Explore trends" section at the
bottom rather than being the entire page.
## 4. Proposed structure
Columns = schools (max 4 visible on desktop, horizontal scroll beyond),
rows = dimensions. Sticky compact school header keeps column identity
while scrolling. Sections, in order:
1. **At a glance** — verdict row per school: Ofsted badge, headline
attainment vs England (dot strip + chip), oversubscription chip,
size, distance (when a location is set).
2. **Ofsted inspection** — must handle all three inspection regimes,
which will coexist in comparisons for years:
- **Legacy graded (pre-Sept 2024):** overall grade badge
(Outstanding/Good/Requires improvement/Inadequate). Subgrades,
where published, are rendered in the **same area-by-rating chip
list UX as report cards** (one row per judgement area, rating as
a chip) — one visual grammar for inspection detail across both
regimes. Where our dataset has no subgrades for an inspection,
say so honestly ("We don't hold area-by-area detail for this
inspection") and point to the school's Ofsted page — never claim
the inspection itself published no detail (graded inspections
always have subgrades; if it was ungraded, the grade is
carried forward and must be labelled as such).
- **Interim ungraded (Sept 2024 Nov 2025):** parsed outcome
("remains Good") shown as the effective grade, marked as such.
- **Renewed framework report card (from Nov 2025):** no overall
grade exists. Render the report card as an area-by-rating list
using Ofsted's 5-point scale (Exceptional / Strong standard /
Expected standard / Attention needed / Urgent improvement) across
the evaluation areas we model (`rc_inclusion`,
`rc_curriculum_teaching`, `rc_achievement`,
`rc_attendance_behaviour`, `rc_personal_development`,
`rc_leadership_governance`, `rc_early_years`, `rc_sixth_form`)
plus the separate safeguarding met/not-met flag. **At-a-glance
summary rule:** never an unlabelled colour strip — summarise by
counting areas per rating, best first ("5 areas Strong standard ·
3 areas Expected standard"), and always name any area rated
Attention needed or Urgent improvement explicitly (never fold
problems into a count), plus "Safeguarding not met" whenever that
flag is false. When everything is Expected standard or better,
add the reassurance line "No areas need attention".
When a comparison mixes regimes, show a one-line comparability note
("Ofsted changed how it reports in Nov 2025 — a report card and an
older overall grade aren't directly comparable"). Never derive a
fake overall grade from report-card areas.
3. **Academics (KS2)** — one dot-strip row per headline measure (RWM
expected, RWM higher, reading/writing/maths expected), each with the
England-average tick and per-school dots; copy must say "tests and
teacher assessments" (writing is TA, not a test). Progress scores
translated to Above/Average/Below chips (CI-based) — **but note KS2
progress measures ended with 2022/23** (no KS1 baseline afterwards)
and return only when the reception-baseline cohort reaches Y6
(~2027/28), so progress chips apply to historical years in the
trends explorer, not the headline view. Sparkline per school over
the full published period, with an honest gap for the cancelled
test years (2019/202020/21). Disadvantaged-pupils row under an
"Equity" subheading, always with cohort size shown and DfE's full
definition (FSM6 **or** looked-after/previously looked-after).
4. **Getting a place** — oversubscription ratio as plain sentence
("184 applications for 80 places"), first-preference success %, trend
vs last year, admissions policy.
5. **Who goes there** — pupils on roll (vs capacity), boys/girls, FSM %,
EAL %, SEN support %, faith, ages, nursery, trust. *Post-v1:* IDACI
decile in words (needs a coverage check of `fact_deprivation` and
the neighbourhood-not-school caveat, §8.7).
6. **Attendance***post-v1.* The KS2 test-day absence fields are the
only per-school absence data we hold; they're near-zero for most
schools and easy to misread as general attendance. Ship only if a
general-absence source lands.
7. **Explore trends** (existing feature, collapsed) — metric picker +
multi-year line chart + table, with an added England-average
reference line and a COVID-gap annotation.
**Metric exposure model (three tiers).** No measure from the current
page is lost; they surface at three levels of prominence:
- **Tier 1 — headline strips (always visible):** RWM expected,
reading/writing/maths expected, RWM higher standard.
- **Tier 2 — "More measures" expansion inside Academics:** GPS and
science expected % (science labelled teacher-assessed), average
scaled scores (reading/maths/GPS, same dot-strip grammar showing
the 100120 window of the 80120 scale, widening below 100, with
the England tick) — one tap/click away, same visual language.
*Post-v1:* gender split and absence (see §4.6).
- **Tier 3 — Explore trends:** the full grouped catalogue (the
current page's ~40 metrics, including equity and school-context
measures, and the GCSE set for secondary phase) drives the
year-by-year chart and table via the grouped metric picker.
The tier assignment is a content decision per phase (secondary:
Attainment 8, Progress 8 banding, grade 5+ English & maths as tier 1;
EBacc and subject entries as tier 2).
Finance (per-pupil spend) is deliberately deferred: low parent value,
risk of misreading. Revisit later.
**Mobile (design target — mobile first):** the desktop grid is the
adaptation, not the other way round. On mobile the layout goes
*measure-first*: each row is one measure with all schools listed under
it (colour dot + short name + value + chip), so comparison never
requires horizontal swiping between school cards. A sticky horizontal
school-chip bar keeps identity and add/remove available while
scrolling. Dot strips already read measure-first and carry over
unchanged. The trend chart scrolls horizontally inside its container.
## 5. Data strategy — existing dataset only
Constraint (agreed 2026-07-11): use only data already in marts plus
fields already present in the `raw` schema extracts we pull today.
No new external sources.
**Gaps in the mockup, resolved within this constraint:**
| Mockup element | Resolution |
|---|---|
| England average for disadvantaged pupils | Compute from our own data: `stg_ees_ks2` already pivots the Disadvantaged breakdown per school; aggregate it (weighted by eligible pupils) into `fact_ks2_national_averages` or compute in the API. Label it "England average (state schools)". |
| England context for FSM / EAL / SEN chips | Compute dataset-wide medians per phase, same pattern as `/api/national-averages` does for KS4. |
| "Much larger than average" size label | Dataset median pupils-on-roll per phase. |
| Ofsted link | We don't have deep links to the latest report, so always link to the school's Ofsted provider page, `https://reports.ofsted.gov.uk/provider/21/{urn}`, derived from URN (label it "the school's Ofsted page", not "the report"). |
**Raw fields we already pull but don't store — promote to marts (one
dbt/pipeline PR, no tap changes):**
- `raw.ees_admissions`: 2nd/3rd preference applications and offers,
total-preference counts, cross-LA applications and offers → richer
"Getting a place" (e.g. "offers reached 2nd-choice families",
competition from outside the borough).
- `raw.ees_ks2_attainment`: progress-measure confidence intervals and
"working towards" % → lets the Above/Average/Below progress chips be
statistically honest (band by CI overlap with 0, mirroring DfE
methodology) instead of thresholding the point estimate.
- `raw.ees_ks4_performance` / `ees_ks4_info`: `progress8_banding`
(DfE's own plain-English "well above average … well below average"
label — exactly the chip we want for secondary), EBacc entry/APS,
grade-5+ English & maths, `attainment8_diffn`/`progress8_diffn`
(disadvantage gaps) → the secondary-phase version of the Academics
section.
- `raw.ees_census`: young-carer % and the ethnicity breakdown →
optional "Who goes there" enrichment; hold for a later iteration
(presentation needs care), but the data requires no new extract.
- `raw.ofsted_inspections` / tap-uk-ofsted: the `rc_*` report-card
columns exist in staging/marts but are stubbed `null` — the tap has a
TODO to map the report-card column names from the Ofsted MI file
(same monthly extract we already download; inspections from Nov 2025
onward carry them). This is the one promotion that needs a small tap
schema addition, and it's a prerequisite for the new-framework Ofsted
display above.
Explicitly out (not in any current extract): school-level phonics,
workforce/teacher data, per-school attendance beyond the KS2 test-day
absence fields, Ofsted report-card documents themselves.
## 6. API changes
Extend `GET /api/compare` response per URN with the same supplementary
blocks the detail endpoint already builds (`get_supplementary_data`):
`ofsted`, `census`, `admissions` (+ `admissions_history`), `deprivation`,
plus a top-level `national_averages` block for the latest year. Reuse the
existing function; no new tables. Response stays backward-compatible
(additive fields only). Add derived helper fields server-side or compute
chips client-side from `national_averages` (client-side preferred — no
schema churn).
## 7. Accessibility & comprehension devices
- Verdict chips are text + colour + position (never colour alone).
- Every acronym has a tooltip using existing `MetricTooltip`.
- "How to read this" one-liner at the top of each section.
- Chart palette: coral `#e07256`, teal `#00949b`, purple `#8664c9`
(validated: lightness band, chroma, CVD separation, contrast — the
current `--chart-2/-4` tokens fail chroma/contrast checks and should
be nudged to these).
- Numbers rounded to whole percents; England tick labelled on first use.
## 8. Expert-review requirements
An adversarial review by an education-data expert (full findings in
`2026-07-11-compare-screen-expert-review.md`) was applied to the
mockups on 2026-07-12. The following are binding requirements for
implementation, beyond what the mockups can show:
1. **Chart truthfulness:** KS2 tests were cancelled 2019/202020/21
only; 2021/22 results exist and must be loaded (school-level rows
and the 2015/16 national figure, 53%, are currently missing from
our dataset — pipeline task). Load official England averages for
GPS/science expected % and the scaled scores (reading/maths/GPS) —
the mockups' scaled-score ticks are indicative until then. Never
render missing years as if time were continuous.
2. **Report-card summaries** count graded areas only — safeguarding is
a separate binary flag, never included in rating counts.
3. **Admissions:** use the real preference-breakdown fields from
`raw.ees_admissions`; never derive "lower-preference offers" as
places first-preference offers. Frame total applications as
"named on N forms" (any rank), lead with first-preference success,
and standardise at-a-glance chips on that one metric. Explain the
equal-preference rule; caveat offers vs final intake (waiting
lists/appeals); condition "distance decides" on the school's actual
oversubscription criteria where we have the admissions-policy field.
4. **Ofsted:** overall grades ended September 2024 (report cards from
November 2025); the interim regime must be renderable. Distinguish
graded (s5) vs ungraded (s8) inspections and surface carried-forward
grades as such; "we don't hold the detail" is a statement about our
dataset, never about the inspection. Verify every scale/area label
against Ofsted's final published toolkit before launch (e.g. "Needs
attention" vs "Attention needed"; "Personal development and
well-being" vs the identically-named legacy judgement). Check
whether a school's latest inspection is merely its latest *graded*
one. Confirm Early years provision subgrades flow through the
pipeline for schools with nurseries.
5. **Subgroup honesty:** disadvantaged-pupil percentages carry cohort
sizes and follow the DfE suppression rule (≤5 pupils suppressed);
state the rule verbatim in the footer.
6. **Benchmark provenance:** official DfE figures and
dataset-computed benchmarks must be labelled distinctly and
consistently everywhere (a computed median is a "benchmark",
not an "England average").
7. **Copy details:** "Latest Ofsted inspection" (not "Ofsted rating");
"EHC plans"; SEN-support benchmark ≈14%; high SEN share may
indicate resourced provision (say so neutrally); "at or above
capacity" rather than "full" (capacity data is often stale);
disambiguate Ofsted's "Expected standard" from the KS2 measure;
give official terms ("first preference") alongside plain English.
Writing has no "higher standard" — its TA outcome is "greater
depth (GDS)"; never list writing under a higher-standard group.
Science and writing are teacher-assessed and must be labelled as
such (no KS2 science test since 2009). House style for benchmark
provenance: official DfE figures say "England average"; computed
figures say "state-school average (computed from our dataset)" —
applied to every chip, tooltip, header note and section intro.
EAL uses the census wording: first language known or believed to
be other than English. If IDACI ships, caveat that it describes
pupils' home neighbourhoods, not the school.
## 9. Rollout
1. **PR 1 (backend):** extend `/api/compare` + tests.
2. **PR 2 (frontend):** new compare layout behind the existing route;
e2e journey updated in the same PR (promotion gate).
3. **Fix separately:** missing third series on the current chart.
## 10. Open questions for review
- Max schools: keep 10 in API but cap visible columns at 4 with scroll?
- Should distance-from-home appear when the user searched by postcode
(data exists via `dim_location`)?
- Keep finance out of v1? (Recommended: yes, out.)