school_compare/docs/superpowers/specs/2026-07-11-compare-screen-expert-review.md

Compare Screen Redesign — Expert Data Review

Date: 2026-07-11 Reviewer: subagent briefed as an English education-standards / DfE-Ofsted data expert Subject: desktop + mobile compare mockups and the redesign spec (2026-07-11-compare-screen-redesign-design.md) Status: first-pass must-fixes applied 2026-07-12; second-pass findings (below) applied 2026-07-12 — mockups + spec §4/§8 updated

Must-fix

1. COVID gap is wrong and drops a real results year. KS2 tests were cancelled 2019/20 and 2020/21 only; they resumed in 2021/22 with published school-level results (England RWM ≈ 59%). The mockup charts omit 2021/22 entirely and the tooltip claims no tests were held 2019/20–2021/22. Fix: add 2021/22 to axis and all series; shrink the gap band; optionally annotate 2021/22 with DfE's post-pandemic comparability caution.
2. Report-card at-a-glance summary miscounts areas. Detail list has 4 Strong / 2 Expected / 1 Attention needed + Safeguarding met, but the summary says "3 areas Expected standard" — it counts safeguarding as a graded area. Safeguarding is a separate binary judgement and must be excluded from rating counts.
3. "Where the offers went" derivation is unsound. Places − 1st-pref offers ≠ "second or third choices": the residual can include 4th–6th preference offers (pan-London scheme) and LA-allocated children who didn't choose the school; and offers don't necessarily equal PAN. Use the real 2nd/3rd-preference fields being promoted from raw.ees_admissions; until then drop the row.
4. Ofsted timeline in the copy is wrong. Overall grades were abolished September 2024, not November 2025; Sept 2024–Nov 2025 inspections kept the four key judgements without an overall grade (ungraded inspections carried grades forward). Neither mockup shows the interim regime, which will dominate real comparisons. Fix copy and add an interim example.
5. Barclay's "published an overall grade only — no area-by-area detail" misdescribes inspections. No inspection type does that; a 2021 graded inspection necessarily had subgrades — the gap is in our dataset. If it was an ungraded (s8) inspection, "Outstanding" is a carried-forward grade and should say so. Fix: "We don't hold area-by-area detail for this inspection", and distinguish graded vs ungraded in the data model.

Should-fix

6. Writing is teacher assessment, not a test — "national tests and teacher assessments"; note TA caveat on the Writing strip.
7. Verify renewed-framework wording against Ofsted's final toolkit: likely "Needs attention" (not "Attention needed") and "Personal development and well-being" (which otherwise collides with the identically-named legacy judgement). Pin every label to the published toolkit.
8. "Expected standard" now means two things on one page (Ofsted area rating vs KS2 measure) — disambiguate in tooltips.
9. Disadvantaged row: DfE definition includes looked-after / previously looked-after children, not just FSM6; benchmark labels inconsistent across desktop/mobile; subgroup percentages need cohort sizes or a volatility threshold before chips are attached.
10. "Trend, last 7 years" spans ten years; sparklines render the COVID gap as equal spacing (the exact defect the audit criticises) and "Improved: 52% → 87%" endpoint-cherry-picks a volatile series.
11. At-a-glance "Getting a place" uses different metrics per school (Barclay is also oversubscribed on total preferences but shows a green chip). Standardise on first-preference success %. Explain the equal-preference rule; condition "living close by matters" on the school's actual oversubscription criteria.
12. "457 applications for 180 places" = total preferences at any rank, not head-to-head applicants; lead with first preferences vs places. Add offers-vs-final-intake (waiting lists/appeals) caveat.
13. Elmhurst's subgrade list is likely missing Early years provision (school has a nursery) — possible pipeline gap.
14. "Ofsted rating" label is obsolete post-Sept-2024 — use "Latest Ofsted inspection"; check whether Oct 2021 is the latest inspection or merely the latest graded one.
15. SEN: "EHCP plans" is redundant; 28% SEN support often indicates resourced provision — add a note; England SEN-support ≈ 14%, not 13%.

Nice-to-have

16. Consistent labelling of official DfE vs dataset-computed benchmarks (and medians shouldn't be called averages inconsistently).
17. England 2015/16 RWM (53%) exists in DfE publications — the null is a dataset gap; source it or the England line looks broken.
18. "1 in 4 first choices missed out" — actually more than 1 in 4.
19. "1,273 of 1,260 places (full)" is over capacity; capacity figures are often stale — say "at or above capacity".
20. State the actual suppression rule (DfE: ≤5 pupils suppressed, small numbers rounded) instead of "a handful".
21. Spec §4.3 progress chips can't exist for displayed years: KS2 progress ended with 2022/23 (no KS1 baseline) and returns ~2027/28 with the reception baseline. Make explicit in the spec. IDACI (spec §4.5) is absent from mockups; if shipped, caveat it describes pupils' neighbourhoods, not the school.
22. Tooltips should give the official term "first preference" alongside the plain-English "first choice".

Overall assessment (verbatim gist)

The bones are genuinely good by education-data standards — England-average anchoring, explicit non-comparability messaging across Ofsted regimes, refusal to synthesise an overall grade, time-true x-axis, neutral FSM/EAL framing — better than most commercial school-comparison sites. But items 1–5 are outright factual errors or misdescriptions that a well-informed parent or Ofsted would catch; the admissions section needs the most conceptual work (equal preference, preferences-vs-applicants, offers-vs-intake). Fix 1–5 before user testing; the rest fold into the planned PRs.

---

Second-pass review (2026-07-12)

Same reviewer, after the must-fixes and the new three-tier metric exposure model were applied.

Verification of first-pass must-fixes

• 1 (COVID/2021/22): resolved. Time-true axis, band covers only the cancelled years, England 58.7% consistent with official figures, dataset gaps break lines honestly; reading/maths England series all match published figures; RWM ≤ min(subject) checks pass.
• 2 (report-card count): resolved — safeguarding excluded, spec §8.2.
• 3 (offers derivation): resolved — row removed, spec §8.3 bans it.
• 4 (Ofsted timeline): resolved on desktop; mobile omits the interim regime clause (see finding 6).
• 5 (Barclay explanation): resolved.

New findings

1. Should-fix — scaled-score strip domain contradicts caption. Caption says "scaled scores run 80–120", strips render 100–120; truncated domain exaggerates small gaps and below-100 averages would fall off the edge. Render 80–120, or caption the 100–120 window honestly and define below-100 behaviour.
2. Should-fix — scaled-score England ticks (106/105/105) unsourced. Plausible but hand-entered; verify against DfE 2024/25 tables and add loading official England scaled scores to the pipeline list (absent from §8.1/§8.6).
3. Should-fix — "Writing" listed under "Higher standard" in the picker. Writing TA outcome is "greater depth" (GDS), never "higher standard". Label "Writing — greater depth (teacher assessment)"; tooltip the combined higher-standard composition.
4. Nice — "grammar & punctuation" summary line drops "spelling" (GPS).
5. Nice — science is teacher-assessed (no KS2 test since 2009) and coarse; tooltip it like writing; reconsider its tier-2 slot.
6. Should-fix — mobile Ofsted copy skips the interim regime (Sept 2024–Nov 2025) that desktop explains. One clause fixes it.
7. Should-fix — benchmark provenance still inconsistent (EAL tooltip unsourced; FSM/disadvantaged chips vs tooltips use three vocabularies; header note says all England averages are official). Adopt one house style: official = "England average", computed = "benchmark / typical state school (our dataset)". Also tighten EAL definition to census wording ("first language known or believed to be other than English").
8. Nice — "community primaries" distance note attached to an academy (Elmhurst); say "non-faith primaries" or condition on policy field.
9. Nice — "Improving since 2022" → "since 2022/23".
10. Nice — England chart tooltips show decimals; §7 mandates whole percents.

Residual gaps not covered by spec §8

11. Spec promises IDACI-in-words, Attendance section, and tier-2 gender/absence that the mockups never show — mark post-v1 or demonstrate, so implementation scope is unambiguous.
12. Add official England scaled-score averages to the pipeline task list.
13. Add the writing/greater-depth terminology rule to §8.7.

Verdict

All must-fixes genuinely resolved; the tier model is conceptually sound ("no measure is lost", honest dataset-gap breaks, grouped picker). Remaining issues are contained: one internal contradiction (80–120 vs 100–120), one provenance inconsistency, one terminology error (writing/GDS). With findings 1–3 and 6–7 addressed, the data framing is fit to put in front of parents.