fix(data): official DfE KS4 national headline averages; drop mislabelled computed means

New ees_ks4_national stream ingests the EES 'National characteristics
summary data' series (England, state-funded, all pupils). The old mart's
unweighted school means were 7-15 points off every headline measure and
produced an impossible national Progress 8 (-0.27). The API's computed
fallback is gone too: the footnote calls these figures official, so an
unbuilt mart now yields an empty series, never a stand-in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
This commit is contained in:
Tudor
2026-07-16 19:07:49 +01:00
co-authored by Claude Fable 5
parent 1d855f3c17
commit c9e324635b
14 changed files with 475 additions and 51 deletions
+10 -9
View File
@@ -1,7 +1,7 @@
"""_national_averages_payload reads persisted marts (computed at import
time) — it must never loop the dataframe per year. The only dataframe work
allowed is the single-latest-year KS4 fallback for the window between a
deploy and the next DAG run."""
time) — it must never aggregate the dataframe. Both marts hold OFFICIAL
DfE figures, so a missing KS4 mart yields an empty secondary series —
never a computed stand-in the UI would mislabel as official."""
import numpy as np
import pandas as pd
@@ -84,10 +84,11 @@ def test_ks4_averages_come_from_the_mart_not_the_dataframe(payload):
assert body["by_year"][-1]["secondary"]["progress_8_score"] == -0.02
def test_missing_ks4_mart_falls_back_to_latest_year_only(payload):
def test_ks4_secondary_empty_when_mart_missing(payload):
# No computed stand-in: the UI labels national figures as official DfE
# data, so an empty mart must yield an empty secondary series.
body = payload(_Ks4MissingSession)
# Fallback computes the latest year from the df: mean(50, 30) = 40.0
assert body["secondary"]["attainment_8_score"] == 40.0
# ...and only the latest year — no historical KS4 loop
ks4_years = [e["year"] for e in body["by_year"] if e["secondary"]]
assert ks4_years == [LATEST]
assert body["secondary"] == {}
assert all(not e["secondary"] for e in body["by_year"])
# The KS2 series is unaffected.
assert body["primary"]["rwm_expected_pct"] == 62.1