fix(data): official DfE KS4 national headline averages; drop mislabelled computed means

New ees_ks4_national stream ingests the EES 'National characteristics
summary data' series (England, state-funded, all pupils). The old mart's
unweighted school means were 7-15 points off every headline measure and
produced an impossible national Progress 8 (-0.27). The API's computed
fallback is gone too: the footnote calls these figures official, so an
unbuilt mart now yields an empty series, never a stand-in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
This commit is contained in:
Tudor
2026-07-16 19:07:49 +01:00
co-authored by Claude Fable 5
parent 1d855f3c17
commit c9e324635b
14 changed files with 475 additions and 51 deletions
+5 -22
View File
@@ -823,11 +823,11 @@ def _national_averages_payload(df: pd.DataFrame) -> dict:
/api/compare.
Both series are persisted marts computed at import time: official DfE
KS2 figures (fact_ks2_national_averages) and dataset-computed KS4
averages (fact_ks4_national_averages) — the API never aggregates the
performance dataframe per request. If the KS4 mart hasn't been built
yet (deploy lands before the next DAG run), fall back to computing the
latest year only — a single-year scan, never the historical loop.
KS2 figures (fact_ks2_national_averages) and official DfE KS4 figures
(fact_ks4_national_averages) — the API never aggregates the performance
dataframe per request. If the KS4 mart hasn't been built yet, the
secondary series is empty — never a computed stand-in, because the UI
labels these figures as official DfE data.
"""
if df.empty:
return {"primary": {}, "secondary": {}}
@@ -867,23 +867,6 @@ def _national_averages_payload(df: pd.DataFrame) -> dict:
primary_by_year = {r.year: _row_metrics(r, _KS2_NATIONAL_METRICS) for r in ks2_rows}
secondary_by_year = {r.year: _row_metrics(r, _KS4_NATIONAL_METRICS) for r in ks4_rows}
if not any(secondary_by_year.values()):
# KS4 mart missing/empty: compute the latest year only.
df_latest = df[df["year"] == latest_year]
sec = (
df_latest[df_latest["attainment_8_score"].notna()]
if "attainment_8_score" in df_latest.columns
else df_latest.iloc[0:0]
)
vals = {}
for col in _KS4_NATIONAL_METRICS:
if col in sec.columns:
v = sec[col].dropna()
if len(v) > 0:
vals[col] = round(float(v.mean()), 2)
if vals:
secondary_by_year[latest_year] = vals
all_years = sorted(set(primary_by_year) | set(secondary_by_year))
by_year = [
{