fix(data): official DfE KS4 national headline averages; drop mislabelled computed means
New ees_ks4_national stream ingests the EES 'National characteristics summary data' series (England, state-funded, all pupils). The old mart's unweighted school means were 7-15 points off every headline measure and produced an impossible national Progress 8 (-0.27). The API's computed fallback is gone too: the footnote calls these figures official, so an unbuilt mart now yields an empty series, never a stand-in. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
This commit is contained in:
+5
-22
@@ -823,11 +823,11 @@ def _national_averages_payload(df: pd.DataFrame) -> dict:
|
||||
/api/compare.
|
||||
|
||||
Both series are persisted marts computed at import time: official DfE
|
||||
KS2 figures (fact_ks2_national_averages) and dataset-computed KS4
|
||||
averages (fact_ks4_national_averages) — the API never aggregates the
|
||||
performance dataframe per request. If the KS4 mart hasn't been built
|
||||
yet (deploy lands before the next DAG run), fall back to computing the
|
||||
latest year only — a single-year scan, never the historical loop.
|
||||
KS2 figures (fact_ks2_national_averages) and official DfE KS4 figures
|
||||
(fact_ks4_national_averages) — the API never aggregates the performance
|
||||
dataframe per request. If the KS4 mart hasn't been built yet, the
|
||||
secondary series is empty — never a computed stand-in, because the UI
|
||||
labels these figures as official DfE data.
|
||||
"""
|
||||
if df.empty:
|
||||
return {"primary": {}, "secondary": {}}
|
||||
@@ -867,23 +867,6 @@ def _national_averages_payload(df: pd.DataFrame) -> dict:
|
||||
primary_by_year = {r.year: _row_metrics(r, _KS2_NATIONAL_METRICS) for r in ks2_rows}
|
||||
secondary_by_year = {r.year: _row_metrics(r, _KS4_NATIONAL_METRICS) for r in ks4_rows}
|
||||
|
||||
if not any(secondary_by_year.values()):
|
||||
# KS4 mart missing/empty: compute the latest year only.
|
||||
df_latest = df[df["year"] == latest_year]
|
||||
sec = (
|
||||
df_latest[df_latest["attainment_8_score"].notna()]
|
||||
if "attainment_8_score" in df_latest.columns
|
||||
else df_latest.iloc[0:0]
|
||||
)
|
||||
vals = {}
|
||||
for col in _KS4_NATIONAL_METRICS:
|
||||
if col in sec.columns:
|
||||
v = sec[col].dropna()
|
||||
if len(v) > 0:
|
||||
vals[col] = round(float(v.mean()), 2)
|
||||
if vals:
|
||||
secondary_by_year[latest_year] = vals
|
||||
|
||||
all_years = sorted(set(primary_by_year) | set(secondary_by_year))
|
||||
by_year = [
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user