tudor/school_compare
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(pipeline): include 2015/16 national averages

Browse Source 
Widen the year filter in stg_ees_ks2_national.sql from >= 201617 to
>= 201516 so the England national-averages line no longer starts a
year late; the catalogue CSV has a real, comparable 201516 row (2015/16
was the first year of the current expected-standard tests, so it's the
correct floor).

GPS/science/scaled-score national columns confirmed present at source
with correct mapping; prod NULLs are stale raw data, backfilled by the
next extract run. No _KS2_NATIONAL_COL_MAP change accompanies this fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
This commit is contained in:
Tudor
2026-07-12 22:02:28 +01:00
co-authored by Claude Fable 5
parent 5f1b6adb44
commit ccd8e73fe8
1 changed files with 7 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pipeline/transform/models/staging/stg_ees_ks2_national.sql
+7 -1
View File
@@ -31,4 +31,10 @@ select
from {{ source('raw', 'ees_ks2_national') }} from {{ source('raw', 'ees_ks2_national') }}
where time_period ~ '^[0-9]+$' where time_period ~ '^[0-9]+$'
and cast(trim(time_period) as integer) >= 201617 -- 2015/16 was the first year of the current expected-standard tests, so it's
-- the correct floor (not 2016/17 -- that excluded a real, comparable national
-- row). GPS/science/scaled-score columns are already mapped correctly end to
-- end (tap.py's _KS2_NATIONAL_COL_MAP + this model select them fine); the
-- prod NULLs for those fields are stale raw.ees_ks2_national data from before
-- the map covered them, not a mapping bug -- no map change accompanies this fix.
and cast(trim(time_period) as integer) >= 201516