fix(pipeline): load 2023/24 KS4/KS2 data and DfE LA averages (C2, H2, part 1 of 2) #185

Merged
tudor merged 11 commits from fix/ks4-2023-24-pipeline into main 2026-10-06 12:42:19 +00:00
Showing only changes of commit 951c666f8a - Show all commits
@@ -0,0 +1,20 @@
{{ config(severity='warn') }}
-- DfE's LA averages cover the latest year with school results. They come from
-- a fixed EES data set (ks4_summary.KS4_SUMMARY_CSV_URL); when DfE publishes a
-- new year under a new data set, the school results move on and this mart does
-- not, so /api/la-averages serves an empty map and every "vs LA avg" goes.
-- A warning, not an error: it must not hold back a new year's school results.
with schools as (
select max(year) as year from {{ ref('stg_ees_ks4') }} where attainment_8_score is not null
),
las as (
select max(year) as year from {{ ref('fact_ks4_la_averages') }}
)
select s.year as latest_school_year, l.year as latest_la_year
from schools s
cross join las l
where l.year is null or l.year < s.year