feat: ingest official DfE KS2 national averages from EES data catalogue
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 19s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 53s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m24s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 19s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 53s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m24s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Replaces computed means from our school dataset with the published DfE national headline figures for the KS2 chart reference line. - tap-uk-ees: new EESKs2NationalStream fetches the stable EES data-catalogue CSV (one row per year, England national total, AllSchools filter) - dbt staging: stg_ees_ks2_national normalises columns, casts to float, filters to years >= 201617 - dbt mart: fact_ks2_national_averages — one row per year, official figures - backend/models: Ks2NationalAverage SQLAlchemy model - backend/app: /api/national-averages queries the mart for KS2 by_year; secondary by_year stays computed (no DfE KS4 national dataset yet) - DAG: extract_ks2_national task added to school_data_annual_ees, runs in parallel with the main EES extract Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -137,10 +137,15 @@ with DAG(
|
||||
task_id="extract_ees",
|
||||
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-ees target-postgres",
|
||||
)
|
||||
# KS2 national headlines run in parallel — small single-CSV download
|
||||
extract_ks2_national = BashOperator(
|
||||
task_id="extract_ks2_national",
|
||||
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-ees target-postgres --select ees_ks2_national",
|
||||
)
|
||||
|
||||
dbt_build_ees = BashOperator(
|
||||
task_id="dbt_build",
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ees_ks2+ stg_legacy_ks2+ stg_ees_ks4+ stg_ees_census+ stg_ees_admissions+",
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ees_ks2+ stg_legacy_ks2+ stg_ees_ks4+ stg_ees_census+ stg_ees_admissions+ stg_ees_ks2_national+",
|
||||
)
|
||||
|
||||
sync_typesense_ees = BashOperator(
|
||||
|
||||
Reference in New Issue
Block a user