feat(census): add demographic columns to EES census tap and staging models
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m39s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s

tap-uk-ees: EESCensusStream now declares 27 data columns (FSM %, EAL %,
ethnicity breakdowns, pupil counts) with clean Singer field names mapped
from the verbose CSV column names (e.g. '% of pupils known to be eligible
for free school meals' → fsm_pct) via a new _column_renames mechanism on
the base stream class.

stg_ees_census: materialised as table, applies safe_numeric to all
percentage/count columns, filters to numeric URNs.

int_pupil_chars_merged + fact_pupil_characteristics: pass all columns
through from staging (previously stubs with only 3 columns).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-27 14:07:48 +00:00
parent 4b02ab3d8a
commit 668e234eb2
4 changed files with 176 additions and 32 deletions

View File

@@ -1,8 +1,34 @@
-- Mart: Pupil characteristics — one row per URN per year
-- TODO: Expand once census data columns are verified and added to staging
select
urn,
year,
phase_type_grouping
phase_type_grouping,
total_pupils,
female_pupils,
male_pupils,
full_time_pupils,
part_time_pupils,
total_boarders,
fsm_eligible_n,
fsm_pct,
eal_pct,
young_carer_pct,
ethnicity_white_british_pct,
ethnicity_white_other_pct,
ethnicity_gypsy_roma_pct,
ethnicity_mixed_wbc_pct,
ethnicity_mixed_wba_pct,
ethnicity_mixed_wa_pct,
ethnicity_mixed_other_pct,
ethnicity_asian_indian_pct,
ethnicity_asian_pakistani_pct,
ethnicity_asian_bangladeshi_pct,
ethnicity_asian_other_pct,
ethnicity_black_caribbean_pct,
ethnicity_black_african_pct,
ethnicity_black_other_pct,
ethnicity_chinese_pct,
ethnicity_other_pct,
ethnicity_unclassified_pct
from {{ ref('int_pupil_chars_merged') }}