tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(stg_ofsted_inspections): handle NULL strings from Ofsted CSV
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m26s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details

Browse Source 
Use nullif+trim for date cast and safe_numeric for integer grades to
handle literal 'NULL' strings present in the new Report Card format CSV.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor
2026-03-27 17:23:46 +00:00
parent 26aa3c2d70
commit 05cb22f1a5
1 changed files with 10 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
pipeline/transform/models/staging/stg_ofsted_inspections.sql
View File

@@ -8,18 +8,18 @@ with source as (
renamed as ( renamed as (
select select
cast(urn as integer) as urn, cast(urn as integer) as urn,
cast(inspection_date as date) as inspection_date, nullif(trim(inspection_date), 'NULL')::date as inspection_date,
inspection_type, inspection_type,
event_type_grouping as framework, event_type_grouping as framework,
-- OEIF grades (1-4 scale) -- OEIF grades (1-4 scale)
cast(overall_effectiveness as integer) as overall_effectiveness, {{ safe_numeric('overall_effectiveness') }}::integer as overall_effectiveness,
cast(quality_of_education as integer) as quality_of_education, {{ safe_numeric('quality_of_education') }}::integer as quality_of_education,
cast(behaviour_and_attitudes as integer) as behaviour_attitudes, {{ safe_numeric('behaviour_and_attitudes') }}::integer as behaviour_attitudes,
cast(personal_development as integer) as personal_development, {{ safe_numeric('personal_development') }}::integer as personal_development,
cast(effectiveness_of_leadership_and_management as integer) as leadership_management, {{ safe_numeric('effectiveness_of_leadership_and_management') }}::integer as leadership_management,
cast(early_years_provision as integer) as early_years_provision, {{ safe_numeric('early_years_provision') }}::integer as early_years_provision,
cast(sixth_form_provision as integer) as sixth_form_provision, {{ safe_numeric('sixth_form_provision') }}::integer as sixth_form_provision,
-- Report Card fields (post-Nov 2025 framework) -- Report Card fields (post-Nov 2025 framework)
-- TODO: add rc_* columns to tap-uk-ofsted schema once CSV column names are confirmed -- TODO: add rc_* columns to tap-uk-ofsted schema once CSV column names are confirmed