fix(pipeline): expand GIAS schema, handle empty strings, scope DAG selectors
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m8s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 34s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m39s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m8s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 34s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m39s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- Declare all 34 columns needed by dbt in GIAS tap schema (target-postgres only persists columns present in the Singer schema message) - Use nullif() for empty-string-to-integer/date casts in staging models - Scope daily DAG dbt build to GIAS models only (stg_gias_establishments+ stg_gias_links+) to avoid errors on unloaded sources - Scope annual EES DAG similarly; remove redundant dbt test steps - Make dim_school gracefully handle missing int_ofsted_latest table Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -2,12 +2,14 @@
|
||||
|
||||
with schools as (
|
||||
select * from {{ ref('stg_gias_establishments') }}
|
||||
),
|
||||
|
||||
latest_ofsted as (
|
||||
select * from {{ ref('int_ofsted_latest') }}
|
||||
)
|
||||
|
||||
{% set ofsted_relation = adapter.get_relation(
|
||||
database=target.database,
|
||||
schema=target.schema,
|
||||
identifier='int_ofsted_latest'
|
||||
) %}
|
||||
|
||||
select
|
||||
s.urn,
|
||||
s.local_authority_code * 1000 + s.establishment_number as laestab,
|
||||
@@ -30,11 +32,19 @@ select
|
||||
s.nursery_provision,
|
||||
s.admissions_policy,
|
||||
|
||||
-- Latest Ofsted
|
||||
-- Latest Ofsted (populated after monthly Ofsted pipeline runs)
|
||||
{% if ofsted_relation is not none %}
|
||||
o.overall_effectiveness as ofsted_grade,
|
||||
o.inspection_date as ofsted_date,
|
||||
o.framework as ofsted_framework
|
||||
{% else %}
|
||||
null::text as ofsted_grade,
|
||||
null::date as ofsted_date,
|
||||
null::text as ofsted_framework
|
||||
{% endif %}
|
||||
|
||||
from schools s
|
||||
left join latest_ofsted o on s.urn = o.urn
|
||||
{% if ofsted_relation is not none %}
|
||||
left join {{ ref('int_ofsted_latest') }} o on s.urn = o.urn
|
||||
{% endif %}
|
||||
where s.status = 'Open'
|
||||
|
||||
Reference in New Issue
Block a user