feat(pipeline): add DAGs for Parent View and IDACI deprivation
Some checks failed
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled

- school_data_monthly_parent_view: runs 1st of month, extracts Ofsted
  Parent View and builds fact_parent_view
- school_data_annual_idaci: manual trigger, extracts IDACI deprivation
  index and builds fact_deprivation

Both tables were missing, causing safe_query to fail and leave the
PostgreSQL transaction in an aborted state, silently killing all
subsequent supplementary data queries including fact_admissions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-27 22:08:12 +00:00
parent 55749bdfaf
commit 1629a8f994

View File

@@ -149,3 +149,53 @@ with DAG(
) )
extract_ees_group >> dbt_build_ees >> sync_typesense_ees extract_ees_group >> dbt_build_ees >> sync_typesense_ees
# ── Monthly DAG (Parent View) ──────────────────────────────────────────
with DAG(
dag_id="school_data_monthly_parent_view",
default_args=default_args,
description="Monthly Ofsted Parent View extraction and transform",
schedule="0 3 1 * *",
start_date=datetime(2025, 1, 1),
catchup=False,
tags=["school-compare", "monthly"],
) as monthly_parent_view_dag:
extract_parent_view = BashOperator(
task_id="extract_parent_view",
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-parent-view target-postgres",
)
dbt_build_parent_view = BashOperator(
task_id="dbt_build",
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_parent_view+ fact_parent_view+",
)
extract_parent_view >> dbt_build_parent_view
# ── Annual DAG (IDACI Deprivation) ────────────────────────────────────
with DAG(
dag_id="school_data_annual_idaci",
default_args=default_args,
description="Annual IDACI deprivation index extraction and transform",
schedule=None, # Triggered manually when new IDACI release is published
start_date=datetime(2025, 1, 1),
catchup=False,
tags=["school-compare", "annual"],
) as annual_idaci_dag:
extract_idaci = BashOperator(
task_id="extract_idaci",
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-idaci target-postgres",
)
dbt_build_idaci = BashOperator(
task_id="dbt_build",
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_idaci+ fact_deprivation+",
)
extract_idaci >> dbt_build_idaci