feat(pipeline): add DAGs for Parent View and IDACI deprivation
Some checks failed
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Some checks failed
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
- school_data_monthly_parent_view: runs 1st of month, extracts Ofsted Parent View and builds fact_parent_view - school_data_annual_idaci: manual trigger, extracts IDACI deprivation index and builds fact_deprivation Both tables were missing, causing safe_query to fail and leave the PostgreSQL transaction in an aborted state, silently killing all subsequent supplementary data queries including fact_admissions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -149,3 +149,53 @@ with DAG(
|
|||||||
)
|
)
|
||||||
|
|
||||||
extract_ees_group >> dbt_build_ees >> sync_typesense_ees
|
extract_ees_group >> dbt_build_ees >> sync_typesense_ees
|
||||||
|
|
||||||
|
|
||||||
|
# ── Monthly DAG (Parent View) ──────────────────────────────────────────
|
||||||
|
|
||||||
|
with DAG(
|
||||||
|
dag_id="school_data_monthly_parent_view",
|
||||||
|
default_args=default_args,
|
||||||
|
description="Monthly Ofsted Parent View extraction and transform",
|
||||||
|
schedule="0 3 1 * *",
|
||||||
|
start_date=datetime(2025, 1, 1),
|
||||||
|
catchup=False,
|
||||||
|
tags=["school-compare", "monthly"],
|
||||||
|
) as monthly_parent_view_dag:
|
||||||
|
|
||||||
|
extract_parent_view = BashOperator(
|
||||||
|
task_id="extract_parent_view",
|
||||||
|
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-parent-view target-postgres",
|
||||||
|
)
|
||||||
|
|
||||||
|
dbt_build_parent_view = BashOperator(
|
||||||
|
task_id="dbt_build",
|
||||||
|
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_parent_view+ fact_parent_view+",
|
||||||
|
)
|
||||||
|
|
||||||
|
extract_parent_view >> dbt_build_parent_view
|
||||||
|
|
||||||
|
|
||||||
|
# ── Annual DAG (IDACI Deprivation) ────────────────────────────────────
|
||||||
|
|
||||||
|
with DAG(
|
||||||
|
dag_id="school_data_annual_idaci",
|
||||||
|
default_args=default_args,
|
||||||
|
description="Annual IDACI deprivation index extraction and transform",
|
||||||
|
schedule=None, # Triggered manually when new IDACI release is published
|
||||||
|
start_date=datetime(2025, 1, 1),
|
||||||
|
catchup=False,
|
||||||
|
tags=["school-compare", "annual"],
|
||||||
|
) as annual_idaci_dag:
|
||||||
|
|
||||||
|
extract_idaci = BashOperator(
|
||||||
|
task_id="extract_idaci",
|
||||||
|
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-idaci target-postgres",
|
||||||
|
)
|
||||||
|
|
||||||
|
dbt_build_idaci = BashOperator(
|
||||||
|
task_id="dbt_build",
|
||||||
|
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_idaci+ fact_deprivation+",
|
||||||
|
)
|
||||||
|
|
||||||
|
extract_idaci >> dbt_build_idaci
|
||||||
|
|||||||
Reference in New Issue
Block a user