fix(pipeline): keep the destinations marts out of the scheduled builds
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m18s
PR Checks / Backend Smoke (pull_request) Successful in 11s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m35s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 1m21s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 19s
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m18s
PR Checks / Backend Smoke (pull_request) Successful in 11s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m35s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 1m21s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 19s
fact_ks4_destinations and fact_ks5_destinations join dim_school, so the daily build's stg_gias_establishments+ and the monthly Ofsted build's dim_school+ both selected them. They also read stg_ees_ks4/ks5_destinations, which only the manually triggered EES DAG builds. Where that DAG hasn't run since the destinations models landed, dbt_build fails with "relation staging.stg_ees_ks4_destinations does not exist", and sync_typesense and invalidate_cache never run. Production's register data has been stuck at about 25 Aug 2026. Both builds now exclude the descendants of the two EES staging models, as the daily build already does for the KS2/KS4 lineage models. The EES DAG still rebuilds the marts when their data changes. test_dag_selectors reads the model graph from the SQL (CI has no dbt) and checks that every scheduled build only reads models it or the daily build builds. It failed for the daily and monthly Ofsted builds before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
423b27140c
commit
65a2619e1d
2 files changed
+103
-2
No files matched your search
@@ -106,9 +106,12 @@ print(f'Validation passed: {{count}} GIAS rows')
|
||||
""",
|
||||
)
|
||||
|
||||
# Marts fed by annual EES staging models are rebuilt by the EES DAG, even
|
||||
# when they join dim_school. Selecting them here fails in any database
|
||||
# where that DAG hasn't run (pipeline/tests/test_dag_selectors.py).
|
||||
dbt_build = BashOperator(
|
||||
task_id="dbt_build",
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_gias_establishments+ stg_gias_links+ gias_code_names+ --exclude int_ks2_with_lineage+ int_ks4_with_lineage+",
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_gias_establishments+ stg_gias_links+ gias_code_names+ --exclude int_ks2_with_lineage+ int_ks4_with_lineage+ stg_ees_ks4_destinations+ stg_ees_ks5_destinations+",
|
||||
)
|
||||
|
||||
sync_typesense = BashOperator(
|
||||
@@ -143,7 +146,7 @@ with DAG(
|
||||
|
||||
dbt_build_ofsted = BashOperator(
|
||||
task_id="dbt_build",
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ofsted_inspections+ int_ofsted_latest+ fact_ofsted_inspection+ dim_school+",
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ofsted_inspections+ int_ofsted_latest+ fact_ofsted_inspection+ dim_school+ --exclude stg_ees_ks4_destinations+ stg_ees_ks5_destinations+",
|
||||
)
|
||||
|
||||
sync_typesense_ofsted = BashOperator(
|
||||
|
||||
Reference in new issue
Block a user