tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(airflow): delete and reimport DAGs on init to clear stale task refs
Some checks failed
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Details
Build and Push Docker Images / Build Integrator (push) Has been cancelled
Details
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Details
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Has been cancelled
Details

Browse Source 
When tasks are removed from a DAG, old serialized metadata in the DB
causes 'Task not found' errors. Delete all DAGs before reserializing
on each deploy to ensure a clean state.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor
2026-03-26 21:28:03 +00:00
parent 54df58746e
commit 03cd1de6af
2 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docker-compose.portainer.yml
View File

@@ -257,7 +257,7 @@ services:
airflow-init: airflow-init:
image: privaterepo.sitaru.org/tudor/school_compare-pipeline:latest image: privaterepo.sitaru.org/tudor/school_compare-pipeline:latest
container_name: schoolcompare_airflow_init container_name: schoolcompare_airflow_init
command: bash -c "airflow db migrate && airflow dags reserialize" command: bash -c "airflow db migrate && airflow dags delete school_data_daily -y 2>/dev/null; airflow dags delete school_data_monthly_ofsted -y 2>/dev/null; airflow dags delete school_data_annual_ees -y 2>/dev/null; airflow dags reserialize"
environment: environment:
AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${DB_USERNAME}:${DB_PASSWORD}@sc_database:5432/${DB_DATABASE_NAME} AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${DB_USERNAME}:${DB_PASSWORD}@sc_database:5432/${DB_DATABASE_NAME}

2
docker-compose.yml
View File

@@ -152,7 +152,7 @@ services:
airflow-init: airflow-init:
image: privaterepo.sitaru.org/tudor/school_compare-pipeline:latest image: privaterepo.sitaru.org/tudor/school_compare-pipeline:latest
container_name: schoolcompare_airflow_init container_name: schoolcompare_airflow_init
command: bash -c "airflow db migrate && airflow dags reserialize" command: bash -c "airflow db migrate && airflow dags delete school_data_daily -y 2>/dev/null; airflow dags delete school_data_monthly_ofsted -y 2>/dev/null; airflow dags delete school_data_annual_ees -y 2>/dev/null; airflow dags reserialize"
environment: *airflow-env environment: *airflow-env
depends_on: depends_on:
db: db: