tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(dag): remove invalid --select flag from meltano run
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 13s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 54s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m12s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details

Browse Source 
meltano run does not support --select; the full tap-uk-ees run already
includes EESKs2NationalStream so no separate task is needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor Sitaru
2026-04-09 14:48:31 +01:00
parent dc66e22d4d
commit 06bf53ac26
1 changed files with 1 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pipeline/dags/school_data_pipeline.py
+1 -5
View File
@@ -133,15 +133,11 @@ with DAG(
) as annual_ees_dag: ) as annual_ees_dag:
with TaskGroup("extract_ees") as extract_ees_group: with TaskGroup("extract_ees") as extract_ees_group:
# Runs all tap-uk-ees streams, including the new ees_ks2_national stream
extract_ees = BashOperator( extract_ees = BashOperator(
task_id="extract_ees", task_id="extract_ees",
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-ees target-postgres", bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-ees target-postgres",
) )
# KS2 national headlines run in parallel — small single-CSV download
extract_ks2_national = BashOperator(
task_id="extract_ks2_national",
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-ees target-postgres --select ees_ks2_national",
)
dbt_build_ees = BashOperator( dbt_build_ees = BashOperator(
task_id="dbt_build", task_id="dbt_build",