feat(pipeline): use GIAS easting/northing for all geocoding, drop postcode step
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
GIAS grid references are the actual school location — far more accurate than postcode centroids. Remove geocode_postcodes.py from the daily DAG and the postcode-not-null filter from dim_location. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -82,17 +82,12 @@ print(f'Validation passed: {{count}} GIAS rows')
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_gias_establishments+ stg_gias_links+ --exclude int_ks2_with_lineage+ int_ks4_with_lineage+",
|
||||
)
|
||||
|
||||
geocode_new = BashOperator(
|
||||
task_id="geocode_new",
|
||||
bash_command=f"cd {PIPELINE_DIR} && python scripts/geocode_postcodes.py",
|
||||
)
|
||||
|
||||
sync_typesense = BashOperator(
|
||||
task_id="sync_typesense",
|
||||
bash_command=f"cd {PIPELINE_DIR} && python scripts/sync_typesense.py",
|
||||
)
|
||||
|
||||
extract_group >> validate_raw >> dbt_build >> geocode_new >> sync_typesense
|
||||
extract_group >> validate_raw >> dbt_build >> sync_typesense
|
||||
|
||||
|
||||
# ── Monthly DAG (Ofsted) ───────────────────────────────────────────────
|
||||
|
||||
@@ -22,4 +22,3 @@ select
|
||||
end as geom
|
||||
from {{ ref('stg_gias_establishments') }} s
|
||||
where s.status = 'Open'
|
||||
and s.postcode is not null
|
||||
|
||||
Reference in New Issue
Block a user