tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(pipeline): use GIAS easting/northing for all geocoding, drop postcode step
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details

Browse Source 
GIAS grid references are the actual school location — far more accurate
than postcode centroids. Remove geocode_postcodes.py from the daily DAG
and the postcode-not-null filter from dim_location.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor
2026-03-26 21:18:59 +00:00
parent d3e655abdb
commit 54df58746e
2 changed files with 1 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
pipeline/dags/school_data_pipeline.py
View File

@@ -82,17 +82,12 @@ print(f'Validation passed: {{count}} GIAS rows')
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_gias_establishments+ stg_gias_links+ --exclude int_ks2_with_lineage+ int_ks4_with_lineage+", bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_gias_establishments+ stg_gias_links+ --exclude int_ks2_with_lineage+ int_ks4_with_lineage+",
) )
geocode_new = BashOperator(
task_id="geocode_new",
bash_command=f"cd {PIPELINE_DIR} && python scripts/geocode_postcodes.py",
)
sync_typesense = BashOperator( sync_typesense = BashOperator(
task_id="sync_typesense", task_id="sync_typesense",
bash_command=f"cd {PIPELINE_DIR} && python scripts/sync_typesense.py", bash_command=f"cd {PIPELINE_DIR} && python scripts/sync_typesense.py",
) )
extract_group >> validate_raw >> dbt_build >> geocode_new >> sync_typesense extract_group >> validate_raw >> dbt_build >> sync_typesense
# ── Monthly DAG (Ofsted) ─────────────────────────────────────────────── # ── Monthly DAG (Ofsted) ───────────────────────────────────────────────

1
pipeline/transform/models/marts/dim_location.sql
View File

@@ -22,4 +22,3 @@ select
end as geom end as geom
from {{ ref('stg_gias_establishments') }} s from {{ ref('stg_gias_establishments') }} s
where s.status = 'Open' where s.status = 'Open'
and s.postcode is not null