tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(pipeline): make total_pupils non-optional for Typesense, add lat/lng to dim_location
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m3s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m29s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details

Browse Source 
- Remove optional flag from total_pupils (Typesense requires default
  sorting field to be non-optional)
- Add latitude/longitude columns to dim_location computed from PostGIS
  geom, for direct use by backend and Typesense sync

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor
2026-03-26 22:45:02 +00:00
parent 5e44d88d23
commit 719f06e480
2 changed files with 14 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
pipeline/scripts/sync_typesense.py
View File

@@ -33,7 +33,7 @@ COLLECTION_SCHEMA = {
{"name": "headteacher_name", "type": "string", "optional": True}, {"name": "headteacher_name", "type": "string", "optional": True},
{"name": "rwm_expected_pct", "type": "float", "optional": True}, {"name": "rwm_expected_pct", "type": "float", "optional": True},
{"name": "progress_8_score", "type": "float", "optional": True}, {"name": "progress_8_score", "type": "float", "optional": True},
{"name": "total_pupils", "type": "int32", "optional": True}, {"name": "total_pupils", "type": "int32"},
], ],
"default_sorting_field": "total_pupils", "default_sorting_field": "total_pupils",
} }
@@ -52,8 +52,8 @@ QUERY_BASE = """
l.postcode, l.postcode,
s.headteacher_name, s.headteacher_name,
s.total_pupils, s.total_pupils,
ST_Y(l.geom) as lat, l.latitude as lat,
ST_X(l.geom) as lng l.longitude as lng
FROM marts.dim_school s FROM marts.dim_school s
LEFT JOIN marts.dim_location l ON s.urn = l.urn LEFT JOIN marts.dim_location l ON s.urn = l.urn
""" """

12
pipeline/transform/models/marts/dim_location.sql
View File

@@ -19,6 +19,16 @@ select
when s.easting is not null and s.northing is not null when s.easting is not null and s.northing is not null
then ST_Transform(ST_SetSRID(ST_MakePoint(s.easting::double precision, s.northing::double precision), 27700), 4326) then ST_Transform(ST_SetSRID(ST_MakePoint(s.easting::double precision, s.northing::double precision), 27700), 4326)
else null else null
end as geom end as geom,
case
when s.easting is not null and s.northing is not null
then ST_Y(ST_Transform(ST_SetSRID(ST_MakePoint(s.easting::double precision, s.northing::double precision), 27700), 4326))
else null
end as latitude,
case
when s.easting is not null and s.northing is not null
then ST_X(ST_Transform(ST_SetSRID(ST_MakePoint(s.easting::double precision, s.northing::double precision), 27700), 4326))
else null
end as longitude
from {{ ref('stg_gias_establishments') }} s from {{ ref('stg_gias_establishments') }} s
where s.status = 'Open' where s.status = 'Open'