feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.
- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00
|
|
|
-- Mart: School location dimension — one row per URN, PostGIS-enabled
|
2026-03-26 21:17:08 +00:00
|
|
|
-- Geometry derived from GIAS easting/northing (British National Grid → WGS84).
|
|
|
|
|
-- For schools missing easting/northing, the geocode script backfills via Postcodes.io.
|
feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.
- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00
|
|
|
|
|
|
|
|
select
|
|
|
|
|
s.urn,
|
|
|
|
|
s.address_line1,
|
|
|
|
|
s.address_line2,
|
|
|
|
|
s.town,
|
|
|
|
|
s.county,
|
|
|
|
|
s.postcode,
|
|
|
|
|
s.local_authority_code,
|
|
|
|
|
s.local_authority_name,
|
|
|
|
|
s.parliamentary_constituency,
|
|
|
|
|
s.urban_rural,
|
|
|
|
|
s.easting,
|
2026-03-26 21:17:08 +00:00
|
|
|
s.northing,
|
|
|
|
|
case
|
|
|
|
|
when s.easting is not null and s.northing is not null
|
2026-03-26 21:45:53 +00:00
|
|
|
then ST_Transform(ST_SetSRID(ST_MakePoint(s.easting::double precision, s.northing::double precision), 27700), 4326)
|
2026-03-26 21:17:08 +00:00
|
|
|
else null
|
2026-03-26 22:45:02 +00:00
|
|
|
end as geom,
|
|
|
|
|
case
|
|
|
|
|
when s.easting is not null and s.northing is not null
|
|
|
|
|
then ST_Y(ST_Transform(ST_SetSRID(ST_MakePoint(s.easting::double precision, s.northing::double precision), 27700), 4326))
|
|
|
|
|
else null
|
|
|
|
|
end as latitude,
|
|
|
|
|
case
|
|
|
|
|
when s.easting is not null and s.northing is not null
|
|
|
|
|
then ST_X(ST_Transform(ST_SetSRID(ST_MakePoint(s.easting::double precision, s.northing::double precision), 27700), 4326))
|
|
|
|
|
else null
|
|
|
|
|
end as longitude
|
feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.
- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00
|
|
|
from {{ ref('stg_gias_establishments') }} s
|
|
|
|
|
where s.status = 'Open'
|