feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Replaces the hand-rolled integrator with a production-grade ELT pipeline using Meltano (Singer taps), dbt Core (medallion architecture), and Apache Airflow (orchestration). Adds Typesense for search and PostGIS for geospatial queries. - 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI) - dbt project: 12 staging, 5 intermediate, 12 mart models - 3 Airflow DAGs (daily/monthly/annual schedules) - Typesense sync + batch geocoding scripts - docker-compose: add Airflow, Typesense; upgrade to PostGIS - Portainer stack definition matching live deployment topology Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
7
pipeline/transform/tests/assert_no_orphaned_facts.sql
Normal file
7
pipeline/transform/tests/assert_no_orphaned_facts.sql
Normal file
@@ -0,0 +1,7 @@
|
||||
-- Custom test: All fact table URNs should exist in dim_school
|
||||
|
||||
select f.urn
|
||||
from {{ ref('fact_ks2_performance') }} f
|
||||
left join {{ ref('dim_school') }} d on f.urn = d.urn
|
||||
where d.urn is null
|
||||
limit 10
|
||||
13
pipeline/transform/tests/assert_urns_in_uk_bounds.sql
Normal file
13
pipeline/transform/tests/assert_urns_in_uk_bounds.sql
Normal file
@@ -0,0 +1,13 @@
|
||||
-- Custom test: All geocoded schools should have coordinates within the UK
|
||||
|
||||
select
|
||||
urn,
|
||||
easting,
|
||||
northing
|
||||
from {{ ref('dim_location') }}
|
||||
where easting is not null
|
||||
and northing is not null
|
||||
and (
|
||||
easting < 0 or easting > 700000
|
||||
or northing < 0 or northing > 1300000
|
||||
)
|
||||
Reference in New Issue
Block a user