feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.
- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00
|
|
|
-- Mart: Canonical school dimension — one row per active URN
|
|
|
|
|
|
|
|
|
|
with schools as (
|
|
|
|
|
select * from {{ ref('stg_gias_establishments') }}
|
|
|
|
|
)
|
|
|
|
|
|
2026-03-26 20:43:24 +00:00
|
|
|
{% set ofsted_relation = adapter.get_relation(
|
|
|
|
|
database=target.database,
|
|
|
|
|
schema=target.schema,
|
|
|
|
|
identifier='int_ofsted_latest'
|
|
|
|
|
) %}
|
|
|
|
|
|
feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.
- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00
|
|
|
select
|
|
|
|
|
s.urn,
|
|
|
|
|
s.local_authority_code * 1000 + s.establishment_number as laestab,
|
|
|
|
|
s.school_name,
|
|
|
|
|
s.phase,
|
|
|
|
|
s.school_type,
|
|
|
|
|
s.academy_trust_name,
|
|
|
|
|
s.academy_trust_uid,
|
|
|
|
|
s.religious_character,
|
|
|
|
|
s.gender,
|
|
|
|
|
s.statutory_low_age || '-' || s.statutory_high_age as age_range,
|
|
|
|
|
s.capacity,
|
|
|
|
|
s.total_pupils,
|
|
|
|
|
concat_ws(' ', s.head_title, s.head_first_name, s.head_last_name) as headteacher_name,
|
|
|
|
|
s.website,
|
|
|
|
|
s.telephone,
|
|
|
|
|
s.open_date,
|
|
|
|
|
s.close_date,
|
|
|
|
|
s.status,
|
|
|
|
|
s.nursery_provision,
|
|
|
|
|
s.admissions_policy,
|
|
|
|
|
|
2026-03-26 20:43:24 +00:00
|
|
|
-- Latest Ofsted (populated after monthly Ofsted pipeline runs)
|
|
|
|
|
{% if ofsted_relation is not none %}
|
feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.
- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00
|
|
|
o.overall_effectiveness as ofsted_grade,
|
|
|
|
|
o.inspection_date as ofsted_date,
|
|
|
|
|
o.framework as ofsted_framework
|
2026-03-26 20:43:24 +00:00
|
|
|
{% else %}
|
|
|
|
|
null::text as ofsted_grade,
|
|
|
|
|
null::date as ofsted_date,
|
|
|
|
|
null::text as ofsted_framework
|
|
|
|
|
{% endif %}
|
feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.
- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00
|
|
|
|
|
|
|
|
from schools s
|
2026-03-26 20:43:24 +00:00
|
|
|
{% if ofsted_relation is not none %}
|
|
|
|
|
left join {{ ref('int_ofsted_latest') }} o on s.urn = o.urn
|
|
|
|
|
{% endif %}
|
feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.
- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00
|
|
|
where s.status = 'Open'
|