Commit Graph

5 Commits

Author SHA1 Message Date
Tudor Sitaru ca5f6a962c fix(dim_school): expand phase inference with name-based fallback
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 12s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 45s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m10s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The case-insensitive "Not Applicable" fix caught schools where GIAS publishes
statutory ages, but some independent schools leave those blank too — they fall
through every branch and end up with null phase and no pill in the UI.

Add a third tier that infers phase from the school name
(Primary/Infant/Junior/Prep vs Secondary/High/Grammar/Senior/Upper) and also
normalise "Not Applicable" handling with trim() + "unknown"/"" exclusion, so
the final else branch can safely return null instead of the catch-all string.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 21:15:54 +01:00
Tudor Sitaru 5b025b98bd fix(dim_school): use case-insensitive comparison for phase inference
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 13s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 50s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m6s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
GIAS provides 'Not Applicable' (capital A) but the check used 'Not applicable',
so the case-sensitive != matched true and skipped the age-range inference.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-02 15:33:04 +01:00
Tudor Sitaru 4c3c3c882d fix(dim_school): infer phase from age range for independent schools
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 12s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 50s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m9s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Independent schools have phase='Not applicable' in GIAS. Now infer
phase from statutory age range: <=11 → Primary, >=11 → Secondary,
spans both → All-through. Falls back to original value if no age data.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-01 16:18:52 +01:00
tudor e7b1ab9f37 fix(pipeline): expand GIAS schema, handle empty strings, scope DAG selectors
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m8s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 34s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m39s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- Declare all 34 columns needed by dbt in GIAS tap schema (target-postgres
  only persists columns present in the Singer schema message)
- Use nullif() for empty-string-to-integer/date casts in staging models
- Scope daily DAG dbt build to GIAS models only (stg_gias_establishments+
  stg_gias_links+) to avoid errors on unloaded sources
- Scope annual EES DAG similarly; remove redundant dbt test steps
- Make dim_school gracefully handle missing int_ofsted_latest table

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 20:43:24 +00:00
tudor 8f02b5125e feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.

- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00