school_compare

5 Commits

Include renames

Author	SHA1	Message	Date
tudorandClaude Opus 4.6	e7b1ab9f37	fix(pipeline): expand GIAS schema, handle empty strings, scope DAG selectors Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s Details Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m8s Details Build and Push Docker Images / Build Integrator (push) Successful in 57s Details Build and Push Docker Images / Build Kestra Init (push) Successful in 34s Details Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m39s Details Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s Details - Declare all 34 columns needed by dbt in GIAS tap schema (target-postgres only persists columns present in the Singer schema message) - Use nullif() for empty-string-to-integer/date casts in staging models - Scope daily DAG dbt build to GIAS models only (stg_gias_establishments+ stg_gias_links+) to avoid errors on unloaded sources - Scope annual EES DAG similarly; remove redundant dbt test steps - Make dim_school gracefully handle missing int_ofsted_latest table Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-26 20:43:24 +00:00
tudorandClaude Opus 4.6	c576bba06a	fix(meltano): remove catalog capability and switch elt to run Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s Details Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s Details Build and Push Docker Images / Build Integrator (push) Successful in 57s Details Build and Push Docker Images / Build Kestra Init (push) Successful in 31s Details Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m26s Details Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s Details The `catalog` capability forced Meltano to run --discover and generate a catalog file (tap.properties.json) before each extraction. This fails because our Singer SDK taps emit schemas inline and don't need external catalog files. Removing the capability makes Meltano invoke taps directly without catalog generation. Also switch from deprecated `meltano elt` to `meltano run` for Meltano 4.x compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-26 13:45:23 +00:00
tudorandClaude Opus 4.6	e815f597ab	fix(dags): use global bin paths and add BashOperator import fallback Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s Details Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s Details Build and Push Docker Images / Build Integrator (push) Successful in 56s Details Build and Push Docker Images / Build Kestra Init (push) Successful in 31s Details Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 49s Details Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s Details - MELTANO_BIN/DBT_BIN pointed to .venv/bin/ but Dockerfile installs globally - Add try/except for BashOperator import to handle both Airflow 3 provider path and legacy path, preventing silent DAG import failures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-26 10:47:18 +00:00
tudorandClaude Opus 4.6	deb4024731	chore(pipeline): bump all dependencies to latest stable versions Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s Details Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s Details Build and Push Docker Images / Build Integrator (push) Successful in 57s Details Build and Push Docker Images / Build Kestra Init (push) Successful in 32s Details Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m45s Details Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s Details - Airflow 2.11 → 3.1 (BashOperator moved to providers-standard) - Meltano 3.5 → 4.1 (meltano.yml version 2, meltanolabs target-postgres) - dbt-postgres 1.9 → 1.10 - singer-sdk 0.39 → 0.53 (all 6 taps) - Typesense Docker 27.1 → 30.1 - Typesense Python client >=2.0 - Python base image 3.12 → 3.13 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-26 09:18:11 +00:00
tudorandClaude Opus 4.6	8f02b5125e	feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s Details Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s Details Build and Push Docker Images / Build Integrator (push) Successful in 56s Details Build and Push Docker Images / Build Kestra Init (push) Successful in 32s Details Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s Details Replaces the hand-rolled integrator with a production-grade ELT pipeline using Meltano (Singer taps), dbt Core (medallion architecture), and Apache Airflow (orchestration). Adds Typesense for search and PostGIS for geospatial queries. - 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI) - dbt project: 12 staging, 5 intermediate, 12 mart models - 3 Airflow DAGs (daily/monthly/annual schedules) - Typesense sync + batch geocoding scripts - docker-compose: add Airflow, Typesense; upgrade to PostGIS - Portainer stack definition matching live deployment topology Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-26 08:37:53 +00:00