Commit Graph
10 Commits
Author SHA1 Message Date
TudorandClaude Opus 5 0c901cd0d1 feat(ci): gate promotion on the image set that actually passed E2E
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m11s
PR Checks / Backend Smoke (pull_request) Successful in 9s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m19s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 36s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 6m3s
Staging health polling asked only whether something answered HTTP 200 at
the base URL. It could not tell the new deployment from the old one, so
journeys could pass against the previous release, and concurrent merges
could move the staging tags underneath a run in flight.

Each staging run now mints a build ID and stamps all three images with
the commit and that ID, as labels and — for frontend and backend — as a
build-time JSON file that environment overrides cannot rewrite.
/release.json reports both identities uncached, and scripts/ci/release.py
polls for the expected pair before and after the journeys. Only then are
the captured build digests tagged verified-<sha>.

Promotion resolves those verified tags to immutable digests, revalidates
their labels, and refuses a mixed or incomplete set before any :prod tag
moves. The whole staging workflow shares one concurrency group with
cancellation disabled, so releases serialise.

The scripts are stdlib-only and unit-tested against mocked registry and
HTTP behaviour; PR checks now run the pipeline and CI suites too. The
runbook records what this cannot prove locally, and that the first
rollout needs a commit built by this workflow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 10:17:50 +01:00
TudorandClaude Opus 5 cd2cbe7be6 build(pipeline): install the destinations tap in the image
meltano install would resolve it from pip_url, but five of the six custom
taps are also installed explicitly and a new plugin failing to appear is
not something you want to debug from a deploy log.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob
2026-08-31 17:37:46 +01:00
TudorandClaude Fable 5 95081d38bd chore: remove the Ofsted Parent View feature end to end
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m39s
PR Checks / Backend Smoke (pull_request) Successful in 5s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 41s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 45s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 1m35s
Removes the 'What Parents Say' section and all supporting elements:

Frontend:
- Drop the OfstedParentView type, the parent_view field, the survey
  section and the 'X% would recommend' callouts in the primary and
  secondary detail views, the Parents nav item, and the parent-view CSS.

Backend:
- Remove the FactParentView model, its loading in data_loader, and
  parent_view from the school-details API response.
- Bump SCHEMA_VERSION to 6 and add an idempotent drop step
  (DROP TABLE IF EXISTS marts.fact_parent_view) to the CLI migration;
  add scripts/sql/drop_fact_parent_view.sql to apply directly to the
  dbt-owned marts DBs on staging and prod.

Pipeline:
- Delete the stg_parent_view + fact_parent_view dbt models and their
  source/schema entries, the tap-uk-parent-view Meltano extractor, and
  the monthly Parent View DAG; drop it from the Dockerfile and the
  staging bootstrap docs.

The rest of dbt (which builds every mart the app reads) is untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 09:01:26 +01:00
TudorandClaude Opus 4.8 3785f0c977 fix(pipeline): invoke dbt via python module to avoid Fusion shadowing
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 53s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 56s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 2m0s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The standalone dbt Fusion binary (dbt-core 2.x) on PATH shadows the
pip-installed classic dbt-postgres ~=1.10 and rejects the Postgres
adapter (dbt1005), breaking every DAG's dbt_build task. Invoke dbt via
`python -m dbt.cli.main` in the DAGs and the Dockerfile dbt deps step so
the classic Postgres-capable engine is always used regardless of PATH.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:41:35 +01:00
tudorandClaude Opus 4.6 1c77a6c593 fix(pipeline): run meltano install in Dockerfile to generate catalogs
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m13s
Build and Push Docker Images / Build Integrator (push) Successful in 58s
Build and Push Docker Images / Build Kestra Init (push) Successful in 33s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Meltano elt requires catalog files (tap.properties.json) to exist.
These are generated by `meltano install` which discovers tap schemas
and installs the target-postgres loader. Without this step, `meltano
elt` fails with "catalog file is missing".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 12:28:59 +00:00
tudorandClaude Opus 4.6 b6a487776b fix(airflow): set DAGS_FOLDER in image env and reserialize on init
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
- Add AIRFLOW__CORE__DAGS_FOLDER env var in Dockerfile so it's always set
- Run `airflow dags reserialize` after `db migrate` in init container so
  DAGs appear immediately without waiting for scheduler scan interval

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 11:05:41 +00:00
tudorandClaude Opus 4.6 914de17d15 fix(pipeline): install curl in pipeline image for healthchecks
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m46s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 09:52:06 +00:00
tudorandClaude Opus 4.6 a7904b627d fix(pipeline): migrate to Airflow 3 API server and SimpleAuthManager
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Build and Push Docker Images / Build Integrator (push) Successful in 58s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Airflow 3 replaced `airflow webserver` with `airflow api-server` and
removed the `airflow users` CLI. Auth is now via SimpleAuthManager
configured through AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS env var.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 09:32:08 +00:00
tudorandClaude Opus 4.6 deb4024731 chore(pipeline): bump all dependencies to latest stable versions
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m45s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
- Airflow 2.11 → 3.1 (BashOperator moved to providers-standard)
- Meltano 3.5 → 4.1 (meltano.yml version 2, meltanolabs target-postgres)
- dbt-postgres 1.9 → 1.10
- singer-sdk 0.39 → 0.53 (all 6 taps)
- Typesense Docker 27.1 → 30.1
- Typesense Python client >=2.0
- Python base image 3.12 → 3.13

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 09:18:11 +00:00
tudorandClaude Opus 4.6 8f02b5125e feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.

- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 08:37:53 +00:00