feat(data): integrate 9 UK government data sources via Kestra
Some checks failed
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 47s
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Frontend (Next.js) (push) Has been cancelled

Adds a full data integration pipeline for enriching school profiles with
supplementary data from Ofsted, GIAS, EES, IDACI, and FBIT.

Backend:
- Bump SCHEMA_VERSION to 3; add 8 new DB tables (ofsted_inspections,
  ofsted_parent_view, school_census, admissions, sen_detail, phonics,
  school_deprivation, school_finance) plus GIAS columns on schools
- Expose all supplementary data via GET /api/schools/{urn}
- Enrich school list responses with ofsted_grade + ofsted_date

Integrator (new service):
- FastAPI HTTP microservice; Kestra calls POST /run/{source}
- 9 source modules: ofsted, gias, parent_view, census, admissions,
  sen_detail, phonics, idaci, finance
- 9 Kestra flow YAMLs with scheduled triggers and 3× retry

Frontend:
- SchoolRow: colour-coded Ofsted badge (Outstanding/Good/RI/Inadequate)
- SchoolDetailView: 7 new sections — Ofsted sub-judgements, Parent View
  survey bars, Admissions, Pupils & Inclusion / SEN, Phonics, Deprivation
  Context, Finances
- types.ts: 8 new interfaces + extended School/SchoolDetailsResponse

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-24 11:44:04 +00:00
parent c49593d4d6
commit dd49ef28b2
36 changed files with 2849 additions and 8 deletions

View File

@@ -77,9 +77,70 @@ services:
retries: 3
start_period: 40s
# Kestra — workflow orchestrator (UI at http://localhost:8080)
kestra:
image: kestra/kestra:latest
container_name: schoolcompare_kestra
ports:
- "8080:8080"
volumes:
- kestra_storage:/app/storage
- ./integrator/flows:/flows
environment:
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://db:5432/kestra
driverClassName: org.postgresql.Driver
username: schoolcompare
password: schoolcompare
kestra:
repository:
type: postgres
queue:
type: postgres
storage:
type: local
local:
base-path: /app/storage
depends_on:
db:
condition: service_healthy
networks:
- schoolcompare-network
restart: unless-stopped
# Data integrator — Python microservice called by Kestra
integrator:
build:
context: ./integrator
dockerfile: Dockerfile
container_name: schoolcompare_integrator
ports:
- "8001:8001"
environment:
DATABASE_URL: postgresql://schoolcompare:schoolcompare@db:5432/schoolcompare
DATA_DIR: /data
PYTHONUNBUFFERED: 1
volumes:
- ./data:/data
depends_on:
db:
condition: service_healthy
networks:
- schoolcompare-network
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8001/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
networks:
schoolcompare-network:
driver: bridge
volumes:
postgres_data:
kestra_storage: