feat(data): integrate 9 UK government data sources via Kestra
Adds a full data integration pipeline for enriching school profiles with
supplementary data from Ofsted, GIAS, EES, IDACI, and FBIT.
Backend:
- Bump SCHEMA_VERSION to 3; add 8 new DB tables (ofsted_inspections,
ofsted_parent_view, school_census, admissions, sen_detail, phonics,
school_deprivation, school_finance) plus GIAS columns on schools
- Expose all supplementary data via GET /api/schools/{urn}
- Enrich school list responses with ofsted_grade + ofsted_date
Integrator (new service):
- FastAPI HTTP microservice; Kestra calls POST /run/{source}
- 9 source modules: ofsted, gias, parent_view, census, admissions,
sen_detail, phonics, idaci, finance
- 9 Kestra flow YAMLs with scheduled triggers and 3× retry
Frontend:
- SchoolRow: colour-coded Ofsted badge (Outstanding/Good/RI/Inadequate)
- SchoolDetailView: 7 new sections — Ofsted sub-judgements, Parent View
survey bars, Admissions, Pupils & Inclusion / SEN, Phonics, Deprivation
Context, Finances
- types.ts: 8 new interfaces + extended School/SchoolDetailsResponse
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -77,9 +77,70 @@ services:
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
# Kestra — workflow orchestrator (UI at http://localhost:8080)
|
||||
kestra:
|
||||
image: kestra/kestra:latest
|
||||
container_name: schoolcompare_kestra
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- kestra_storage:/app/storage
|
||||
- ./integrator/flows:/flows
|
||||
environment:
|
||||
KESTRA_CONFIGURATION: |
|
||||
datasources:
|
||||
postgres:
|
||||
url: jdbc:postgresql://db:5432/kestra
|
||||
driverClassName: org.postgresql.Driver
|
||||
username: schoolcompare
|
||||
password: schoolcompare
|
||||
kestra:
|
||||
repository:
|
||||
type: postgres
|
||||
queue:
|
||||
type: postgres
|
||||
storage:
|
||||
type: local
|
||||
local:
|
||||
base-path: /app/storage
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- schoolcompare-network
|
||||
restart: unless-stopped
|
||||
|
||||
# Data integrator — Python microservice called by Kestra
|
||||
integrator:
|
||||
build:
|
||||
context: ./integrator
|
||||
dockerfile: Dockerfile
|
||||
container_name: schoolcompare_integrator
|
||||
ports:
|
||||
- "8001:8001"
|
||||
environment:
|
||||
DATABASE_URL: postgresql://schoolcompare:schoolcompare@db:5432/schoolcompare
|
||||
DATA_DIR: /data
|
||||
PYTHONUNBUFFERED: 1
|
||||
volumes:
|
||||
- ./data:/data
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- schoolcompare-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8001/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
|
||||
networks:
|
||||
schoolcompare-network:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
kestra_storage:
|
||||
|
||||
Reference in New Issue
Block a user