feat(data): integrate 9 UK government data sources via Kestra
Adds a full data integration pipeline for enriching school profiles with
supplementary data from Ofsted, GIAS, EES, IDACI, and FBIT.
Backend:
- Bump SCHEMA_VERSION to 3; add 8 new DB tables (ofsted_inspections,
ofsted_parent_view, school_census, admissions, sen_detail, phonics,
school_deprivation, school_finance) plus GIAS columns on schools
- Expose all supplementary data via GET /api/schools/{urn}
- Enrich school list responses with ofsted_grade + ofsted_date
Integrator (new service):
- FastAPI HTTP microservice; Kestra calls POST /run/{source}
- 9 source modules: ofsted, gias, parent_view, census, admissions,
sen_detail, phonics, idaci, finance
- 9 Kestra flow YAMLs with scheduled triggers and 3× retry
Frontend:
- SchoolRow: colour-coded Ofsted badge (Outstanding/Good/RI/Inadequate)
- SchoolDetailView: 7 new sections — Ofsted sub-judgements, Parent View
survey bars, Admissions, Pupils & Inclusion / SEN, Phonics, Deprivation
Context, Finances
- types.ts: 8 new interfaces + extended School/SchoolDetailsResponse
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 11:44:04 +00:00
|
|
|
id: finance-annual-update
|
|
|
|
|
namespace: schoolcompare.data
|
|
|
|
|
description: Fetch FBIT financial benchmarking data from DfE API for all schools
|
|
|
|
|
|
|
|
|
|
triggers:
|
|
|
|
|
- id: annual-schedule
|
|
|
|
|
type: io.kestra.plugin.core.trigger.Schedule
|
|
|
|
|
cron: "0 4 1 12 *" # 1 December annually at 04:00
|
|
|
|
|
|
|
|
|
|
tasks:
|
|
|
|
|
- id: download
|
|
|
|
|
type: io.kestra.plugin.core.http.Request
|
|
|
|
|
uri: http://integrator:8001/run/finance?action=download
|
|
|
|
|
method: POST
|
|
|
|
|
timeout: PT120M # Fetches per-school from API — ~20k schools
|
|
|
|
|
|
|
|
|
|
- id: load
|
|
|
|
|
type: io.kestra.plugin.core.http.Request
|
|
|
|
|
uri: http://integrator:8001/run/finance?action=load
|
|
|
|
|
method: POST
|
|
|
|
|
timeout: PT30M
|
|
|
|
|
|
|
|
|
|
retry:
|
2026-03-24 20:08:17 +00:00
|
|
|
type: constant
|
feat(data): integrate 9 UK government data sources via Kestra
Adds a full data integration pipeline for enriching school profiles with
supplementary data from Ofsted, GIAS, EES, IDACI, and FBIT.
Backend:
- Bump SCHEMA_VERSION to 3; add 8 new DB tables (ofsted_inspections,
ofsted_parent_view, school_census, admissions, sen_detail, phonics,
school_deprivation, school_finance) plus GIAS columns on schools
- Expose all supplementary data via GET /api/schools/{urn}
- Enrich school list responses with ofsted_grade + ofsted_date
Integrator (new service):
- FastAPI HTTP microservice; Kestra calls POST /run/{source}
- 9 source modules: ofsted, gias, parent_view, census, admissions,
sen_detail, phonics, idaci, finance
- 9 Kestra flow YAMLs with scheduled triggers and 3× retry
Frontend:
- SchoolRow: colour-coded Ofsted badge (Outstanding/Good/RI/Inadequate)
- SchoolDetailView: 7 new sections — Ofsted sub-judgements, Parent View
survey bars, Admissions, Pupils & Inclusion / SEN, Phonics, Deprivation
Context, Finances
- types.ts: 8 new interfaces + extended School/SchoolDetailsResponse
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 11:44:04 +00:00
|
|
|
maxAttempts: 2
|
2026-03-24 20:08:17 +00:00
|
|
|
interval: PT30M
|