feat(data): integrate 9 UK government data sources via Kestra
Adds a full data integration pipeline for enriching school profiles with
supplementary data from Ofsted, GIAS, EES, IDACI, and FBIT.
Backend:
- Bump SCHEMA_VERSION to 3; add 8 new DB tables (ofsted_inspections,
ofsted_parent_view, school_census, admissions, sen_detail, phonics,
school_deprivation, school_finance) plus GIAS columns on schools
- Expose all supplementary data via GET /api/schools/{urn}
- Enrich school list responses with ofsted_grade + ofsted_date
Integrator (new service):
- FastAPI HTTP microservice; Kestra calls POST /run/{source}
- 9 source modules: ofsted, gias, parent_view, census, admissions,
sen_detail, phonics, idaci, finance
- 9 Kestra flow YAMLs with scheduled triggers and 3× retry
Frontend:
- SchoolRow: colour-coded Ofsted badge (Outstanding/Good/RI/Inadequate)
- SchoolDetailView: 7 new sections — Ofsted sub-judgements, Parent View
survey bars, Admissions, Pupils & Inclusion / SEN, Phonics, Deprivation
Context, Finances
- types.ts: 8 new interfaces + extended School/SchoolDetailsResponse
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
25
integrator/flows/admissions.yml
Normal file
25
integrator/flows/admissions.yml
Normal file
@@ -0,0 +1,25 @@
|
||||
id: admissions-annual-update
|
||||
namespace: schoolcompare.data
|
||||
description: Download and load school admissions data via EES API
|
||||
|
||||
triggers:
|
||||
- id: annual-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 4 1 7 *" # 1 July annually at 04:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/admissions?action=download
|
||||
method: POST
|
||||
timeout: PT20M
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/admissions?action=load
|
||||
method: POST
|
||||
timeout: PT30M
|
||||
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
delay: PT15M
|
||||
25
integrator/flows/census.yml
Normal file
25
integrator/flows/census.yml
Normal file
@@ -0,0 +1,25 @@
|
||||
id: census-annual-update
|
||||
namespace: schoolcompare.data
|
||||
description: Download and load School Census (SPC) data via EES API
|
||||
|
||||
triggers:
|
||||
- id: annual-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 4 1 9 *" # 1 September annually at 04:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/census?action=download
|
||||
method: POST
|
||||
timeout: PT20M
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/census?action=load
|
||||
method: POST
|
||||
timeout: PT30M
|
||||
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
delay: PT15M
|
||||
25
integrator/flows/finance.yml
Normal file
25
integrator/flows/finance.yml
Normal file
@@ -0,0 +1,25 @@
|
||||
id: finance-annual-update
|
||||
namespace: schoolcompare.data
|
||||
description: Fetch FBIT financial benchmarking data from DfE API for all schools
|
||||
|
||||
triggers:
|
||||
- id: annual-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 4 1 12 *" # 1 December annually at 04:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/finance?action=download
|
||||
method: POST
|
||||
timeout: PT120M # Fetches per-school from API — ~20k schools
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/finance?action=load
|
||||
method: POST
|
||||
timeout: PT30M
|
||||
|
||||
retry:
|
||||
maxAttempts: 2
|
||||
delay: PT30M
|
||||
30
integrator/flows/gias.yml
Normal file
30
integrator/flows/gias.yml
Normal file
@@ -0,0 +1,30 @@
|
||||
id: gias-weekly-update
|
||||
namespace: schoolcompare.data
|
||||
description: Download and load GIAS (Get Information About Schools) bulk CSV
|
||||
|
||||
triggers:
|
||||
- id: weekly-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 3 * * 0" # Every Sunday at 03:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/gias?action=download
|
||||
method: POST
|
||||
timeout: PT30M
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/gias?action=load
|
||||
method: POST
|
||||
timeout: PT30M
|
||||
|
||||
errors:
|
||||
- id: notify-failure
|
||||
type: io.kestra.plugin.core.log.Log
|
||||
message: "GIAS update FAILED: {{ error.message }}"
|
||||
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
delay: PT10M
|
||||
25
integrator/flows/idaci.yml
Normal file
25
integrator/flows/idaci.yml
Normal file
@@ -0,0 +1,25 @@
|
||||
id: idaci-annual-check
|
||||
namespace: schoolcompare.data
|
||||
description: Download IoD2019 IDACI file and compute deprivation scores for all schools
|
||||
|
||||
triggers:
|
||||
- id: annual-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 5 1 1 *" # 1 January annually at 05:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/idaci?action=download
|
||||
method: POST
|
||||
timeout: PT10M
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/idaci?action=load
|
||||
method: POST
|
||||
timeout: PT60M
|
||||
|
||||
retry:
|
||||
maxAttempts: 2
|
||||
delay: PT30M
|
||||
32
integrator/flows/ofsted.yml
Normal file
32
integrator/flows/ofsted.yml
Normal file
@@ -0,0 +1,32 @@
|
||||
id: ofsted-monthly-update
|
||||
namespace: schoolcompare.data
|
||||
description: Download and load Ofsted Monthly Management Information CSV
|
||||
|
||||
triggers:
|
||||
- id: monthly-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 2 1 * *" # 1st of each month at 02:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/ofsted?action=download
|
||||
method: POST
|
||||
allowFailed: false
|
||||
timeout: PT10M
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/ofsted?action=load
|
||||
method: POST
|
||||
allowFailed: false
|
||||
timeout: PT30M
|
||||
|
||||
errors:
|
||||
- id: notify-failure
|
||||
type: io.kestra.plugin.core.log.Log
|
||||
message: "Ofsted update FAILED: {{ error.message }}"
|
||||
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
delay: PT10M
|
||||
30
integrator/flows/parent_view.yml
Normal file
30
integrator/flows/parent_view.yml
Normal file
@@ -0,0 +1,30 @@
|
||||
id: parent-view-monthly-check
|
||||
namespace: schoolcompare.data
|
||||
description: Download and load Ofsted Parent View open data (released ~3x/year)
|
||||
|
||||
triggers:
|
||||
- id: monthly-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 3 1 * *" # 1st of each month at 03:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/parent_view?action=download
|
||||
method: POST
|
||||
timeout: PT10M
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/parent_view?action=load
|
||||
method: POST
|
||||
timeout: PT20M
|
||||
|
||||
errors:
|
||||
- id: notify-failure
|
||||
type: io.kestra.plugin.core.log.Log
|
||||
message: "Parent View update FAILED: {{ error.message }}"
|
||||
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
delay: PT10M
|
||||
25
integrator/flows/phonics.yml
Normal file
25
integrator/flows/phonics.yml
Normal file
@@ -0,0 +1,25 @@
|
||||
id: phonics-annual-update
|
||||
namespace: schoolcompare.data
|
||||
description: Download and load Phonics Screening Check data via EES API
|
||||
|
||||
triggers:
|
||||
- id: annual-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 5 1 9 *" # 1 September annually at 05:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/phonics?action=download
|
||||
method: POST
|
||||
timeout: PT20M
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/phonics?action=load
|
||||
method: POST
|
||||
timeout: PT30M
|
||||
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
delay: PT15M
|
||||
25
integrator/flows/sen_detail.yml
Normal file
25
integrator/flows/sen_detail.yml
Normal file
@@ -0,0 +1,25 @@
|
||||
id: sen-detail-annual-update
|
||||
namespace: schoolcompare.data
|
||||
description: Download and load SEN primary need breakdown via EES API
|
||||
|
||||
triggers:
|
||||
- id: annual-schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 4 15 9 *" # 15 September annually at 04:00
|
||||
|
||||
tasks:
|
||||
- id: download
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/sen_detail?action=download
|
||||
method: POST
|
||||
timeout: PT20M
|
||||
|
||||
- id: load
|
||||
type: io.kestra.plugin.core.http.Request
|
||||
uri: http://integrator:8001/run/sen_detail?action=load
|
||||
method: POST
|
||||
timeout: PT30M
|
||||
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
delay: PT15M
|
||||
Reference in New Issue
Block a user