integrator/flows/sen_detail.yml

id: sen-detail-annual-update
namespace: schoolcompare.data
description: Download and load SEN primary need breakdown via EES API

triggers:
  - id: annual-schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 4 15 9 *"   # 15 September annually at 04:00

tasks:
  - id: download
    type: io.kestra.plugin.core.http.Request
    uri: http://integrator:8001/run/sen_detail?action=download
    method: POST
    timeout: PT20M

  - id: load
    type: io.kestra.plugin.core.http.Request
    uri: http://integrator:8001/run/sen_detail?action=load
    method: POST
    timeout: PT30M

retry:
  maxAttempts: 3
  delay: PT15M
feat(data): integrate 9 UK government data sources via Kestra Adds a full data integration pipeline for enriching school profiles with supplementary data from Ofsted, GIAS, EES, IDACI, and FBIT. Backend: - Bump SCHEMA_VERSION to 3; add 8 new DB tables (ofsted_inspections, ofsted_parent_view, school_census, admissions, sen_detail, phonics, school_deprivation, school_finance) plus GIAS columns on schools - Expose all supplementary data via GET /api/schools/{urn} - Enrich school list responses with ofsted_grade + ofsted_date Integrator (new service): - FastAPI HTTP microservice; Kestra calls POST /run/{source} - 9 source modules: ofsted, gias, parent_view, census, admissions, sen_detail, phonics, idaci, finance - 9 Kestra flow YAMLs with scheduled triggers and 3× retry Frontend: - SchoolRow: colour-coded Ofsted badge (Outstanding/Good/RI/Inadequate) - SchoolDetailView: 7 new sections — Ofsted sub-judgements, Parent View survey bars, Admissions, Pupils & Inclusion / SEN, Phonics, Deprivation Context, Finances - types.ts: 8 new interfaces + extended School/SchoolDetailsResponse Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> 2026-03-24 11:44:04 +00:00			`id: sen-detail-annual-update`
			`namespace: schoolcompare.data`
			`description: Download and load SEN primary need breakdown via EES API`

			`triggers:`
			`- id: annual-schedule`
			`type: io.kestra.plugin.core.trigger.Schedule`
			`cron: "0 4 15 9 *" # 15 September annually at 04:00`

			`tasks:`
			`- id: download`
			`type: io.kestra.plugin.core.http.Request`
			`uri: http://integrator:8001/run/sen_detail?action=download`
			`method: POST`
			`timeout: PT20M`

			`- id: load`
			`type: io.kestra.plugin.core.http.Request`
			`uri: http://integrator:8001/run/sen_detail?action=load`
			`method: POST`
			`timeout: PT30M`

			`retry:`
			`maxAttempts: 3`
			`delay: PT15M`