The MI CSV contains both OEIF and RC column sets simultaneously — OEIF columns
are populated for older inspections, RC columns for post-Nov-2025 inspections.
File-level detection wrongly classified all schools based on column presence alone.
Replace _detect_framework(df) with _framework_for_row(row):
- ReportCard: any rc_* column has a value
- OEIF: overall_effectiveness or quality_of_education has a value
- None: neither has data (no graded inspection on record)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The old OEIF CSV contains columns whose names include substrings like
'inclusion' and 'achievement', causing _detect_framework() to wrongly return
'ReportCard' for pre-Nov-2025 inspections.
Fix: check for OEIF-specific phrases first ('overall effectiveness', 'quality
of education', 'behaviour and attitudes'). Only if none are found, look for
multi-word RC-specific phrases. Default to OEIF as a safe fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The v4 migration already ran before _apply_schema_alterations() was added,
so the new ofsted_inspections columns were never created. Bump to v5 so the
next backend restart re-runs the migration and applies the ALTER TABLE statements.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
create_all() only creates missing tables; it won't modify tables that already
exist from an older schema version. Add _apply_schema_alterations() which runs
idempotent ADD COLUMN IF NOT EXISTS statements after every migration so
supplementary tables (like ofsted_inspections) gain new columns without
dropping their existing data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ofsted replaced single overall grades with Report Cards from Nov 2025.
Both systems are retained during the transition period.
- DB: new framework + 9 RC columns on ofsted_inspections (schema v4)
- Integrator: auto-detect OEIF vs Report Card from CSV column headers;
parse 5-level RC grades and safeguarding met/not-met
- API: expose all new fields in the ofsted response dict
- Frontend: branch on framework='ReportCard' to show safeguarding badge
+ 8-category grid; fall back to legacy OEIF layout otherwise;
always show inspection date in both layouts
- CSS: rcGrade1–5 and safeguardingMet/NotMet classes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove standalone back button div (looked out of place)
- Back button now lives in the sticky section nav bar, styled as a
bordered pill with coral accent — consistent with page design
- Fix sticky nav top offset from 0 to 3rem so it sticks below the
site-wide header instead of sliding behind it
- Increase scroll-margin-top on cards to 6rem to account for both
site header and section nav height
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two issues caused the backend to drop and reimport school data on restart:
1. schema_version table was in the drop list inside run_full_migration(),
so after any migration the breadcrumb was destroyed and the next
restart would see no version → re-trigger migration
2. Schema version was set after migration, so a crash mid-migration
left no version → infinite re-migration loop
Fix: remove schema_version from the drop list, and set the version
before running migration so crashes don't cause loops.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UX audit round 2:
- Remove Summary Strip (duplicated Ofsted grade + parent happy/safe/recommend)
- Fold "% would recommend" into Ofsted section header
- Merge SATs Results + Subject Breakdown into one section
- Merge Results Over Time chart + Year-by-Year table into one section
- Add sticky section nav with dynamic pills based on available data
- Unify colour system: replace ad-hoc pill colours with semantic status classes
- Guard Pupils & Inclusion so it only renders with actual data
- Add year to Admissions section title
- Fix progress score 0.0 colour (was neutral gap at ±0.1, now at 0)
- Remove unused .metricTrend CSS class
Page reduced from 16 to 13 sections.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The EES statistics API only exposes ~13 publications; admissions data is not
among them. Switch to the EES content API (content.explore-education-statistics.
service.gov.uk) which covers all publications.
- ees.py: add get_content_release_id() and download_release_zip_csv() that
fetch the release ZIP and extract a named CSV member from it
- admissions.py: use corrected slug (primary-and-secondary-school-applications-
and-offers), correct column names from actual CSV (school_urn,
total_number_places_offered, times_put_as_1st_preference, etc.), derive
first_preference_offers_pct from offer/application ratio, filter to primary
schools only, keep most recent year per URN
Also includes SchoolDetailView UX redesign: parent-first section ordering,
plain-English labels, national average benchmarks, progress score colour
coding, expanded header, quick summary strip, and CSS consolidation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before dropping tables, save all existing lat/lon coordinates keyed by URN.
After reimport, merge cached coordinates with any newly geocoded ones so
schools that already have coordinates skip the postcodes.io API call.
This makes repeated reimports fast and avoids re-geocoding ~15k schools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ofsted renamed all columns in the OEIF framework:
- grades are now 'Latest OEIF overall effectiveness' etc.
- dates are 'Inspection start date of latest OEIF graded inspection'
Replace flat COLUMN_MAP with a priority list per field so both current
OEIF and legacy column names work without duplicate-column conflicts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kestra's HTTP client socket read timeout is shorter than any reasonable
wait for a full geocoded migration. POST /api/admin/reimport-ks2 returns
immediately with {status:started}; the backend runs the job in a thread.
Check GET /api/admin/reimport-ks2/status or watch the UI for schools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ofsted CSV has a variable number of preamble rows (title, filter warning,
etc.) before the real column headers. Scan up to 10 rows to find the one
containing a URN column rather than assuming a fixed offset.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The geocoding pass over ~15k schools takes longer than any reasonable
HTTP timeout. New approach:
- POST /api/admin/reimport-ks2 starts migration in background thread,
returns {"status":"started"} immediately
- GET /api/admin/reimport-ks2/status returns {running, done}
- ks2.py polls status every 30s (max 2h) before returning
- Kestra flow timeout bumped to PT2H
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add geocode query param to /api/admin/reimport-ks2 (defaults true).
ks2.py passes ?geocode=true so postcodes are resolved to lat/lng in
the same migration pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kestra requires retry.type to be set (e.g. constant, exponential).
Also rename delay -> interval which is the correct field for constant retry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Waits up to 120s for /api/v1/flows/search to respond before attempting
imports, giving a clearer error if the URL is wrong or kestra isn't up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- POST /api/v1/flows with Content-Type: application/x-yaml (not the
ZIP-based /import endpoint)
- On 409 (already exists), fall back to PUT /api/v1/flows/{ns}/{id}
so redeployment updates existing flows rather than failing
- Print HTTP response body on error for easier debugging
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port 8081 /health responds as soon as Kestra is up; the flows/search
API on 8080 can be slow or return errors during startup.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kestra/kestra:latest is ~500MB; the registry rejects the push.
The init container only needs to POST flow YAMLs to the Kestra REST API
(/api/v1/flows/import), which curl handles fine from a tiny alpine base.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All four custom services now pull pre-built images from the registry
instead of building on the host. Also switches the integrator data
volume to a named volume (supplementary_data) since bind mounts to
./data won't exist on the Portainer host.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bind mounts don't work on the remote Portainer host since the files
aren't present there. Instead, Dockerfile.init copies the flow YAMLs
into a dedicated image (kestra/kestra:latest base) that is built in CI
and pulled by Portainer like the other images.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--flow-path is not a valid Kestra flag; flows must be pushed explicitly.
kestra-init waits for Kestra to pass its healthcheck then runs
'kestra flow namespace update schoolcompare.data /flows --no-delete'
to import all flow YAMLs. Runs once on stack start; --no-delete means
any flows created manually in the UI are not removed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without an explicit command the container prints help and exits,
causing Docker to restart it indefinitely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- backend: POST /api/admin/reimport-ks2 runs full CSV migration in a thread
- backend/docker-compose: ADMIN_API_KEY env var (default: changeme) so the
key is stable across restarts and the integrator can call the endpoint
- integrator: sources/ks2.py triggers the backend endpoint (900s timeout)
- integrator: flows/ks2.yml Kestra flow (manual trigger, no schedule)
To re-ingest after a DB wipe: trigger the ks2-reimport flow from the
Kestra UI at http://localhost:8080.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a full data integration pipeline for enriching school profiles with
supplementary data from Ofsted, GIAS, EES, IDACI, and FBIT.
Backend:
- Bump SCHEMA_VERSION to 3; add 8 new DB tables (ofsted_inspections,
ofsted_parent_view, school_census, admissions, sen_detail, phonics,
school_deprivation, school_finance) plus GIAS columns on schools
- Expose all supplementary data via GET /api/schools/{urn}
- Enrich school list responses with ofsted_grade + ofsted_date
Integrator (new service):
- FastAPI HTTP microservice; Kestra calls POST /run/{source}
- 9 source modules: ofsted, gias, parent_view, census, admissions,
sen_detail, phonics, idaci, finance
- 9 Kestra flow YAMLs with scheduled triggers and 3× retry
Frontend:
- SchoolRow: colour-coded Ofsted badge (Outstanding/Good/RI/Inadequate)
- SchoolDetailView: 7 new sections — Ofsted sub-judgements, Parent View
survey bars, Admissions, Pupils & Inclusion / SEN, Phonics, Deprivation
Context, Finances
- types.ts: 8 new interfaces + extended School/SchoolDetailsResponse
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Line 1: school name (bold) + school type (muted gray)
Line 2: R,W&M % · Progress score + band · Pupil count
Line 3: local authority · distance (location searches)
Actions (View / Add) are vertically centred on the right across all lines.
Progress uses reading score, falling back to writing then maths. Removed
the old nameScore grouping and separate meta/progress rows in favour of
the cleaner 3-line structure.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Set explicit height:2rem, line-height:1, font-family:inherit on all children
of button group containers. Browsers apply different default line-height and
font-family to <button> vs <a>, causing height differences that persist even
with identical padding and display:inline-flex.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
<a> tags are display:inline by default and don't respect vertical padding,
while <button> is inline-block. Mixed anchor/button pairs (View/Add) rendered
at different heights despite identical padding. Apply display:inline-flex +
align-items:center to every button-styled element across SchoolRow, RankingsView,
and SchoolCard. Add border:1px solid transparent to borderless buttons so total
box size matches bordered siblings.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Group school name and score together on the left using a nameScore flex
container, so the percentage sits close to the name rather than pushed to the
far right. Action buttons get slightly more padding on desktop (0.4375rem v
0.3125rem). On mobile the scoreLabel is now visible inline instead of hidden,
so the percentage reads as R,W&M not a bare number.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FilterBar was sending radius in km (e.g. 1.6) but the backend expects miles,
causing the "Showing schools within X miles" banner to display the wrong value.
Change option values to miles (0.5, 1, 3, 5, 10) and default from 1.6 to 1.
school.distance from the API is already in miles (backend haversine uses
R=3959). SchoolRow was dividing by 1609.34 giving 0.0 mi; CompactSchoolItem
was dividing by 1.60934. Both now display school.distance directly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the school card grid with a scannable row list that shows 3x more
results per screen. Each row shows: school name + R,W&M % with trend,
area/type meta, and reading/writing/maths progress scores with plain-English
band labels (e.g. "above average") instead of raw numbers.
Add lib/metrics.ts as a single source of truth for plain-language metric
explanations and the progressBand() helper. Map view toggle is unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Addresses 28 issues identified in UX audit (P0–P3 severity):
P0 — Critical:
- Fix compare URL sharing: seed ComparisonContext from SSR initialData
when localStorage is empty, making /compare?urns=... links shareable
- Remove permanently broken "Avg. Scaled Score" column from school
detail historical data table
P1 — High priority:
- Add radius selector (0.5–10 mi) to postcode search in FilterBar
- Make Add to Compare a toggle (remove) on SchoolCards
- Hide hero title/description once a search is active
- Show school count + quick-search prompts on empty landing page
- Compare empty state opens in-page school search modal directly
- Remove URN from school detail header (irrelevant to end users)
- Move map above performance chart in school detail page
- Add ← Back navigation to school detail page
- Add sort controls to search results (RWM%, distance, A–Z)
- Show metric descriptions below metric selector
- Expand ComparisonToast to list school names with per-school remove
- Add progress score explainer (0 = national average) throughout
P2 — Medium:
- Remove console.log statements from ComparisonView
- Colour-code comparison school cards to match chart line colours
- Replace plain loading text with LoadingSkeleton in ComparisonView
- Rankings empty state uses shared EmptyState component
- Rankings year filter shows actual year e.g. "2023 (Latest)"
- Rankings subtitle shows top-N count
- Add View link alongside Add button in rankings table
- Remove placeholder Privacy Policy / Terms links from footer
- Replace untappable 10px info icons with visible metric hint text
- Show active filter chips in search results header
P3 — Polish:
- Remove redundant "Home" nav link (logo already links home)
- Add / and Ctrl+K keyboard shortcut to focus search input
- Add Share button to compare page (copies URL to clipboard)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Hide empty state placeholder on initial load
- Add prominent hero mode to FilterBar when no search is active
- Fix SchoolCard test TypeScript and assertion errors
- Redesign landing page with unified Omnibox search
- Add ComparisonToast for better comparison flow visibility
- Add visual 'Added' state to SchoolCard
- Add info tooltips to educational metrics
- Optimize mobile map view with Bottom Sheet
- Standardize distance display to miles
