Commit Graph
115 Commits
Author SHA1 Message Date
TudorandClaude Opus 5 c5af476213 feat(places): /api/places registry and place detail endpoints
The registry is cached for the process and reset by the same admin endpoint
that rebuilds the sitemaps, so places and sitemap always describe the same
corpus rather than drifting apart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-21 18:13:05 +01:00
TudorandClaude Opus 5 de853b90b3 feat(places): London localities and postcode districts
The GIAS town field puts 1,819 London schools under the single value
'London', so it cannot answer 'schools in Battersea' — a query that appears in
the baseline. No single field can: parliamentary constituency gives Battersea
but not Canary Wharf, admin_ward gives Canary Wharf but not Battersea, and
neither gives Clapham or Shoreditch. So a locality is curated, defined by the
postcode districts it covers, which needs no new ingestion.

A locality may not shadow a published town: the registry raises rather than
silently costing a page that carries real demand. One below the threshold is
logged rather than raising, because a locality can legitimately be too small.

The pipeline seed mirrors the module, with a test guarding the drift — the
same arrangement gias_codes has, and for the same reason: the backend image
does not contain pipeline/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-21 18:11:56 +01:00
TudorandClaude Opus 5 759d9f5cea feat(places): registry of towns and authorities
Two namespaces because 67 town names collide with an authority name and
neither set contains the other — postal towns cross authority boundaries, so
Bedford the town holds 104 schools against the authority's 86.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-21 18:10:37 +01:00
TudorandClaude Opus 5 07c97a46c5 fix(seo): a school is publishable on any year's results, not the latest
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m2s
PR Checks / Backend Smoke (pull_request) Successful in 8s
PR Checks / Build Backend (no push) (pull_request) Successful in 16s
PR Checks / Build Frontend (no push) (pull_request) Successful in 44s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 53s
_school_sitemap_rows tested only the latest year's row, which quietly dropped
every school with results in its history but a null row for the most recent
year — a school that stopped reporting, or whose figures were suppressed for
small-cohort disclosure.

The Mallard Academy (150367) is the case that caught it: real KS2 results for
2015-16 through 2018-19, then null rows from 2022-23 on. Its detail page shows
all four years; the sitemap omitted it. Sampling 40 of the 2,206 excluded
schools found 4 like this, so roughly 220 real pages were being withheld.

Publishable is now a property of the school, computed across every row, while
lastmod still comes from the latest row so the most recent Ofsted date wins.
The field list is a module constant shared with _has_publishable_data so the
two checks cannot drift.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-21 00:04:32 +01:00
TudorandClaude Opus 5 2208ad93c1 feat(seo): split the sitemap into a per-family index
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m4s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 44s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m2s
Search Console reports coverage per submitted sitemap, so one file per page
family is what will make W2's location pages measurable when they land. The
index's lastmod is generation time, which is the correct semantic there —
unlike on a <url>, where it would be a claim we cannot support.

Children sit under /sitemaps/ because Next only treats a whole bracketed path
segment as dynamic; a route folder named sitemap-[...parts] would be read as a
literal static segment and never match. Confirmed by the build output, which
lists /sitemaps/[...parts] as a dynamic route.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-20 23:15:00 +01:00
TudorandClaude Opus 5 24f3cb4c65 fix(seo): submit only school pages that have something to show
Drops the schools with neither results nor an Ofsted grade, adds /admissions
which was never listed, replaces the invented priority and changefreq with a
lastmod taken from each school's Ofsted date.

lastmod is omitted where no date is known rather than defaulted to now. An
always-now lastmod is a claim Google learns to distrust; absent honestly
means unknown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-20 23:15:00 +01:00
TudorandClaude Opus 5 5ddc7314fd fix(seo): canonicalise on the www host, which is the one that serves 200
The apex 301s to www at Cloudflare, but metadataBase, the school-page
canonical, robots.txt's Sitemap: line and the sitemap's own <loc> entries all
named the apex. Every one of those pointed Google at a redirect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-20 23:15:00 +01:00
TudorandClaude Opus 5 c9a1892bfb feat(admissions): publish the latest cut-off only, holding history back
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m4s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 16s
PR Checks / Build Frontend (no push) (pull_request) Successful in 45s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 11s
PR Checks / AI Code Review (Claude) (pull_request) Failing after 3m17s
Earlier years are to become a paid feature, so they stop being published.

The load-bearing part is that this is a change to the API, not only to the
page. /api/schools/{urn} is public and unauthenticated: leaving
admission_distance_history in the payload while declining to render it would
have handed the whole record to anyone who opened the network tab. It is
withheld at the source, and the page follows.

Nothing changes upstream. The tap, the plausibility band and
fact_admission_distance are untouched and still load every published year, so
restoring history for entitled callers is a change to one function in
data_loader rather than a re-collection.

What the reader now gets is the latest figure on the Admissions tile, and a
Distance section that answers the question the number alone cannot: whether
their own address falls inside it. Retitled to "How far away are you?", which
is what it now does — the previous title described a record that is no longer
there.

Removed with the history: the trend chart, the year-by-year table, the
per-year verdict strip, the trend summary and the coverage note, along with
their CSS. The section goes from 743px to 417px.

One consequence worth naming. A run of years used to soften a single close
call — a home just outside one year's cut-off was usually inside another. With
one year published, the "too close to call" band is the entire safety margin
between a parent and a place they do not have, so the verdict now names its
year, and the three outcomes are tinted apart rather than distinguished by
wording alone.

The existing stylesheet test earned its keep here: the three verdict classes
were referenced before they were written, and it caught them. Unstyled, a
"beyond the cut-off" result would have been indistinguishable from an "inside"
one — the exact failure the longhand class map was written to prevent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WDvkyqqHABm4bmth2kjAxE
2026-08-20 18:44:57 +01:00
TudorandClaude Opus 5 a72323874f feat(admissions): add the cut-off history, map and postcode check
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m3s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 44s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m39s
Completes the last-distance-offered feature against the mockup: the
year-by-year record, the same numbers drawn over real streets, and the
reader's own address measured against them.

Serving the history
  The first cut deliberately served only the latest year, because a plain
  series would draw a trend line straight through gaps that are absences of
  publication, not of a cut-off. That reasoning is answered rather than
  abandoned: cutoffYearRows classifies every year in the span, and the chart
  breaks the line rather than interpolating across it.

  A missing year is not one fact but three. It may be unpublished; it may be
  a year the school was not oversubscribed; or there may be no record at all.
  Collapsing them into "no data" throws away the reassuring case and hides
  the important caveat, so each is stated in words in the table.

  The claim is held to what the data supports. fact_admissions.oversubscribed
  compares FIRST PREFERENCES against places, which does not establish that
  every applicant was offered one — so the copy says "places available on
  first preferences" and a test asserts the stronger claim never appears.

The trend summary is not a verdict
  It names both endpoints and their years and lets the reader conclude. It is
  withheld below four published points, and a swing under a tenth of the
  earlier figure is reported as "broadly the same" rather than dressed up as
  a direction.

The postcode check
  This is the only place on the site that answers a question about a family
  rather than a school, so most of the care went into what it refuses to say.
  postcodes.io returns a centroid covering roughly fifteen addresses, which
  against a 500 m cut-off is a fifth of the whole distance — so a margin
  inside 100 m returns "too close to call" rather than a place a family does
  not have. Unpublished years count as unknown, never as a pass. The limits
  are stated before the check is used, not revealed with the answer.

  The postcode is geocoded in the browser and never stored.

Both templates
  Banded and selective secondaries are exactly where this matters most, so
  the detail is shared. The primary page gives it a third tab; the secondary
  page is one flat panel by design and renders it inline.

Absence is explained rather than reported. A selective school's missing
figure is explained by how it admits; a consistently undersubscribed school
reads as good news.

Also makes the batch loader's test double honour ORDER BY. It was a no-op,
so "latest row per URN" was really "first row in the fixture" and the test
would have passed with the sort reversed or removed.

Verified: 214 frontend tests, 54 backend, 45/53 e2e green against staging
(the 8 cut-off journeys skip until the DAG runs). Rendered offline against
the real compiled CSS in both themes and at 390px; every new surface clears
WCAG AA, measured on composited pixels.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WDvkyqqHABm4bmth2kjAxE
2026-08-16 13:42:06 +01:00
TudorandClaude Opus 5 88c653215d feat(admissions): show the last distance offered where councils publish it
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m3s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 31s
PR Checks / Build Frontend (no push) (pull_request) Successful in 44s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 1m12s
PR Checks / AI Code Review (Claude) (pull_request) Failing after 2m7s
Adds the cut-off distance a parent actually asks about — "how close do we
need to live?" — end to end: a Singer tap, dbt staging and mart models, an
Airflow DAG, and a tile on both detail templates. 3,597 schools across 57
local authorities carry a figure; the rest are unchanged.

There is no national source for this. Each LA publishes its own cut-offs in
its own format, and the collected CSV is transcribed from PDFs, spreadsheets
and web pages — so most of the work here is deciding what is safe to show.

Data
  * tap-uk-school-distance loads the CSV verbatim into raw. Keyed on
    (urn, year, school_name), because school_name carries the admission
    route: (urn, year) alone collides on 118 keys and a reload would have
    silently dropped every band but one.
  * stg_school_distance applies a 25 m – 25 km plausibility band. The source
    contains 0.0-mile rows (published where a school filled on a higher
    criterion), 1-metre cut-offs, and one reading 533 miles — ~4% of rows,
    all of which would put a visibly wrong number on a live page.
  * fact_admission_distance collapses routes to one row per school per year
    using the furthest, and keeps route_count so the page can say the figure
    is the widest of several bands rather than the one for a given child.

Serving
  * Kept out of fact_admissions: that mart is EES-derived and near-complete
    for England, this one covers 57 LAs, and the two refresh independently.
  * Latest year only. Coverage is ragged — a school may have 2021 and 2026
    and nothing between — so a history array would invite a trend line drawn
    through gaps that are absences of publication, not of a cut-off.
  * The Admissions section now renders on either source. 3% of the schools
    that render have a cut-off and no EES admissions row, and gating on
    admissions alone would have hidden the figure on those pages.

Interface
  * The year travels with the figure everywhere it appears; a cut-off
    detached from its admissions round is not a fact about anything.
  * "Not a fixed catchment — it moves every year" sits under every instance,
    because that is the inference a parent will otherwise draw.
  * Replaces a hardcoded "Historical distance cut-off data is not available
    for this school" that appeared on every secondary page, including the
    ones whose council does publish it. The absence is now stated only when
    it is real, and names the authority that would hold it.

The tint costs the muted tokens their AA margin: measured on the composited
backdrop (not the computed one, which reports the untinted card), --text-muted
falls to 4.09:1 in dark theme. The tile uses --text-secondary instead — 6.50:1
dark, 6.60:1 light.

The DAG is manual, like the other annual ones: councils publish on allocation
day, each on its own timetable, so there is no date worth scheduling against.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WDvkyqqHABm4bmth2kjAxE
2026-08-15 22:48:30 +01:00
TudorandClaude Opus 4.8 a102508ef1 fix(data): strip any table alias in missing-column matcher, not just s.
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m1s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 42s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 9s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m49s
The graceful-degradation fallback keys off the column named in a Postgres
UndefinedColumn error, but the matcher only stripped an `s.` alias. The two
new dim_location columns (county, parliamentary_constituency) are selected
via the `l.` alias and Postgres reports them unquoted as
"column l.county does not exist" — which the old regex failed to match at
all, returning None.

If dim_school is rebuilt (telephone/nursery present) but dim_location is not
yet (county/parliamentary_constituency missing) — plausible since they are
independently-rebuilt dbt models — the fallback branch never matched and
load_school_data_as_dataframe() returned an empty DataFrame, showing zero
schools sitewide instead of degrading those columns to NULL.

Generalise the alias prefix to `\w+\.` and cover the l.-qualified case in
tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 09:58:27 +01:00
TudorandClaude Opus 4.8 0186227ced feat(detail): surface GIAS identity/contact details, drop unwired sections
PR Checks / Frontend Typecheck + Tests (pull_request) Canceled after 58s
PR Checks / Backend Smoke (pull_request) Canceled after 0s
PR Checks / Build Backend (no push) (pull_request) Canceled after 0s
PR Checks / Build Frontend (no push) (pull_request) Canceled after 0s
PR Checks / Build Pipeline (no push) (pull_request) Canceled after 0s
PR Checks / AI Code Review (Claude) (pull_request) Canceled after 0s
Add seven school-identity fields to the detail header (both primary and
secondary views): age range, religious character, nursery and sixth-form
indicators as chips; telephone (tel: link), county and parliamentary
constituency as header details. religious_denomination, age_range and
has_sixth_form were already served; telephone, nursery_provision, county
and parliamentary_constituency are newly wired through the marts query
(with a NULL fallback for un-rebuilt marts, mirroring has_sixth_form) and
the school_info API response.

Remove three UI sections the backend never populated (always null): Year 1
Phonics, the SEN "types of additional needs" breakdown, and the average
class-size card — along with their now-dead props, route plumbing, and the
SenDetail/Phonics types + class_size_avg field.

Extend the e2e detail journey to assert the Phonics section is gone and the
new header fields render when the record carries them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 09:23:08 +01:00
TudorandClaude Opus 4.8 8a9ba30cc2 fix(detail): compare each SATs bar to its own national benchmark
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m2s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 45s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 9s
The KS2 SATs chart drew a single national-average line spanning the
full height of each subject's chart area, positioned at the national
*expected* value. But the area stacks two bars — Expected and Exceeding
— and the higher-standard/greater-depth national is a very different,
much lower figure (e.g. reading higher standard ~29% vs expected ~75%).
So the line crossed the Exceeding bar at the wrong place, making every
school's exceeding result look far below national when it wasn't.

The per-subject higher-standard nationals were already computed in the
fact_ks2_national_averages mart; they just weren't serialized. Fix:

- backend: add reading_high_pct, writing_gd_pct (writing = greater
  depth) and maths_high_pct to the national-averages payload.
- SchoolDetailView: pass a nationalExceedingPct per subject, mapping
  writing to the greater-depth figure.
- SatsChart: replace the single full-height line with a national marker
  on each bar's own track (coral tick + "nat X%" in the bar header), so
  Expected and Exceeding each sit against the correct benchmark.

KS2 only; the secondary Attainment 8 chart already uses one line for
one measure and is untouched.

Verified: tsc --noEmit, next build, and backend pytest (national
averages marts, incl. a new test guarding the per-subject nationals).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-21 14:55:43 +01:00
TudorandClaude Fable 5 5ec4f3f7cd fix(list/map): badge report-card schools as Report Card, not their carried-forward grade
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m5s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 47s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m48s
The search-result cards and map pins keyed report-card detection off
ofsted_framework === 'ReportCard', but the API sets ofsted_framework to the
raw event grouping ('Schools - S5'); worse, ofsted_grade (the carried-forward
legacy grade) was checked first and won. So report-card schools were badged
by their old grade — Barclay's pin/card read 'Outstanding · 2021' instead of
'Report Card · 2026'. Same root cause as the detail-page fix, different
surface.

Expose ofsted_rc_date on the list serialization (the report-card inspection
date, non-null only for report cards) and make both badge builders treat a
present rc-date as winning over any grade, using its year. Removes the dead
framework === 'ReportCard' branches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-18 21:14:57 +01:00
TudorandClaude Fable 5 5944d88f0b fix(compare): expert sign-off must-fixes — phase-matched admissions, Ofsted sentinel codes, selective-school copy
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m4s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 46s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m53s
M1: admissions rounds are now selected by the active phase tab
(admissionsForPhase) — an all-through school's Year 7 round no longer
masquerades as Reception odds beside pure primaries; honest per-cell and
section fallbacks name the round (Reception / Year 7).
M2: Ofsted sentinel codes (9 = not applicable) never render as judgement
chips, and the sixth-form judgement — previously dropped — now renders
for schools that have one.
M3: 'What this means' is phase- and type-aware: selective schools get
entrance-test framing, secondary faith schools a faith-criteria note, and
the primaries' distance template never appears on the secondary tab
(admissions_policy now exposed in compare school_info).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-17 12:40:42 +01:00
TudorandClaude Fable 5 c9e324635b fix(data): official DfE KS4 national headline averages; drop mislabelled computed means
New ees_ks4_national stream ingests the EES 'National characteristics
summary data' series (England, state-funded, all pupils). The old mart's
unweighted school means were 7-15 points off every headline measure and
produced an impossible national Progress 8 (-0.27). The API's computed
fallback is gone too: the footnote calls these figures official, so an
unbuilt mart now yields an empty series, never a stand-in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-16 19:07:49 +01:00
TudorandClaude Fable 5 1d855f3c17 fix(compare): census-sourced FSM/EAL benchmarks; never fall back across measure definitions
The FSM chip anchored against disadvantaged_pct (a different measure,
FSM6+CLA) whenever fsm_pct was null — which it always was, since the
performance df has no fsm_pct. New fact_census_benchmarks mart supplies
pupil-weighted FSM/EAL means per phase; the KS2-column medians that
produced a bogus 50% 'secondary disadvantaged' anchor are gone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-16 19:05:15 +01:00
TudorandClaude Fable 5 9773483221 fix(compare): date report cards with their own inspection date, never the legacy one
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-16 14:48:17 +01:00
Tudor b4b0249a06 Fix Ofsted transitional inspections, phase tab exclusions, and FSM benchmark comparison
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m9s
PR Checks / Backend Smoke (pull_request) Successful in 8s
PR Checks / Build Backend (no push) (pull_request) Successful in 19s
PR Checks / Build Frontend (no push) (pull_request) Successful in 47s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Failing after 10s
2026-07-15 17:23:40 +01:00
TudorandClaude Fable 5 315f1feede perf(api): batch supplementary queries — one per table, not five per school
PR Checks / Backend Smoke (pull_request) Canceled after 0s
PR Checks / Build Backend (no push) (pull_request) Canceled after 0s
PR Checks / Build Frontend (no push) (pull_request) Canceled after 0s
PR Checks / Build Pipeline (no push) (pull_request) Canceled after 0s
PR Checks / AI Code Review (Claude) (pull_request) Canceled after 0s
PR Checks / Frontend Typecheck + Tests (pull_request) Canceled after 8m16s
get_supplementary_data ran ~5 sequential DB round-trips per URN, so
/api/compare scaled at ~37ms/school (measured on staging: 1 school 155ms,
3 schools 220ms, 6 schools 340ms). get_supplementary_data_batch fetches
each table once with WHERE urn IN (...) and groups in Python, collapsing
5*N round-trips to a constant 5. get_supplementary_data is now a thin
wrapper so the detail endpoint is unchanged; the compare endpoint makes
one batched call. Each table degrades independently on failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-14 22:42:10 +01:00
TudorandClaude Fable 5 52f8994401 perf(api): persist KS4 national averages as a mart; stop per-request aggregation
fact_ks4_national_averages is computed once at dbt build time (covered by
the EES DAG's stg_ees_ks4+ selector). _national_averages_payload now reads
both national-averages marts instead of scanning the performance dataframe
per year on every /api/compare request (~250ms saved per call). Fallback
for the deploy-before-DAG window computes the latest year only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-14 13:05:03 +01:00
TudorandClaude Fable 5 d0e71e2cf0 feat(api): compare school_info carries GIAS facts for the community section
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m36s
PR Checks / Backend Smoke (pull_request) Successful in 6s
PR Checks / Build Backend (no push) (pull_request) Successful in 16s
PR Checks / Build Frontend (no push) (pull_request) Successful in 48s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Failing after 11s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-14 00:04:03 +01:00
TudorandClaude Fable 5 c0f31a5941 feat(api): compare endpoint carries supplementary blocks, national averages and benchmarks
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m45s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 22s
PR Checks / Build Frontend (no push) (pull_request) Successful in 51s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 37s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 3m12s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-13 18:48:28 +01:00
TudorandClaude Fable 5 cec7941b44 feat(api): computed state-school benchmarks
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-13 18:45:30 +01:00
TudorandClaude Fable 5 dbaa15c099 feat(api): expose progress CIs, KS4 banding/gaps, admissions detail, report-card labels
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-13 18:44:29 +01:00
TudorandClaude Fable 5 b5b47ca135 feat(api): Ofsted report-card labels and provider-page URL
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-13 18:42:08 +01:00
TudorandClaude Fable 5 0c89b2c34e feat(api): map compare-foundation mart columns
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-13 18:40:55 +01:00
tudor 58e90fef61 Merge pull request 'fix(api): blank-name GIAS sentinel codes map to empty string, not Unknown(n)' (#27) from fix/gias-blank-name-codes into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 20s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 54s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m5s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Successful in 41s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Successful in 9s
Reviewed-on: #27
2026-07-09 21:21:08 +00:00
TudorandClaude Fable 5 3710529e49 fix(api): map blank-name GIAS sentinel codes to empty string, not Unknown
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m40s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 20s
PR Checks / Build Frontend (no push) (pull_request) Successful in 54s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 37s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m35s
ReligiousCharacter 99 (~4k schools) and AdmissionsPolicy 9 (~5.6k) carry a
code with a blank name in the GIAS CSV; the generator skipped them so they
hit the Unknown(<code>) path — wrongly triggering the Faith-priority tag
and polluting filters. Blank-only codes now map to "" (byte-identical to
the old name pipeline); accepted_values lists extended to match the seed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 22:04:53 +01:00
TudorandClaude Fable 5 74ca76d150 fix(api): match missing-column fallbacks on the DBAPI error, not the statement
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m41s
PR Checks / Backend Smoke (pull_request) Successful in 6s
PR Checks / Build Backend (no push) (pull_request) Successful in 20s
PR Checks / Build Frontend (no push) (pull_request) Successful in 53s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m41s
str(ProgrammingError) embeds the full SQL, which contains every column
name — the substring check matched any error and could take the wrong
retry branch. Parse the missing column from exc.orig instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 19:29:58 +01:00
TudorandClaude Fable 5 4b75152ee0 fix(api): fall back to legacy name-column query when marts predate code migration
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m41s
PR Checks / Backend Smoke (pull_request) Successful in 6s
PR Checks / Build Backend (no push) (pull_request) Successful in 19s
PR Checks / Build Frontend (no push) (pull_request) Successful in 47s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Failing after 3m8s
Closes the deploy window flagged by CI review — the backend now works
against both the old (name) and new (code) mart schemas.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 14:48:06 +01:00
TudorandClaude Fable 5 f1a013ec01 feat(api): translate GIAS codes to names at the query boundary
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 10:48:53 +01:00
TudorandClaude Fable 5 e188c2ff4b feat: GIAS code->name dictionaries generated from live bulk CSV
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 10:38:19 +01:00
TudorandClaude Fable 5 6f602f4a9e feat(api): expose GIAS establishment status on school payloads
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 22:05:55 +01:00
TudorandClaude Fable 5 4d226fd616 test: drop unused fake exception helper
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m41s
PR Checks / Backend Smoke (pull_request) Successful in 6s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 47s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 53s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 3m17s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:36:01 +01:00
TudorandClaude Fable 5 a524cdc591 fix(api): survive missing has_sixth_form column and numpy bool serialization
- data_loader.load_school_data_as_dataframe now catches a ProgrammingError
  whose message mentions has_sixth_form (psycopg2 UndefinedColumn) and
  retries with a NULL-AS-has_sixth_form query variant, so the API keeps
  serving data (and the app.py column-fallback branch stays reachable)
  even before the nightly pipeline has rebuilt marts.dim_school.
- utils.convert_to_native now handles numpy.bool_ so GET /api/schools/{urn}
  doesn't 500 once has_sixth_form is a populated bool-dtype column.
- Update the now-stale comment on the app.py age-range fallback branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:33:25 +01:00
TudorandClaude Fable 5 1d149ffc48 feat(api): drive has_sixth_form filter and payloads from GIAS flag
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 10:36:39 +01:00
TudorandClaude Fable 5 d11faefebd feat(pipeline): derive dim_school.has_sixth_form from GIAS flag
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 10:30:39 +01:00
TudorandClaude Opus 4.8 536832a524 chore: drop committed .pyc files, ignore __pycache__ everywhere
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m37s
PR Checks / Backend Smoke (pull_request) Successful in 6s
PR Checks / Build Backend (no push) (pull_request) Successful in 29s
PR Checks / Build Frontend (no push) (pull_request) Successful in 42s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 1m54s
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 09:37:17 +01:00
TudorandClaude Opus 4.8 87642b7b06 fix(api): serialize schools that have no performance rows
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m39s
PR Checks / Backend Smoke (pull_request) Successful in 8s
PR Checks / Build Backend (no push) (pull_request) Successful in 30s
PR Checks / Build Frontend (no push) (pull_request) Successful in 42s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m0s
Schools without KS2/KS4 results (special post-16 institutions, sixth-form
centres, PRUs, new schools) come back from the marts LEFT JOIN with NaN in
every numeric column. school_info passed those raw pandas values straight
into JSONResponse, which renders with allow_nan=False, so the detail
endpoint 500d and the frontend turned that into a 404 on every such SEO
landing page.

Run school_info values through convert_to_native (the same treatment
yearly_data already gets), add backend unit tests plus a pytest step in PR
checks, and an e2e journey that finds a results-less school via the search
API and asserts its page renders.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 09:22:54 +01:00
TudorandClaude Fable 5 95081d38bd chore: remove the Ofsted Parent View feature end to end
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m39s
PR Checks / Backend Smoke (pull_request) Successful in 5s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 41s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 45s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 1m35s
Removes the 'What Parents Say' section and all supporting elements:

Frontend:
- Drop the OfstedParentView type, the parent_view field, the survey
  section and the 'X% would recommend' callouts in the primary and
  secondary detail views, the Parents nav item, and the parent-view CSS.

Backend:
- Remove the FactParentView model, its loading in data_loader, and
  parent_view from the school-details API response.
- Bump SCHEMA_VERSION to 6 and add an idempotent drop step
  (DROP TABLE IF EXISTS marts.fact_parent_view) to the CLI migration;
  add scripts/sql/drop_fact_parent_view.sql to apply directly to the
  dbt-owned marts DBs on staging and prod.

Pipeline:
- Delete the stg_parent_view + fact_parent_view dbt models and their
  source/schema entries, the tap-uk-parent-view Meltano extractor, and
  the monthly Parent View DAG; drop it from the Dockerfile and the
  staging bootstrap docs.

The rest of dbt (which builds every mart the app reads) is untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 09:01:26 +01:00
TudorandClaude Fable 5 6c872ce726 fix(api): accept academic-year codes in rankings year filter
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m36s
PR Checks / Backend Smoke (pull_request) Successful in 5s
PR Checks / Build Backend (no push) (pull_request) Successful in 20s
PR Checks / Build Frontend (no push) (pull_request) Successful in 52s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 1m44s
The rankings endpoint validated year with le=2100, but the database
stores academic-year codes like 201819, so any explicit year selection
returned a 422 and the rankings page rendered its empty state. Widen
the bound to cover the codes and extend the e2e journey to pick a
specific year and assert the table stays populated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 22:13:56 +01:00
TudorandClaude Opus 4.8 9e4cf9dfbf fix(ofsted): apply ungraded fallback on school detail page too
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 18s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 52s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 18s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The detail page reads overall_effectiveness from the API and showed
"Not rated" for ungraded-only schools, even though the list badge uses
the coalesced grade. Coalesce the ungraded fallback into the API's
overall_effectiveness so the detail page shows the same grade.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 22:11:45 +01:00
TudorandClaude Opus 4.8 2332ee6347 feat(ofsted): fall back to ungraded inspection outcome for school grade
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 18s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 48s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Ungraded (Section 8) inspections don't assign a fresh grade — the export
only gives free text like "School remains Good". Parse that text into a
grade (remains Outstanding -> 1, remains Good -> 2, else null) and use it
as a last-resort fallback when no graded overall effectiveness exists.

Also retain schools that have only an ungraded inspection (no graded date)
by coalescing the inspection date, so ~8.5k previously-dropped schools now
carry a grade.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 21:44:01 +01:00
TudorandClaude Opus 4.8 4de7e559e9 feat(admissions): surface multi-year admissions trend on school detail
Build and Push Docker Images / Build Backend (FastAPI) (pull_request) Successful in 22s
Build and Push Docker Images / Build Frontend (Next.js) (pull_request) Successful in 53s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (pull_request) Successful in 11s
Build and Push Docker Images / Trigger Portainer Update (pull_request) Has been skipped
The school detail page only showed the latest admissions year. We store
every year, which is more decision-relevant for parents (the trend and its
consistency matter more than a single noisy year).

Backend now returns the full admissions_history (oldest first) alongside the
existing latest-year object. The primary SchoolDetailView gains a header
toggle ("This year | N-year trend") that swaps the Q&A for an SVG sparkline
of the first-choice offer rate. The toggle only appears when >=2 years carry
an offer rate; otherwise it falls back to the single-year card. Both views
share one CSS-grid cell so switching causes no layout shift.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-19 18:41:03 +01:00
Tudor SitaruandClaude Opus 4.7 62eeee5f7c perf: cache aggressively and trim client bundle
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 1m1s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 53s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 2m4s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Frontend
- Dynamic-import Chart.js components on detail/compare views so Chart.js
  no longer ships in initial JS.
- Drop force-dynamic on home, compare, rankings so internal data fetches
  reuse Next.js's per-call revalidate cache.
- Switch /school/[slug] to ISR with a 7-day revalidate window (school
  data updates annually).
- Preconnect to analytics + postcodes.io; remove redundant defer on the
  Umami Script tag (afterInteractive already covers it).
- Bump images.minimumCacheTTL to 1 year.
- Extract HowItWorks and Editorial sections as server components passed
  to HomeView via slot props so their JSX stays out of the client bundle.

Backend
- Add GZipMiddleware (min 512 bytes).
- Add CacheAndETagMiddleware: per-path Cache-Control with long s-maxage
  + stale-while-revalidate, ETag generation, and 304 on If-None-Match.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-02 13:46:45 +01:00
Tudor SitaruandClaude Opus 4.7 38d033f6a9 fix(rankings): expose selected metric under stable value key
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 22s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 55s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
The /api/rankings endpoint returned each row keyed by the metric's
column name (e.g. rwm_high_pct) but never under a generic `value`
field. The frontend RankingItem type and RankingsView both read
ranking.value, so every row rendered "—" for every metric — the
default rwm_expected_pct included.

Add `df["value"] = df[metric]` before JSON serialisation so the
frontend gets the value it has always expected. The raw metric
column is still in the row for any caller that wants it explicitly.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-19 09:31:15 +01:00
Tudor SitaruandClaude Opus 4.7 675601869b feat(detail): show pupil gender split on school detail pages
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 19s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 46s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Upgrades the existing "Pupils" stat to include a compact split bar and
percentage hint for mixed schools (single-sex schools already carry a
"Boys's/Girls's school" badge, so the split would be redundant).

Wires fact_pupil_characteristics into the API: new SQLAlchemy model and
a real census block in /api/schools/{urn} replacing the prior null stub.

On the primary detail page the inline "Pupils: 241" text is replaced by
a richer block (display number + bar + "52% girls · 48% boys"). On the
secondary detail page the existing "Total pupils" hero stat card grows
the bar and hint beneath the number. Both fall back to the previous
text-only rendering when census gender data is missing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-17 22:36:33 +01:00
Tudor SitaruandClaude Sonnet 4.6 f05bbba613 perf: resolve all P1–P5 performance issues from code review
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 21s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 50s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
P1 (backend/data_loader.py): Add load_latest_school_data() which pre-computes
the one-row-per-school latest-year snapshot (groupby, prev-year trend merge)
once at startup instead of on every /api/schools request. get_schools route
now starts from the cached snapshot rather than rebuilding it.

S3 (backend/app.py): Wrap synchronous geocode_single_postcode() call in
asyncio.to_thread() so postcode lookups no longer block the uvicorn event
loop. Admin reload endpoint also uses to_thread for both cache primes.

P2 (nextjs-app/components/HomeView.tsx): Add mapParamsRef guard so switching
back to map view does not re-fetch 500 schools when search params haven't
changed. Reset ref on new searches so fresh data is always fetched.

P3 (nextjs-app/lib/chartSetup.ts): Extract Chart.js registration into a
shared side-effect module. ComparisonChart and PerformanceChart now import
it instead of each calling ChartJS.register() independently.

P4 (backend/database.py): Remove unnecessary db.commit() from the read-only
get_db_session() context manager — saves a DB round-trip on every request.

P5 (backend/database.py): Add pool_recycle=1800 to SQLAlchemy engine to
prevent stale TCP connections from accumulating in long-running processes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-15 22:45:46 +01:00
Tudor SitaruandClaude Opus 4.6 6d685b7e8a refactor(admissions): rename published_admission_number to places_offered
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 18s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 46s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Failing after 13s
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
The staging model aliased EES's total_number_places_offered column as
published_admission_number, but PAN is the school's published capacity
(not exposed by EES at school level) — what we actually have is the
count of places offered in a given admissions round. The misnomer
propagated to the mart, SQLAlchemy model, API response, TS types, and
UI copy ("places per year", "(PAN)").

Rename end-to-end and fix the UI labels:
  - "29 places for 42 first-choice applications"
      → "29 places offered for 42 first-choice applications"
  - "Reception/Year 7 places per year"
      → "Reception/Year 7 places offered"
  - drop the misleading "(PAN)" suffix in the secondary view

Also add a comment in stg_ees_admissions clarifying this is the number
of places offered, not PAN. Requires dbt to rebuild fact_admissions
(marts are materialized as tables) before the backend can start.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-14 09:45:43 +01:00