int_ks4_with_lineage references stg_legacy_ks4 but the model was never
selected for build, causing a missing relation error.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LegacyKS2Stream now auto-detects ZIP vs bare CSV — if the download is a ZIP
it extracts england_ks2final.csv; if it's a plain CSV file it reads directly.
This keeps backwards compatibility while allowing both streams to share the
same DfE annual archive URLs.
legacy_ks2_urls updated to point at the same 4 ZIPs as legacy_ks4_urls so
only one set of archives needs to be maintained going forward.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors the existing legacy KS2 pattern to fill the gap before EES hosted
KS4 data. Four files changed:
- tap-uk-ees: LegacyKS4Stream downloads each year's DfE Compare School
Performance ZIP, extracts england_ks4final.csv, maps 416 legacy columns
to Singer fields, strips % suffixes. Registered in discover_streams().
TapUKEES.config_jsonschema gains legacy_ks4_urls setting.
- stg_legacy_ks4.sql: safe_numeric casts + NULL placeholders for columns
not present in legacy format (ebacc_avg_score, gcse_grade_91_pct,
prior_attainment_avg, sen_pct).
- int_ks4_with_lineage.sql: adds all_ks4 CTE unioning stg_ees_ks4 and
stg_legacy_ks4, matching the int_ks2_with_lineage pattern.
- _stg_sources.yml + meltano.yml: source declaration and setting definition
for legacy_ks4. URLs configured per-year once provided.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs prevented historical secondary school data from loading:
1. stg_ees_ks4.sql filtered breakdown_topic = 'Total' only, but EES
releases prior to 2023/24 use breakdown_topic = 'All pupils' (matching
the KS2 convention). All older years were silently dropped to zero rows.
Fix: accept both values with an IN clause.
2. get_all_releases() in tap-uk-ees fetched only the first page of the
EES releases API. Now follows all pages via the paging.totalPages field
so no historical release is missed when more than 20 exist.
After re-running the annual EES pipeline, secondary school comparison
charts should show data across all available years (2018/19 onwards).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Once the user explicitly clicks a phase tab, suppress auto-phase detection
so switching to Secondary (or Primary) can't be snapped back by the effect
that fires when comparisonData re-fetches.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The useEffect that auto-selects the Primary/Secondary tab was calling
setComparePhase() without updating selectedMetric. When a URL carries a
primary metric (e.g. metric=rwm_expected_pct) but the shortlisted schools
are secondary, the tab would switch to Secondary while the metric stayed
at rwm_expected_pct — which is null for all secondary schools, causing
every card to show "–" and requiring a manual tab toggle to fix.
Fix: after determining the phase from school data, check whether the
current metric belongs to that phase's category list. If not, reset to
the phase default (attainment_8_score for secondary, rwm_expected_pct
for primary). A metric that already fits the phase (e.g. the URL already
carried attainment_8_score) is preserved unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
P1 (backend/data_loader.py): Add load_latest_school_data() which pre-computes
the one-row-per-school latest-year snapshot (groupby, prev-year trend merge)
once at startup instead of on every /api/schools request. get_schools route
now starts from the cached snapshot rather than rebuilding it.
S3 (backend/app.py): Wrap synchronous geocode_single_postcode() call in
asyncio.to_thread() so postcode lookups no longer block the uvicorn event
loop. Admin reload endpoint also uses to_thread for both cache primes.
P2 (nextjs-app/components/HomeView.tsx): Add mapParamsRef guard so switching
back to map view does not re-fetch 500 schools when search params haven't
changed. Reset ref on new searches so fresh data is always fetched.
P3 (nextjs-app/lib/chartSetup.ts): Extract Chart.js registration into a
shared side-effect module. ComparisonChart and PerformanceChart now import
it instead of each calling ChartJS.register() independently.
P4 (backend/database.py): Remove unnecessary db.commit() from the read-only
get_db_session() context manager — saves a DB round-trip on every request.
P5 (backend/database.py): Add pool_recycle=1800 to SQLAlchemy engine to
prevent stale TCP connections from accumulating in long-running processes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New /admissions route with AdmissionsView client component
- Live countdowns (days until) to Primary/Secondary deadlines and Offer Days
- Step-by-step timelines for both tracks with highlighted milestone rows
- Tips section covering equal preference rule, late applications, waiting lists
- Homepage countdown strip (4 cards) between discovery chips and how-it-works
- Admissions nav link and footer link added
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GCSE top metrics (Att8, P8, Eng+Maths 4+/5+) promoted to heroStatCard
teal-tinted cards with Playfair serif values and DeltaChip vs national
- Attainment 8 visual bar: 0–80 scale with coral national-average marker
and pill label, mirrors the SatsChart concept for a score metric
- Progress 8 number line: −3 to +3 axis showing CI band, zero baseline,
and a teal/coral dot for the school's score (hidden when P8 suspended)
- SEN section upgraded from plain metricCard to heroStatCard grid
- History table moved into a details/summary accordion (collapsed by
default); PerformanceChart now lives at the top of the History section
always visible above the disclosure
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Hero: Playfair heading with coral italic accent, teal eyebrow pill,
richer sub-copy describing both primary and secondary coverage
- Discovery: geolocation "Schools near me" button (reverse-geocodes via
postcodes.io → /?postcode=…&radius=1), plus Start exploring chips
linking to /rankings and /compare
- How it works: 3-card grid showing miniature real-UI previews for
Performance (primary SATs cascade + secondary Att8 bar), Ofsted
inspection card, and side-by-side Compare table
- Editorial: text column + factbox (totalSchools, LA count, coverage
dates) rendered inside a white card below the how-it-works section
- Footer: expanded to 3 columns (brand blurb, Product, Resources);
links updated to / /rankings /compare and real gov.uk/ofsted URLs
- All new sections visible only on landing (no search active)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Renames the RWM-specific .rwmHero* classes to generic .heroStat*
and reuses them on the three Pupils & Inclusion headline cards
(disadvantaged, EAL, SEN support). Grid switches to auto-fit to
accommodate both 2- and 3-card layouts. SEN primary-need breakdown
stays on the dense .metricCard style.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Top two combined cards now use a teal-tinted background with a
2.1rem Playfair serif teal value and a compact uppercase label —
consistent with the hero treatment shown in the design mockup.
Scoped via new .rwmHero* classes so other metric cards on the page
keep their existing style.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bars drop from 22px to 14px with softer 4px radius, percentages render
in Playfair serif to match the rest of the detail page typography,
national-average line opacity bumped from 25% to 35% for legibility.
Subject name and ruler proportions also aligned with the mockup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Explains the intersection semantics of RWM combined — a pupil is only
counted if they met the bar in all three subjects — with a math line
showing the per-subject percentages collapsing to the combined figure.
Only renders when all four values are present; national-average pill
markers on the cascade are untouched.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Oversubscribed badge + "Demand exceeds capacity" text lived at
the bottom of the tile as an afterthought. It's the headline finding —
a parent should read it first, then let the Q&A supply the detail.
Replace the footer badge with a Playfair Display sentence directly
under the section title: "This school is oversubscribed." (state word
coloured coral for oversubscribed, teal for undersubscribed). The
"Demand exceeds capacity" / "Supply meets demand" line sits below as
quiet muted text — present for context but not competing with the
headline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The "How Hard to Get Into This School" tile mixed a progress bar
(places vs first-choice) with three text metric cards, making the
data feel fragmented and hiding the real narrative. The progress bar
also broke visually when undersubscribed and didn't scale to different
school sizes.
Replace with a typographic Q&A list that answers the questions
parents actually ask — "How many places were offered?", "How many
families wanted this school first?", "How many got their first
choice?", "How many applied in total?" — with a verdict footer
(Oversubscribed / Not oversubscribed + one-sentence explanation).
The third row now uses first_preference_offers (already in the API
response) to show "27 of 42 (64.3%)" instead of just the percentage,
giving the raw count parents actually want.
Each row is independently null-gated; rows stack vertically under
480px so the Playfair numeral stays legible.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The staging model aliased EES's total_number_places_offered column as
published_admission_number, but PAN is the school's published capacity
(not exposed by EES at school level) — what we actually have is the
count of places offered in a given admissions round. The misnomer
propagated to the mart, SQLAlchemy model, API response, TS types, and
UI copy ("places per year", "(PAN)").
Rename end-to-end and fix the UI labels:
- "29 places for 42 first-choice applications"
→ "29 places offered for 42 first-choice applications"
- "Reception/Year 7 places per year"
→ "Reception/Year 7 places offered"
- drop the misleading "(PAN)" suffix in the secondary view
Also add a comment in stg_ees_admissions clarifying this is the number
of places offered, not PAN. Requires dbt to rebuild fact_admissions
(marts are materialized as tables) before the backend can start.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Labels were sharing a flex row with the bar, so the bar's width %
was computed against the flex remainder after the label rather than
the full chart area. A 96% bar rendered around the 75% ruler mark,
and mobile was worse. Move labels to a header row above each bar and
give the bar a full-width track, so X% now aligns with X% on the ruler.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Redesign the School Details page for better parent comprehension:
- New SatsChart component: horizontal cascade bars with ruler scale and
national average marker (teal/coral palette matching site theme)
- Admissions section: visual progress bar showing 1st-preference demand
vs available places, colour-coded by oversubscription status
- Historical data: collapse raw year-by-year table behind a disclosure
element while keeping the performance line chart always visible
- EAL metric: add national average comparison via DeltaChip (backend now
includes eal_pct in national averages endpoint)
- New formatWithSuppression utility for null/suppressed data handling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dim_school.sql was checking for int_ofsted_latest in target.schema (wrong schema)
due to the custom generate_schema_name macro using literal schema names. The
model lives in 'intermediate', so ofsted_grade/date/framework were always NULL
in dim_school, causing all list cards to show 'Not yet inspected'.
Fix 1: data_loader.py joins marts.fact_ofsted_inspection with DISTINCT ON to
get latest inspection per school — no pipeline re-run needed.
Fix 2: dim_school.sql uses schema='intermediate' so future dbt runs correctly
denormalise the Ofsted summary into dim_school.
Replaces the bare Leaflet popup with a mini card showing school name,
3-state Ofsted badge (OEIF grade / ReportCard / pending), phase tag,
headline metric (Att8 for secondary, RWM% for primary) with delta vs
LA/national average, and a styled View Details button. Threads
nationalAvgRwm and laAverages from HomeView → SchoolMap → LeafletMapInner.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add nationalAvgRwm state fetched from /api/national-averages on mount
- Pass nationalAvgRwm to SchoolRow (vs-national delta now active in list view)
- Pass nationalAvgRwm to SchoolMap (prop accepted, threaded to Task 7)
- Redesign CompactSchoolItem: Ofsted badge + single headline metric + delta
- Fix stray backslash in SchoolRow.module.css .vsNationalFlat selector
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add ofsted_framework field to School type
- Add OfstedListBadge interface and buildOfstedListBadge pure function to utils.ts
- Add fetchNationalAverages API function that calls GET /api/national-averages
- Add test suite for buildOfstedListBadge (all 6 new tests pass)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The school_info object was missing total_pupils entirely, so the frontend
always fell back to the KS4 exam cohort from yearly_data. Now selects
s.total_pupils (GIAS NumberOfPupils — full school roll) as gias_total_pupils
in the main query and exposes it as total_pupils on school_info.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
latestResults.total_pupils in KS4 data is the Year 11 exam cohort (~1/7
of the school), not the full school roll. Prefer schoolInfo.total_pupils
(sourced from GIAS NumberOfPupils) which is the statutory census headcount.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
meltano run does not support --select; the full tap-uk-ees run already
includes EESKs2NationalStream so no separate task is needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces computed means from our school dataset with the published DfE
national headline figures for the KS2 chart reference line.
- tap-uk-ees: new EESKs2NationalStream fetches the stable EES data-catalogue
CSV (one row per year, England national total, AllSchools filter)
- dbt staging: stg_ees_ks2_national normalises columns, casts to float,
filters to years >= 201617
- dbt mart: fact_ks2_national_averages — one row per year, official figures
- backend/models: Ks2NationalAverage SQLAlchemy model
- backend/app: /api/national-averages queries the mart for KS2 by_year;
secondary by_year stays computed (no DfE KS4 national dataset yet)
- DAG: extract_ks2_national task added to school_data_annual_ees,
runs in parallel with the main EES extract
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the dashed reference line was a flat horizontal at the latest
year's national average across all historical data, implying the national
figure was constant. Now the backend returns per-year averages in `by_year`
and the chart maps each data year to its own national average, so the
reference line correctly reflects how the national picture changed over time
(including COVID recovery dip/recovery).
- backend: /api/national-averages now includes `by_year` list alongside
existing `year`/`primary`/`secondary` latest-year snapshot
- types: NationalAverages extended with `by_year: NationalAveragesYear[]`
- PerformanceChart: accepts `nationalByYear` prop; builds per-year series
aligned to school data years, falling back to scalar prop if absent
- SchoolDetailView + SecondarySchoolDetailView: pass `nationalAvg.by_year`
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Bump school name to clamp(2rem,5vw,3.25rem) Playfair Display
- Add hero signal chips strip (framework-aware Ofsted + coral Oversubscribed)
- Add at-a-glance stats row: Att8 with delta vs national, Ofsted serif tile, first-choice rate
- Active section highlighting in sticky nav via IntersectionObserver
- Collapse OEIF Ofsted section to prose when all sub-grades match overall
- Pass nationalAtt8Avg reference line to PerformanceChart
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
TypeScript compile error: 'navItems' used before its declaration.
The IntersectionObserver useEffect referenced navItems in its dep array
but was placed above the navItems const declaration. Move it to just
after navItems is built.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Uses IntersectionObserver on each section element. Multiple thresholds
(0, 0.1, 0.25, 0.5, 0.75, 1.0) track the intersection ratio of every
section simultaneously; whichever has the highest visible ratio at any
moment becomes the active item. rootMargin offsets for the sticky nav
height so a section is only considered active once it's genuinely in
view beneath the bar.
Active link gets .sectionNavLinkActive — coral background + white text,
matching the phase tab active style used elsewhere in the product.
Observer is cleaned up on unmount.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Drop the auto-generated italic summary sentence from the hero — adds
little beyond what the chips and stats already convey.
- Oversubscribed hero chip: tone-gold → tone-coral so it reads as a
warning rather than a neutral highlight.
- Remove unused buildSchoolSummary import and .heroSummary CSS.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
History chart
- Strip the redundant school-name title (already the page heading)
- Default to 2 visible lines: Reading, Writing & Maths expected % (teal,
bold) + Exceeding (gold, lighter); progress score lines hidden by
default, togglable via legend
- Add dashed national average reference line for RWM (primary) or
Attainment 8 (secondary) so the school's trajectory is always in
context
- Add trend summary chip above the chart computed from the data
("↓ Peaked at 90% (2016/17), currently 70%")
- Add COVID footnote when 2019/20 and 2020/21 data is absent
Ofsted section
- Collapse the four identical "Outstanding / Outstanding / Outstanding /
Outstanding" boxes into a single prose line when all sub-grades match
the overall verdict; show individual cards only when grades differ
SATs sub-metrics
- DeltaChip vs national average on Expected level row for Reading,
Writing and Maths (national averages already in the API response)
Admissions
- Fix label: "Year 3 places per year" → "Reception places per year" for
primary schools
Pupils & Inclusion
- DeltaChip + national avg hint on Eligible for pupil premium and
Pupils receiving SEN support (both keys present in /api/national-averages)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The three at-a-glance stats were misaligned — the serif "Outstanding" tile
sat noticeably above the "70%" and "64%" numerals because the serif variant
used a smaller font. That pushed its label row ("INSPECTED NOVEMBER 2023")
up too, breaking the horizontal rhythm across the row.
- Give .heroStatNumber and .heroStatNumberSerif a shared min-height tied
to the largest clamp value, plus display: flex; align-items: flex-end.
Content bottom-aligns inside the box, so every stat's label sits at the
same Y regardless of how tall the actual glyph is.
- Bump the serif variant up slightly (1.75rem → 2.25rem clamp) so it
feels closer in weight to the numerals while still leaving room for
longer words like "Requires Improvement".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The staging model was mapping EES column ``proportion_1stprefs_v_totaloffers``
straight onto ``first_preference_offer_pct``. That raw column is not a
percentage — it is a ratio of first-preference applications to total offers
(an oversubscription indicator, >1 means oversubscribed), so OLQH rendered
as "1%" when the true first-choice success rate is 27/42 = 64%.
The frontend display code is not at fault and is not patched here —
data-quality issues must be fixed at the source.
- stg_ees_admissions: compute ``first_preference_offer_pct`` as
``100 * number_1st_preference_offers / times_put_as_1st_preference`` —
of families who listed this school first, the % that received an offer
(0–100). Guard against divide-by-zero.
- stg_ees_admissions: expose the legitimate EES ratio as the new column
``oversubscription_ratio`` (1st-preference applications per place) for
future use, clearly named.
- fact_admissions, FactAdmissions model, data_loader: propagate the new
``oversubscription_ratio`` column.
- SchoolAdmissions type: document both columns inline.
- buildSchoolSummary: reword the oversubscription clause so it reads
sensibly across the whole 0–100 range (no more "just 64%").
- Hero chip subtitle: clearer phrasing "X% of first-choice applicants
offered a place".
Requires a dbt run of stg_ees_admissions and fact_admissions on deploy
so the new column materialises.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- heroChip: swap ``min-width: 180px; flex: 0 1 auto`` for
``flex: 0 0 240px`` so every chip in the strip is the same width
regardless of content. Title gets nowrap + ellipsis as insurance
against accidental overflow.
- heroChipTitle / Sub / Detail: align line-heights (1.3 / 1.4 / 1.4)
so an OEIF chip (title + sub) and a Report Card chip
(title + sub + detail) sit on the same vertical rhythm.
- heroSummary: drop the 64ch max-width — the sentence should read at
the natural hero width.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Drop the "Above national average" chip — the DeltaChip under the 70% /
Attainment 8 number already carries the same signal, so the chip was
duplicative and added noise.
- At-a-glance stats: switch from grid(auto-fit) to flex with a fixed
3rem column gap so the numbers cluster at the start of the row rather
than spreading across the full width of the header card.
- Add to Compare button: larger padding, bumped font, soft shadow, and
self-align centre so it sits in balance with the bigger headline
rather than floating tiny in the top-right corner.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The API returns ``framework`` as the literal string "NULL" for older OEIF
inspections (it comes from the upstream ``event_type_grouping`` column),
not real null. The original render path checks ``=== 'ReportCard'`` and
correctly treats anything else as OEIF — but buildOfstedHeroChip inverted
that and treated anything not exactly equal to ``'OEIF'`` as Report Card,
so OLQH (inspected Nov 2023, Outstanding) was being labelled as a Report
Card school in the hero strip and the at-a-glance tile.
- Invert the helper: only branch into Report Card when framework is
explicitly ``'ReportCard'``; treat OEIF / null / "NULL" / anything else
as OEIF, and require ``overall_effectiveness`` to render the grade word.
- Replace the toneClass field (which reused .ofstedGrade{N} / .rcGrade{N}
badge classes and dragged in their backgrounds) with a clean tone enum
``teal | green | gold | coral | neutral``. The serif Ofsted heroStat
picked up the badge background and rendered as a green box around
"Report Card" — gone now.
- Hero chip backgrounds use color-mix() against the tone variable so all
five tones share one rule.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Elevates the primary school detail hero from a flat report header into a
scannable editorial block. Parents can read the headline signal in seconds.
- A1: bump .schoolName to clamp(2rem, 5vw, 3.25rem) Playfair.
- A2: framework-aware signal chip strip via new buildOfstedHeroChip() helper.
Branches on ofsted.framework so Report Card schools never show a fake
overall grade — they get "Ofsted Report Card" + inspection date +
Safeguarding: Met/Not met. OEIF schools keep the grade word.
- A3: oversized Playfair stats — Reading, Writing & Maths % (primary) or
Attainment 8 (secondary) with inline DeltaChip vs national, Ofsted
verdict with tone colouring, and first-choice offer rate.
- B1: italic serif one-sentence summary via buildSchoolSummary() helper,
also framework-aware so Report Card schools are described by framework,
not a synthetic grade.
- C1: new DeltaChip component reused in the two headline KS2 metric cards
(rwm_expected_pct, rwm_high_pct).
All copy uses "Reading, Writing & Maths" in full. Secondary detail view
untouched in this slice.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tudor Sitaru