From 0309b27c849912bce4e5b295ac7f2ddcb85ccb50 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 10:02:21 +0100 Subject: [PATCH 01/10] docs: exam results phase taxonomy and sixth-form separation spec Co-Authored-By: Claude Fable 5 --- .../2026-07-07-exam-phase-taxonomy-design.md | 212 ++++++++++++++++++ 1 file changed, 212 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md diff --git a/docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md b/docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md new file mode 100644 index 0000000..49832d9 --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md @@ -0,0 +1,212 @@ +# Exam Results Taxonomy — Phase Grouping and Sixth-Form Separation + +**Date:** 2026-07-07 +**Status:** Approved design (taxonomy/analysis only — no implementation in this doc's scope) + +## Purpose + +Classify every exam-result metric SchoolCompare displays today into four phase +groups — **Primary**, **Secondary**, **Sixth form**, **Other** — and define an +authoritative rule for separating schools that have a sixth form from those +that don't. This document is the reference for: + +1. How the UI should group results sections and rankings by phase. +2. The future KS5 (A-level) ingestion work — the Sixth form group lists the + concrete DfE metrics as placeholders with source columns. +3. Replacing the fragile `age_range contains "18"` heuristic with the GIAS + `OfficialSixthForm` flag. + +## 1. Grouping principle + +Metrics are grouped by **the key stage of the assessment**, not by the phase +of the school displaying them. An all-through school (4–18) shows metrics in +all three exam groups; a pure primary shows only the Primary group. + +| Group | Assessments | Key stage | Taken at age | Data status | +|---|---|---|---|---| +| **Primary** | KS2 SATs (reading, writing TA, maths, GPS, science TA) | KS2 | 10–11 (Year 6) | ✅ Live — `marts.fact_ks2_performance` | +| **Secondary** | GCSEs, Attainment 8 / Progress 8, EBacc | KS4 | 15–16 (Year 11) | ✅ Live — `marts.fact_ks4_performance` | +| **Sixth form** | A levels, applied general, tech levels | KS5 (16–18) | 17–18 (Year 12–13) | ⏳ Not ingested — placeholders in §4 | +| **Other** | Non-exam context displayed alongside results | n/a | n/a | ✅ Live — various marts | + +Not covered (not displayed today, candidates for future "Other"/Primary): +EYFS Good Level of Development, Year 1 Phonics check, Year 4 Multiplication +Tables Check, KS1 assessments (no longer published at school level by DfE). + +## 2. Metric-by-metric mapping (current site) + +Every key in `backend/schemas.py` `METRIC_DEFINITIONS` — the single source of +truth for what the site displays — mapped to its phase group. `category` is +the existing schema category; source columns are the DfE names used at +ingestion (legacy performance-tables CSV for KS2, EES for KS4). + +### Primary (KS2 SATs) + +| Metric key | Category | DfE source column | +|---|---|---| +| `rwm_expected_pct` | expected | `PTRWM_EXP` | +| `reading_expected_pct` | expected | `PTREAD_EXP` | +| `writing_expected_pct` | expected | `PTWRITTA_EXP` | +| `maths_expected_pct` | expected | `PTMAT_EXP` | +| `gps_expected_pct` | expected | `PTGPS_EXP` | +| `science_expected_pct` | expected | `PTSCITA_EXP` | +| `rwm_high_pct` | higher | `PTRWM_HIGH` | +| `reading_high_pct` | higher | `PTREAD_HIGH` | +| `writing_high_pct` | higher | `PTWRITTA_HIGH` | +| `maths_high_pct` | higher | `PTMAT_HIGH` | +| `gps_high_pct` | higher | `PTGPS_HIGH` | +| `reading_progress` | progress | `READPROG` | +| `writing_progress` | progress | `WRITPROG` | +| `maths_progress` | progress | `MATPROG` | +| `reading_avg_score` | average | `READ_AVERAGE` | +| `maths_avg_score` | average | `MAT_AVERAGE` | +| `gps_avg_score` | average | `GPS_AVERAGE` | +| `rwm_expected_boys_pct` | gender | `PTRWM_EXP_B` | +| `rwm_expected_girls_pct` | gender | `PTRWM_EXP_G` | +| `rwm_high_boys_pct` | gender | `PTRWM_HIGH_B` | +| `rwm_high_girls_pct` | gender | `PTRWM_HIGH_G` | +| `rwm_expected_disadvantaged_pct` | equity | `PTRWM_EXP_FSM6CLA1A` | +| `rwm_expected_non_disadvantaged_pct` | equity | `PTRWM_EXP_NotFSM6CLA1A` | +| `disadvantaged_gap` | equity | `DIFFN_RWM_EXP` | +| `reading_absence_pct` | absence | `PTREAD_AT` | +| `gps_absence_pct` | absence | `PTGPS_AT` | +| `maths_absence_pct` | absence | `PTMAT_AT` | +| `writing_absence_pct` | absence | `PTWRITTA_AD` | +| `science_absence_pct` | absence | `PTSCITA_AD` | +| `rwm_expected_3yr_pct` | trends | `PTRWM_EXP_3YR` | +| `reading_avg_3yr` | trends | `READ_AVERAGE_3YR` | +| `maths_avg_3yr` | trends | `MAT_AVERAGE_3YR` | + +The absence metrics measure absence *from KS2 tests*, so they belong to +Primary even though they are not attainment scores. National comparators for +this group come from `marts.fact_ks2_national_averages`. + +### Secondary (KS4 / GCSE) + +| Metric key | Category | EES source column | +|---|---|---| +| `attainment_8_score` | gcse | `attainment8_average` | +| `progress_8_score` | gcse | `progress8_average` | +| `english_maths_standard_pass_pct` | gcse | `engmath_94_percent` | +| `english_maths_strong_pass_pct` | gcse | `engmath_95_percent` | +| `ebacc_entry_pct` | gcse | `ebacc_entering_percent` | +| `ebacc_standard_pass_pct` | gcse | `ebacc_94_percent` | +| `ebacc_strong_pass_pct` | gcse | `ebacc_95_percent` | +| `ebacc_avg_score` | gcse | `ebacc_aps_average` | +| `gcse_grade_91_pct` | gcse | `gcse_91_percent` | + +Also stored in `marts.fact_ks4_performance` (and `fact_performance`) but not +yet in `METRIC_DEFINITIONS` — Secondary group members when surfaced: +`progress_8_lower_ci`, `progress_8_upper_ci`, `progress_8_english`, +`progress_8_maths`, `progress_8_ebacc`, `progress_8_open`, +`prior_attainment_avg` (KS2 baseline of the GCSE cohort), `sen_pct`. + +### Sixth form (KS5) + +No metrics today. The secondary school detail view renders a static note +("Post-16 destination data coming soon") when the school has a sixth form. +Placeholders for ingestion are specified in §4. + +### Other (non-exam context) + +Displayed alongside results but not tied to any assessment: + +| Metric key / surface | Category | Source | +|---|---|---| +| `disadvantaged_pct` | context | KS2 CSV `PTFSM6CLA1A` | +| `eal_pct` | context | KS2 CSV `PTEALGRP2` | +| `sen_support_pct` | context | KS2 CSV `PSENELK` (KS4 fallback `sen_no_ehcp_pupil_percent`) | +| `stability_pct` | context | KS2 CSV `PTMOBN` | +| Ofsted grades incl. `sixth_form_provision` / `rc_sixth_form` | — | `marts.fact_ofsted_inspection` | +| Admissions (offers, oversubscription) | — | `marts.fact_admissions` | +| Finance (per-pupil spend, cost shares) | — | `marts.fact_finance` | +| Deprivation (IDACI) | — | `marts.fact_deprivation` | +| Pupil characteristics (census) | — | `marts.fact_pupil_characteristics` | + +Note: the context metrics are cohort characteristics of the KS2 cohort at +source, but they are presented (and should stay presented) as school-level +context, so they group as Other, not Primary. + +## 3. Sixth-form separation + +### Definition (authoritative) + +> A school **has a sixth form** iff GIAS `OfficialSixthForm (name)` = +> `"Has a sixth form"` for its URN. + +GIAS values are `Has a sixth form`, `Does not have a sixth form`, and +`Not applicable` / blank. `Not applicable` (nurseries, primaries, PRUs) maps +to **false**. This field is the DfE's registry flag, updated continuously, +and is the only source that correctly classifies: + +- 16–19 sixth-form colleges and UTCs (age ranges like `14-19`, `16-19` that + the current substring heuristic misclassifies as *no* sixth form); +- schools whose statutory age range extends to 18 on paper but which have no + registered post-16 provision. + +### Pipeline change (future work) + +1. `stg_gias_establishments.sql`: add + `"OfficialSixthForm (name)" as official_sixth_form`. +2. `dim_school.sql` (+ `models.py` `DimSchool`, `_marts_schema.yml`): add + `has_sixth_form boolean` = `official_sixth_form = 'Has a sixth form'`. +3. Expose `has_sixth_form` on the school API payloads. + +### Current heuristic — audit of `age_range` ~ "18" sites + +All must migrate to the `has_sixth_form` flag once exposed: + +| Site | Current behaviour | +|---|---| +| `backend/app.py:419-422` | `/api/schools?has_sixth_form=yes\|no` filters on `age_range.str.contains("18")` | +| `nextjs-app/components/SecondarySchoolDetailView.tsx:101` | "Sixth form" badge + coming-soon note from `age_range?.includes('18')` | +| `nextjs-app/components/FilterBar.tsx:370-372` | Filter labels hard-code "(11-18)" / "(11-16)" — labels should drop the age-range parenthetical since sixth form ≠ age range | + +Fallback rule: if GIAS is blank for a URN (rare; new establishments), fall +back to the age-range heuristic and log the URN. + +### UI separation rules + +- **School page**: schools with `has_sixth_form = true` show a Sixth form + results section (placeholder until KS5 data lands); schools without never + show it. Badge on the header as today, but driven by the flag. +- **Search/rankings filter**: "With sixth form" / "Without sixth form" uses + the flag; applies to secondary and all-through phases. +- **Comparison**: when comparing a with-sixth-form school against one + without, the Sixth form group renders "No sixth form" for the latter + rather than blank cells, making the structural difference explicit. + +## 4. Sixth form placeholders — future KS5 ingestion spec + +Source: DfE "A level and other 16 to 18 results" (EES, preferred — matches +the KS4 EES tap) or legacy performance-tables `england_ks5final.csv`. +Column names below are from the legacy KS5 CSV; verify against the EES +release chosen at ingestion time. + +| Proposed metric key | Name | Legacy source column | Type | +|---|---|---|---| +| `alevel_aps_per_entry` | A level average points per entry | `TALLPPE_ALEV_1618` | score | +| `alevel_avg_grade` | A level average grade (e.g. B-) | `TALLPPEGRD_ALEV_1618` | grade | +| `academic_aps_per_entry` | Academic qualifications APS per entry | `TALLPPE_ACAD_1618` | score | +| `applied_general_aps_per_entry` | Applied general APS per entry | `TALLPPE_AGEN_1618` | score | +| `tech_level_aps_per_entry` | Tech level APS per entry | `TALLPPE_TLEV_1618` | score | +| `english_progress_1618` | English progress (16–18, unfinished GCSE 4+) | `PROGENG_1618` | score | +| `maths_progress_1618` | Maths progress (16–18) | `PROGMAT_1618` | score | +| `ks5_cohort_size` | Students at end of 16–18 study | `TALLPUP_1618` | count | +| `alevel_3plus_aab_pct` | % achieving AAB+ in ≥2 facilitating subjects | `TAAB2FAC_1618` | percentage | +| `ks5_retention_pct` | Retention (completed main programme) | study-programme retention measure | percentage | +| `ks5_destinations_pct` | Sustained education/employment destination | 16–18 destination measures dataset | percentage | + +Proposed landing shape mirrors KS4: `stg_ees_ks5.sql` → +`int_ks5_with_lineage.sql` → `marts.fact_ks5_performance` (one row per URN +per year), joined into `fact_performance`, with a `category: "sixth_form"` +(or `"alevel"`) block added to `METRIC_DEFINITIONS`. + +## 5. Out of scope + +- Any implementation (pipeline, API, or UI changes) — this is the taxonomy + reference; implementation work items are §3 "Pipeline change", the + heuristic migration audit, and §4 ingestion, each to be planned separately. +- Middle schools (deemed secondary/primary): they follow the assessment-based + grouping automatically — no special casing. +- Independent schools: no DfE performance data published; unaffected. From 87f4c6dd404a9021f9102e2811420507b4428b79 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 10:21:02 +0100 Subject: [PATCH 02/10] docs: implementation plan for GIAS sixth-form flag Co-Authored-By: Claude Fable 5 --- .../plans/2026-07-07-gias-sixth-form-flag.md | 560 ++++++++++++++++++ 1 file changed, 560 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md diff --git a/docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md b/docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md new file mode 100644 index 0000000..1a7224d --- /dev/null +++ b/docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md @@ -0,0 +1,560 @@ +# GIAS OfficialSixthForm Flag Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ingest GIAS's authoritative `OfficialSixthForm` flag into `marts.dim_school.has_sixth_form` and replace every `age_range contains "18"` heuristic in the backend and frontend with it. + +**Architecture:** Data flows tap → raw → dbt staging → dbt mart → backend SQL → API payload → Next.js components. The GIAS Singer tap must declare the new CSV column (target-postgres only persists declared columns); the dbt staging model renames it; `dim_school` derives a boolean (with a statutory-age fallback for blank GIAS values); the backend exposes it on list + detail payloads and uses it for the `has_sixth_form=yes|no` filter; the frontend badge/note/filter-labels switch from the age-range substring check to the flag. + +**Tech Stack:** Singer SDK (tap), dbt (Postgres), FastAPI + pandas, Next.js + TypeScript, pytest, Jest/RTL. + +**Spec:** `docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md` §3. + +## Global Constraints + +- A school **has a sixth form** iff GIAS `OfficialSixthForm (name)` = `"Has a sixth form"`. `"Does not have a sixth form"` and `"Not applicable"` → false. Blank/NULL (rare) → fall back to `statutory_high_age >= 18`. +- The public API filter parameter stays `has_sixth_form=yes|no` (unchanged contract). +- Filter dropdown labels must drop the age-range parentheticals: "With sixth form" / "Without sixth form" (sixth form ≠ age range). +- Never push to `main`; work stays on branch `feat/gias-sixth-form-flag` (create from `docs/exam-phase-taxonomy` so the spec is included, or from `main` if that branch has merged). +- The dbt models cannot be run locally (no pipeline DB); dbt changes are verified by review + `python -c` schema asserts + existing CI. Do NOT attempt to start a local server. +- The backend marts tables are dbt `table` materializations — rebuilt on every pipeline run, so **no ALTER TABLE migration is needed** for `marts.dim_school`. +- Deployment ordering: the tap must run before dbt on the first pipeline run after deploy (this is already the DAG order: extract → transform). Until that run happens, `has_sixth_form` is absent from the DB; the backend must treat a missing column as "flag false / fallback", never crash. + +--- + +### Task 1: Ingest `OfficialSixthForm (name)` — tap schema + dbt staging + +**Files:** +- Modify: `pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py:31-66` (Singer schema) +- Modify: `pipeline/transform/models/staging/stg_gias_establishments.sql` (add renamed column) + +**Interfaces:** +- Produces: raw column `"OfficialSixthForm (name)"` in `raw.gias_establishments`; staging column `official_sixth_form` (text: `Has a sixth form` / `Does not have a sixth form` / `Not applicable` / NULL) consumed by Task 2. + +- [ ] **Step 1: Add the property to the Singer schema** + +In `tap.py`, inside `GIASEstablishmentsStream.schema = th.PropertiesList(...)`, add after the `th.Property("PhaseOfEducation (name)", th.StringType),` line: + +```python + th.Property("OfficialSixthForm (name)", th.StringType), +``` + +- [ ] **Step 2: Verify the tap module still imports and declares the column** + +Run: +```bash +cd /Users/tudor/projects/school_compare/pipeline/plugins/extractors/tap-uk-gias && \ +python3 -c " +import ast, sys +src = open('tap_uk_gias/tap.py').read() +ast.parse(src) +assert '\"OfficialSixthForm (name)\"' in src.replace(\"'\", '\"') +print('OK: tap declares OfficialSixthForm (name)') +" +``` +Expected: `OK: tap declares OfficialSixthForm (name)` +(Uses `ast.parse` instead of importing because `singer_sdk` is not installed locally.) + +- [ ] **Step 3: Add the column to the staging model** + +In `stg_gias_establishments.sql`, in the `renamed` CTE, add after the `"PhaseOfEducation (name)" as phase,` line: + +```sql + nullif(trim("OfficialSixthForm (name)"), '') as official_sixth_form, +``` + +- [ ] **Step 4: Sanity-check the SQL edit** + +Run: +```bash +grep -n "official_sixth_form" /Users/tudor/projects/school_compare/pipeline/transform/models/staging/stg_gias_establishments.sql +``` +Expected: one line showing the new column inside the `renamed` CTE (before `from source`). + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py pipeline/transform/models/staging/stg_gias_establishments.sql +git commit -m "feat(pipeline): ingest GIAS OfficialSixthForm into staging + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 2: Derive `dim_school.has_sixth_form` (dbt mart + schema tests + SQLAlchemy model) + +**Files:** +- Modify: `pipeline/transform/models/marts/dim_school.sql` (add derived column) +- Modify: `pipeline/transform/models/marts/_marts_schema.yml` (document + test the column) +- Modify: `backend/models.py:13-38` (`DimSchool` — add column) + +**Interfaces:** +- Consumes: `official_sixth_form` text column from Task 1's staging model. +- Produces: `marts.dim_school.has_sixth_form boolean not null`, and `DimSchool.has_sixth_form = Column(Boolean)` for the backend. Task 3 selects it as `s.has_sixth_form`. + +- [ ] **Step 1: Add the derived column to `dim_school.sql`** + +In the `select`, add after the `s.age_range` line (`s.statutory_low_age || '-' || s.statutory_high_age as age_range,`): + +```sql + -- Authoritative sixth-form flag (spec §3): GIAS OfficialSixthForm. + -- "Not applicable" (nurseries, primaries, PRUs) => false. Blank GIAS + -- value (rare, new establishments) falls back to the statutory age range. + case + when s.official_sixth_form = 'Has a sixth form' then true + when s.official_sixth_form in ('Does not have a sixth form', 'Not applicable') then false + else coalesce(s.statutory_high_age >= 18, false) + end as has_sixth_form, +``` + +- [ ] **Step 2: Add schema documentation + tests in `_marts_schema.yml`** + +Under `- name: dim_school` → `columns:`, add after the `phase` column block: + +```yaml + - name: has_sixth_form + description: > + Authoritative sixth-form flag from GIAS OfficialSixthForm. + "Has a sixth form" => true; "Does not have a sixth form" and + "Not applicable" => false; blank GIAS value falls back to + statutory_high_age >= 18. Replaces the age_range-contains-"18" + heuristic (spec 2026-07-07 §3). + tests: + - not_null + - accepted_values: + values: [true, false] +``` + +- [ ] **Step 3: Add the column to the `DimSchool` SQLAlchemy model** + +In `backend/models.py`, in `class DimSchool`, add after `age_range = Column(String(20))`: + +```python + has_sixth_form = Column(Boolean) +``` + +- [ ] **Step 4: Verify SQL/YAML/Python all parse** + +Run: +```bash +cd /Users/tudor/projects/school_compare && \ +python3 -c " +import yaml +y = yaml.safe_load(open('pipeline/transform/models/marts/_marts_schema.yml')) +dim = [m for m in y['models'] if m['name'] == 'dim_school'][0] +cols = [c['name'] for c in dim['columns']] +assert 'has_sixth_form' in cols, cols +print('OK: schema yml documents has_sixth_form') +" && \ +grep -c "has_sixth_form" pipeline/transform/models/marts/dim_school.sql && \ +python3 -c "import ast; ast.parse(open('backend/models.py').read()); print('OK: models.py parses')" +``` +Expected: `OK: schema yml documents has_sixth_form`, grep count `>= 1`, `OK: models.py parses`. + +- [ ] **Step 5: Commit** + +```bash +git add pipeline/transform/models/marts/dim_school.sql pipeline/transform/models/marts/_marts_schema.yml backend/models.py +git commit -m "feat(pipeline): derive dim_school.has_sixth_form from GIAS flag + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 3: Backend — expose `has_sixth_form` and replace the filter heuristic + +**Files:** +- Modify: `backend/data_loader.py:117-215` (`_MAIN_QUERY` — select the column) +- Modify: `backend/schemas.py:536-553` (`SCHOOL_COLUMNS` — include in list payloads) +- Modify: `backend/app.py:419-422` (filter) and `backend/app.py:589-610` (detail `school_info`) +- Test: `backend/tests/test_sixth_form_flag.py` (new) + +**Interfaces:** +- Consumes: `marts.dim_school.has_sixth_form` (Task 2). +- Produces: `has_sixth_form: bool | null` field on `GET /api/schools` items and on `GET /api/schools/{urn}` → `school_info`. Filter `GET /api/schools?has_sixth_form=yes|no` now driven by the flag. Frontend (Task 4) reads `school.has_sixth_form`. + +- [ ] **Step 1: Write the failing tests** + +Create `backend/tests/test_sixth_form_flag.py`: + +```python +"""Tests for the GIAS-driven has_sixth_form flag (spec 2026-07-07 §3). + +The filter and payloads must use dim_school.has_sixth_form, not the old +age_range-contains-"18" substring heuristic. The key regression case is a +16-19 sixth-form college: flag true, but "16-19" contains no "18". +""" + +import numpy as np +import pandas as pd +import pytest +from fastapi.testclient import TestClient + + +def _schools_df() -> pd.DataFrame: + """Latest-year snapshot rows as produced by load_latest_school_data.""" + base = { + "local_authority": "Testshire", + "school_type": "Academy", + "phase": "Secondary", + "address": "1 Test Street", + "town": "Testtown", + "postcode": "TS1 1AA", + "religious_denomination": None, + "gender": "Mixed", + "admissions_policy": None, + "ofsted_grade": np.nan, + "ofsted_date": None, + "ofsted_framework": None, + "latitude": 51.5, + "longitude": -0.1, + "year": 202425, + "total_pupils": 1000, + "rwm_expected_pct": np.nan, + "attainment_8_score": 50.0, + } + return pd.DataFrame( + [ + # 11-18 school WITH a registered sixth form + {**base, "urn": 100001, "school_name": "Alpha High", + "age_range": "11-18", "has_sixth_form": True}, + # 16-19 college: old heuristic said NO ("16-19" has no "18"), + # GIAS flag says YES — must appear in the yes-filter results + {**base, "urn": 100002, "school_name": "Beta Sixth Form College", + "age_range": "16-19", "has_sixth_form": True}, + # 11-18 age range on paper but NO registered sixth form: + # old heuristic said YES, GIAS flag says NO + {**base, "urn": 100003, "school_name": "Gamma Academy", + "age_range": "11-18", "has_sixth_form": False}, + # Missing flag (pipeline not yet re-run) — must not crash, + # must not match the yes-filter + {**base, "urn": 100004, "school_name": "Delta School", + "age_range": "11-16", "has_sixth_form": None}, + ] + ) + + +@pytest.fixture() +def client(monkeypatch): + from backend import app as app_module + + monkeypatch.setattr(app_module, "load_latest_school_data", _schools_df) + monkeypatch.setattr(app_module, "load_school_data", _schools_df) + monkeypatch.setattr(app_module, "get_supplementary_data", lambda db, urn: {}) + return TestClient(app_module.app, raise_server_exceptions=False) + + +def _urns(resp): + return sorted(s["urn"] for s in resp.json()["schools"]) + + +def test_filter_yes_uses_flag_not_age_range(client): + resp = client.get("/api/schools?has_sixth_form=yes") + assert resp.status_code == 200, resp.text + # 16-19 college included; 11-18-without-sixth-form excluded + assert _urns(resp) == [100001, 100002] + + +def test_filter_no_uses_flag_not_age_range(client): + resp = client.get("/api/schools?has_sixth_form=no") + assert resp.status_code == 200, resp.text + # Gamma (flag false) and Delta (flag missing => not true) + assert _urns(resp) == [100003, 100004] + + +def test_list_payload_includes_flag(client): + resp = client.get("/api/schools") + assert resp.status_code == 200, resp.text + by_urn = {s["urn"]: s for s in resp.json()["schools"]} + assert by_urn[100002]["has_sixth_form"] is True + assert by_urn[100003]["has_sixth_form"] is False + assert by_urn[100004]["has_sixth_form"] is None + + +def test_detail_payload_includes_flag(client): + resp = client.get("/api/schools/100002") + assert resp.status_code == 200, resp.text + assert resp.json()["school_info"]["has_sixth_form"] is True +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests/test_sixth_form_flag.py -v` +Expected: FAIL — `test_filter_yes_uses_flag_not_age_range` asserts `[100001, 100002]` but the age-range heuristic returns `[100001, 100003]`; the payload tests fail with `KeyError: 'has_sixth_form'`. + +- [ ] **Step 3: Select the column in `_MAIN_QUERY`** + +In `backend/data_loader.py`, in `_MAIN_QUERY`, add after `s.age_range,`: + +```sql + s.has_sixth_form, +``` + +- [ ] **Step 4: Include it in list payloads** + +In `backend/schemas.py`, in `SCHOOL_COLUMNS`, add after `"age_range",`: + +```python + "has_sixth_form", +``` + +(`app.py` builds list responses from `SCHOOL_COLUMNS ∩ df.columns`, so a DB that predates the pipeline re-run simply omits the field — no crash.) + +- [ ] **Step 5: Replace the filter heuristic in `app.py`** + +Replace lines 419-422: + +```python + if has_sixth_form == "yes": + df_latest = df_latest[df_latest["age_range"].str.contains("18", na=False)] + elif has_sixth_form == "no": + df_latest = df_latest[~df_latest["age_range"].str.contains("18", na=False)] +``` + +with: + +```python + # GIAS OfficialSixthForm flag (dim_school.has_sixth_form). NULL (flag not + # yet populated by the pipeline) is treated as "no sixth form". + if has_sixth_form in ("yes", "no"): + if "has_sixth_form" in df_latest.columns: + flag = df_latest["has_sixth_form"].eq(True) + else: # DB predates the pipeline re-run — fall back to age range + flag = df_latest["age_range"].str.contains("18", na=False) + df_latest = df_latest[flag if has_sixth_form == "yes" else ~flag] +``` + +- [ ] **Step 6: Add the flag to the detail payload** + +In `backend/app.py` `school_info` dict (line ~598), add after `"age_range": latest.get("age_range", ""),`: + +```python + "has_sixth_form": latest.get("has_sixth_form"), +``` + +(`convert_to_native` already maps NaN/None → null and numpy bools → bool.) + +- [ ] **Step 7: Run the new tests** + +Run: `cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests/test_sixth_form_flag.py -v` +Expected: 4 passed. + +- [ ] **Step 8: Run the full backend suite** + +Run: `cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests -v` +Expected: all pass (the pre-existing `test_school_details.py` df has no `has_sixth_form` column — `latest.get()` returns None, serialized as null). + +- [ ] **Step 9: Commit** + +```bash +git add backend/data_loader.py backend/schemas.py backend/app.py backend/tests/test_sixth_form_flag.py +git commit -m "feat(api): drive has_sixth_form filter and payloads from GIAS flag + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 4: Frontend — badge, note, row tag, and filter labels use the flag + +**Files:** +- Modify: `nextjs-app/lib/types.ts:10-30` (`School` interface) +- Modify: `nextjs-app/components/SecondarySchoolDetailView.tsx:101` (badge + coming-soon note) +- Modify: `nextjs-app/components/SecondarySchoolRow.tsx:25-27` (row tag) +- Modify: `nextjs-app/components/FilterBar.tsx:370-372` (labels only — param name unchanged) +- Test: `nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx` (new) + +**Interfaces:** +- Consumes: `has_sixth_form: boolean | null` on both list items and `school_info` (Task 3; both are typed as `School`). +- Produces: no new exports — behavior change only. + +- [ ] **Step 1: Write the failing test** + +Create `nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx`: + +```tsx +/** + * SecondarySchoolRow — sixth-form tag must come from the GIAS + * has_sixth_form flag, not the age_range-contains-"18" heuristic. + */ + +import '@testing-library/jest-dom'; +import { render, screen } from '@testing-library/react'; +import { SecondarySchoolRow } from '@/components/SecondarySchoolRow'; +import type { School } from '@/lib/types'; + +const base = { + urn: 100002, + school_name: 'Beta Sixth Form College', + local_authority: 'Testshire', + school_type: 'Academy', + phase: 'Secondary', + gender: 'Mixed', + attainment_8_score: 50.0, +} as unknown as School; + +describe('SecondarySchoolRow sixth-form tag', () => { + it('shows the tag for a 16-19 college with the GIAS flag set', () => { + render( + , + ); + expect(screen.getByText('Sixth form')).toBeInTheDocument(); + }); + + it('hides the tag for an 11-18 school without a registered sixth form', () => { + render( + , + ); + expect(screen.queryByText('Sixth form')).not.toBeInTheDocument(); + }); + + it('hides the tag when the flag is missing (pipeline not yet re-run)', () => { + render( + , + ); + expect(screen.queryByText('Sixth form')).not.toBeInTheDocument(); + }); +}); +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd /Users/tudor/projects/school_compare/nextjs-app && npx jest __tests__/components/SecondarySchoolRow.test.tsx` +Expected: FAIL — first test can't find "Sixth form" ("16-19" fails the substring check), second test finds an unexpected "Sixth form" tag. (If TS complains that `has_sixth_form` is not on `School`, that is the same failure — proceed.) + +- [ ] **Step 3: Add the field to the `School` type** + +In `nextjs-app/lib/types.ts`, in `export interface School`, add after `age_range: string | null;`: + +```ts + has_sixth_form?: boolean | null; +``` + +- [ ] **Step 4: Switch `SecondarySchoolRow` to the flag** + +Replace the helper at `SecondarySchoolRow.tsx:25-27`: + +```ts +function hasSixthForm(school: School): boolean { + return school.age_range?.includes('18') ?? false; +} +``` + +with: + +```ts +function hasSixthForm(school: School): boolean { + // GIAS OfficialSixthForm flag; missing (pipeline not yet re-run) => false. + return school.has_sixth_form ?? false; +} +``` + +- [ ] **Step 5: Switch `SecondarySchoolDetailView` to the flag** + +Replace line 101: + +```ts + const hasSixthForm = schoolInfo.age_range?.includes('18') ?? false; +``` + +with: + +```ts + // GIAS OfficialSixthForm flag; missing (pipeline not yet re-run) => false. + const hasSixthForm = schoolInfo.has_sixth_form ?? false; +``` + +(This drives both the header "Sixth form" badge at line ~230 and the "Post-16 destination data coming soon" note at line ~715 — no changes needed there.) + +- [ ] **Step 6: Fix the filter labels in `FilterBar.tsx`** + +Replace: + +```tsx + + +``` + +with: + +```tsx + + +``` + +- [ ] **Step 7: Run the new test and verify it passes** + +Run: `cd /Users/tudor/projects/school_compare/nextjs-app && npx jest __tests__/components/SecondarySchoolRow.test.tsx` +Expected: 3 passed. + +- [ ] **Step 8: Run the full frontend checks** + +Run: `cd /Users/tudor/projects/school_compare/nextjs-app && npx tsc --noEmit && npx jest` +Expected: typecheck clean, all Jest suites pass. + +- [ ] **Step 9: Verify no heuristic remains** + +Run: +```bash +grep -rn "includes('18')\|contains(\"18\")" /Users/tudor/projects/school_compare/nextjs-app/components /Users/tudor/projects/school_compare/backend --include="*.tsx" --include="*.ts" --include="*.py" | grep -v test +``` +Expected: only the documented fallback inside `app.py` (DB-predates-pipeline branch); no other hits. + +- [ ] **Step 10: Commit** + +```bash +git add nextjs-app/lib/types.ts nextjs-app/components/SecondarySchoolRow.tsx nextjs-app/components/SecondarySchoolDetailView.tsx nextjs-app/components/FilterBar.tsx nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx +git commit -m "feat(ui): sixth-form badge, note and filter labels use GIAS flag + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 5: Update the spec status + PR + +**Files:** +- Modify: `docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md` (§3 "Pipeline change (future work)" → implemented) + +**Interfaces:** +- Consumes: everything above merged into the branch. +- Produces: PR ready for review; e2e journeys are the promotion gate (no journey currently exercises the sixth-form filter, and the API contract is unchanged, so no e2e change is required — state this in the PR body). + +- [ ] **Step 1: Mark spec §3 pipeline change as implemented** + +In the spec, change the §3 heading `### Pipeline change (future work)` to `### Pipeline change (implemented 2026-07-07)` and append one line at the end of that subsection: + +```markdown +Implemented in `feat/gias-sixth-form-flag` — see +`docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md`. +``` + +- [ ] **Step 2: Commit** + +```bash +git add docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md +git commit -m "docs: mark sixth-form flag pipeline change implemented + +Co-Authored-By: Claude Fable 5 " +``` + +- [ ] **Step 3: Push and open the PR (Gitea)** + +Push the branch, then create the PR against `main` using the Gitea API via the git credential helper (token-header auth 401s on this Gitea; basic auth from `git credential fill` works): + +```bash +git push -u origin feat/gias-sixth-form-flag +``` + +PR title: `feat: drive sixth-form separation from GIAS OfficialSixthForm flag` +PR body must note: (1) API contract unchanged (`has_sixth_form=yes|no`), (2) flag is NULL until the next pipeline run — backend and frontend degrade to "no sixth form" / age-range fallback, (3) no e2e journey change needed, and end with the standard generation footer. + +- [ ] **Step 4: Verify CI passes** + +Watch the PR checks (typecheck, tests, builds, AI review). All must pass before merge; merging deploys to staging automatically. From 3b35849bb32ed72bde4a8c4fe0e09b4649c228f7 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 10:28:08 +0100 Subject: [PATCH 03/10] feat(pipeline): ingest GIAS OfficialSixthForm into staging Co-Authored-By: Claude Fable 5 --- pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py | 1 + pipeline/transform/models/staging/stg_gias_establishments.sql | 1 + 2 files changed, 2 insertions(+) diff --git a/pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py b/pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py index 250dc25..66dec17 100644 --- a/pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py +++ b/pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py @@ -33,6 +33,7 @@ class GIASEstablishmentsStream(Stream): th.Property("EstablishmentName", th.StringType), th.Property("TypeOfEstablishment (name)", th.StringType), th.Property("PhaseOfEducation (name)", th.StringType), + th.Property("OfficialSixthForm (name)", th.StringType), th.Property("LA (code)", th.StringType), th.Property("LA (name)", th.StringType), th.Property("EstablishmentNumber", th.StringType), diff --git a/pipeline/transform/models/staging/stg_gias_establishments.sql b/pipeline/transform/models/staging/stg_gias_establishments.sql index 3cea609..8763a86 100644 --- a/pipeline/transform/models/staging/stg_gias_establishments.sql +++ b/pipeline/transform/models/staging/stg_gias_establishments.sql @@ -14,6 +14,7 @@ renamed as ( "EstablishmentName" as school_name, "TypeOfEstablishment (name)" as school_type, "PhaseOfEducation (name)" as phase, + nullif(trim("OfficialSixthForm (name)"), '') as official_sixth_form, "Gender (name)" as gender, "ReligiousCharacter (name)" as religious_character, "AdmissionsPolicy (name)" as admissions_policy, From d11faefebd531ef64a8f7e6564f09fa41b4fe1e5 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 10:30:39 +0100 Subject: [PATCH 04/10] feat(pipeline): derive dim_school.has_sixth_form from GIAS flag Co-Authored-By: Claude Fable 5 --- backend/models.py | 1 + pipeline/transform/models/marts/_marts_schema.yml | 11 +++++++++++ pipeline/transform/models/marts/dim_school.sql | 8 ++++++++ 3 files changed, 20 insertions(+) diff --git a/backend/models.py b/backend/models.py index 314fe59..a83bcb7 100644 --- a/backend/models.py +++ b/backend/models.py @@ -24,6 +24,7 @@ class DimSchool(Base): religious_character = Column(String(100)) gender = Column(String(20)) age_range = Column(String(20)) + has_sixth_form = Column(Boolean) capacity = Column(Integer) total_pupils = Column(Integer) headteacher_name = Column(String(200)) diff --git a/pipeline/transform/models/marts/_marts_schema.yml b/pipeline/transform/models/marts/_marts_schema.yml index 7fb0201..a575a95 100644 --- a/pipeline/transform/models/marts/_marts_schema.yml +++ b/pipeline/transform/models/marts/_marts_schema.yml @@ -16,6 +16,17 @@ models: tests: - not_null: severity: warn + - name: has_sixth_form + description: > + Authoritative sixth-form flag from GIAS OfficialSixthForm. + "Has a sixth form" => true; "Does not have a sixth form" and + "Not applicable" => false; blank GIAS value falls back to + statutory_high_age >= 18. Replaces the age_range-contains-"18" + heuristic (spec 2026-07-07 §3). + tests: + - not_null + - accepted_values: + values: [true, false] - name: status tests: - accepted_values: diff --git a/pipeline/transform/models/marts/dim_school.sql b/pipeline/transform/models/marts/dim_school.sql index fba0296..a09a016 100644 --- a/pipeline/transform/models/marts/dim_school.sql +++ b/pipeline/transform/models/marts/dim_school.sql @@ -52,6 +52,14 @@ select s.religious_character, s.gender, s.statutory_low_age || '-' || s.statutory_high_age as age_range, + -- Authoritative sixth-form flag (spec §3): GIAS OfficialSixthForm. + -- "Not applicable" (nurseries, primaries, PRUs) => false. Blank GIAS + -- value (rare, new establishments) falls back to the statutory age range. + case + when s.official_sixth_form = 'Has a sixth form' then true + when s.official_sixth_form in ('Does not have a sixth form', 'Not applicable') then false + else coalesce(s.statutory_high_age >= 18, false) + end as has_sixth_form, s.capacity, s.total_pupils, concat_ws(' ', s.head_title, s.head_first_name, s.head_last_name) as headteacher_name, From 1d149ffc487930c3bf3f66c17f2313d19331f5a9 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 10:36:39 +0100 Subject: [PATCH 05/10] feat(api): drive has_sixth_form filter and payloads from GIAS flag Co-Authored-By: Claude Fable 5 --- backend/app.py | 13 ++-- backend/data_loader.py | 1 + backend/schemas.py | 1 + backend/tests/test_sixth_form_flag.py | 97 +++++++++++++++++++++++++++ 4 files changed, 108 insertions(+), 4 deletions(-) create mode 100644 backend/tests/test_sixth_form_flag.py diff --git a/backend/app.py b/backend/app.py index a146f19..9b63008 100644 --- a/backend/app.py +++ b/backend/app.py @@ -416,10 +416,14 @@ async def get_schools( df_latest = df_latest[df_latest["gender"].str.lower() == gender.lower()] if admissions_policy: df_latest = df_latest[df_latest["admissions_policy"].str.lower() == admissions_policy.lower()] - if has_sixth_form == "yes": - df_latest = df_latest[df_latest["age_range"].str.contains("18", na=False)] - elif has_sixth_form == "no": - df_latest = df_latest[~df_latest["age_range"].str.contains("18", na=False)] + # GIAS OfficialSixthForm flag (dim_school.has_sixth_form). NULL (flag not + # yet populated by the pipeline) is treated as "no sixth form". + if has_sixth_form in ("yes", "no"): + if "has_sixth_form" in df_latest.columns: + flag = df_latest["has_sixth_form"].eq(True) + else: # DB predates the pipeline re-run — fall back to age range + flag = df_latest["age_range"].str.contains("18", na=False) + df_latest = df_latest[flag if has_sixth_form == "yes" else ~flag] # Include key result metrics for display on cards location_cols = ["latitude", "longitude"] @@ -596,6 +600,7 @@ async def get_school_details(request: Request, urn: int): "address": latest.get("address", ""), "religious_denomination": latest.get("religious_denomination", ""), "age_range": latest.get("age_range", ""), + "has_sixth_form": latest.get("has_sixth_form"), "latitude": latest.get("latitude"), "longitude": latest.get("longitude"), "phase": latest.get("phase"), diff --git a/backend/data_loader.py b/backend/data_loader.py index ab9337d..6221461 100644 --- a/backend/data_loader.py +++ b/backend/data_loader.py @@ -125,6 +125,7 @@ _MAIN_QUERY = text(""" s.religious_character AS religious_denomination, s.gender, s.age_range, + s.has_sixth_form, s.admissions_policy, s.capacity, s.total_pupils AS gias_total_pupils, diff --git a/backend/schemas.py b/backend/schemas.py index 6d46541..a757469 100644 --- a/backend/schemas.py +++ b/backend/schemas.py @@ -543,6 +543,7 @@ SCHOOL_COLUMNS = [ "postcode", "religious_denomination", "age_range", + "has_sixth_form", "gender", "admissions_policy", "ofsted_grade", diff --git a/backend/tests/test_sixth_form_flag.py b/backend/tests/test_sixth_form_flag.py new file mode 100644 index 0000000..fe1782a --- /dev/null +++ b/backend/tests/test_sixth_form_flag.py @@ -0,0 +1,97 @@ +"""Tests for the GIAS-driven has_sixth_form flag (spec 2026-07-07 §3). + +The filter and payloads must use dim_school.has_sixth_form, not the old +age_range-contains-"18" substring heuristic. The key regression case is a +16-19 sixth-form college: flag true, but "16-19" contains no "18". +""" + +import numpy as np +import pandas as pd +import pytest +from fastapi.testclient import TestClient + + +def _schools_df() -> pd.DataFrame: + """Latest-year snapshot rows as produced by load_latest_school_data.""" + base = { + "local_authority": "Testshire", + "school_type": "Academy", + "phase": "Secondary", + "address": "1 Test Street", + "town": "Testtown", + "postcode": "TS1 1AA", + "religious_denomination": None, + "gender": "Mixed", + "admissions_policy": None, + "ofsted_grade": np.nan, + "ofsted_date": None, + "ofsted_framework": None, + "latitude": 51.5, + "longitude": -0.1, + "year": 202425, + "total_pupils": 1000, + "rwm_expected_pct": np.nan, + "attainment_8_score": 50.0, + } + return pd.DataFrame( + [ + # 11-18 school WITH a registered sixth form + {**base, "urn": 100001, "school_name": "Alpha High", + "age_range": "11-18", "has_sixth_form": True}, + # 16-19 college: old heuristic said NO ("16-19" has no "18"), + # GIAS flag says YES — must appear in the yes-filter results + {**base, "urn": 100002, "school_name": "Beta Sixth Form College", + "age_range": "16-19", "has_sixth_form": True}, + # 11-18 age range on paper but NO registered sixth form: + # old heuristic said YES, GIAS flag says NO + {**base, "urn": 100003, "school_name": "Gamma Academy", + "age_range": "11-18", "has_sixth_form": False}, + # Missing flag (pipeline not yet re-run) — must not crash, + # must not match the yes-filter + {**base, "urn": 100004, "school_name": "Delta School", + "age_range": "11-16", "has_sixth_form": None}, + ] + ) + + +@pytest.fixture() +def client(monkeypatch): + from backend import app as app_module + + monkeypatch.setattr(app_module, "load_latest_school_data", _schools_df) + monkeypatch.setattr(app_module, "load_school_data", _schools_df) + monkeypatch.setattr(app_module, "get_supplementary_data", lambda db, urn: {}) + return TestClient(app_module.app, raise_server_exceptions=False) + + +def _urns(resp): + return sorted(s["urn"] for s in resp.json()["schools"]) + + +def test_filter_yes_uses_flag_not_age_range(client): + resp = client.get("/api/schools?has_sixth_form=yes") + assert resp.status_code == 200, resp.text + # 16-19 college included; 11-18-without-sixth-form excluded + assert _urns(resp) == [100001, 100002] + + +def test_filter_no_uses_flag_not_age_range(client): + resp = client.get("/api/schools?has_sixth_form=no") + assert resp.status_code == 200, resp.text + # Gamma (flag false) and Delta (flag missing => not true) + assert _urns(resp) == [100003, 100004] + + +def test_list_payload_includes_flag(client): + resp = client.get("/api/schools") + assert resp.status_code == 200, resp.text + by_urn = {s["urn"]: s for s in resp.json()["schools"]} + assert by_urn[100002]["has_sixth_form"] is True + assert by_urn[100003]["has_sixth_form"] is False + assert by_urn[100004]["has_sixth_form"] is None + + +def test_detail_payload_includes_flag(client): + resp = client.get("/api/schools/100002") + assert resp.status_code == 200, resp.text + assert resp.json()["school_info"]["has_sixth_form"] is True From 0934c8f38ca1cea44b70407f3da5fc27d0f26586 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 10:40:42 +0100 Subject: [PATCH 06/10] feat(ui): sixth-form badge, note and filter labels use GIAS flag Co-Authored-By: Claude Fable 5 --- .../components/SecondarySchoolRow.test.tsx | 46 +++++++++++++++++++ nextjs-app/components/FilterBar.tsx | 4 +- .../components/SecondarySchoolDetailView.tsx | 3 +- nextjs-app/components/SecondarySchoolRow.tsx | 3 +- nextjs-app/lib/types.ts | 1 + 5 files changed, 53 insertions(+), 4 deletions(-) create mode 100644 nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx diff --git a/nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx b/nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx new file mode 100644 index 0000000..affaecb --- /dev/null +++ b/nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx @@ -0,0 +1,46 @@ +/** + * SecondarySchoolRow — sixth-form tag must come from the GIAS + * has_sixth_form flag, not the age_range-contains-"18" heuristic. + */ + +import '@testing-library/jest-dom'; +import { render, screen } from '@testing-library/react'; +import { SecondarySchoolRow } from '@/components/SecondarySchoolRow'; +import type { School } from '@/lib/types'; + +const base = { + urn: 100002, + school_name: 'Beta Sixth Form College', + local_authority: 'Testshire', + school_type: 'Academy', + phase: 'Secondary', + gender: 'Mixed', + attainment_8_score: 50.0, +} as unknown as School; + +describe('SecondarySchoolRow sixth-form tag', () => { + it('shows the tag for a 16-19 college with the GIAS flag set', () => { + render( + , + ); + expect(screen.getByText('Sixth form')).toBeInTheDocument(); + }); + + it('hides the tag for an 11-18 school without a registered sixth form', () => { + render( + , + ); + expect(screen.queryByText('Sixth form')).not.toBeInTheDocument(); + }); + + it('hides the tag when the flag is missing (pipeline not yet re-run)', () => { + render( + , + ); + expect(screen.queryByText('Sixth form')).not.toBeInTheDocument(); + }); +}); diff --git a/nextjs-app/components/FilterBar.tsx b/nextjs-app/components/FilterBar.tsx index 51ee0eb..91abf00 100644 --- a/nextjs-app/components/FilterBar.tsx +++ b/nextjs-app/components/FilterBar.tsx @@ -368,8 +368,8 @@ export function FilterBar({ disabled={isPending} > - - + + {admissionsPolicyOptions.length > 0 && ( diff --git a/nextjs-app/components/SecondarySchoolDetailView.tsx b/nextjs-app/components/SecondarySchoolDetailView.tsx index a8e8b2c..8210313 100644 --- a/nextjs-app/components/SecondarySchoolDetailView.tsx +++ b/nextjs-app/components/SecondarySchoolDetailView.tsx @@ -98,7 +98,8 @@ export function SecondarySchoolDetailView({ const secondaryAvg = nationalAvg?.secondary ?? {}; - const hasSixthForm = schoolInfo.age_range?.includes('18') ?? false; + // GIAS OfficialSixthForm flag; missing (pipeline not yet re-run) => false. + const hasSixthForm = schoolInfo.has_sixth_form ?? false; const hasFinance = finance != null && finance.per_pupil_spend != null; const hasDeprivation = deprivation != null && deprivation.idaci_decile != null; const hasLocation = schoolInfo.latitude != null && schoolInfo.longitude != null; diff --git a/nextjs-app/components/SecondarySchoolRow.tsx b/nextjs-app/components/SecondarySchoolRow.tsx index 61097ae..c32dab4 100644 --- a/nextjs-app/components/SecondarySchoolRow.tsx +++ b/nextjs-app/components/SecondarySchoolRow.tsx @@ -23,7 +23,8 @@ function detectAdmissionsTag(school: School): string | null { } function hasSixthForm(school: School): boolean { - return school.age_range?.includes('18') ?? false; + // GIAS OfficialSixthForm flag; missing (pipeline not yet re-run) => false. + return school.has_sixth_form ?? false; } interface SecondarySchoolRowProps { diff --git a/nextjs-app/lib/types.ts b/nextjs-app/lib/types.ts index 816516e..dc1cd2a 100644 --- a/nextjs-app/lib/types.ts +++ b/nextjs-app/lib/types.ts @@ -17,6 +17,7 @@ export interface School { school_type_code: string | null; religious_denomination: string | null; age_range: string | null; + has_sixth_form?: boolean | null; // Address address1: string | null; From 3fcb1340d442726aaef2771c9daa9dad4589c764 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 10:43:08 +0100 Subject: [PATCH 07/10] docs: mark sixth-form flag pipeline change implemented Co-Authored-By: Claude Fable 5 --- .../specs/2026-07-07-exam-phase-taxonomy-design.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md b/docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md index 49832d9..0ff976a 100644 --- a/docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md +++ b/docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md @@ -144,7 +144,7 @@ and is the only source that correctly classifies: - schools whose statutory age range extends to 18 on paper but which have no registered post-16 provision. -### Pipeline change (future work) +### Pipeline change (implemented 2026-07-07) 1. `stg_gias_establishments.sql`: add `"OfficialSixthForm (name)" as official_sixth_form`. @@ -152,6 +152,9 @@ and is the only source that correctly classifies: `has_sixth_form boolean` = `official_sixth_form = 'Has a sixth form'`. 3. Expose `has_sixth_form` on the school API payloads. +Implemented in `feat/gias-sixth-form-flag` — see +`docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md`. + ### Current heuristic — audit of `age_range` ~ "18" sites All must migrate to the `has_sixth_form` flag once exposed: From a524cdc591fd2af7611c29dadcd5e1e05e180676 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 13:33:25 +0100 Subject: [PATCH 08/10] fix(api): survive missing has_sixth_form column and numpy bool serialization - data_loader.load_school_data_as_dataframe now catches a ProgrammingError whose message mentions has_sixth_form (psycopg2 UndefinedColumn) and retries with a NULL-AS-has_sixth_form query variant, so the API keeps serving data (and the app.py column-fallback branch stays reachable) even before the nightly pipeline has rebuilt marts.dim_school. - utils.convert_to_native now handles numpy.bool_ so GET /api/schools/{urn} doesn't 500 once has_sixth_form is a populated bool-dtype column. - Update the now-stale comment on the app.py age-range fallback branch. Co-Authored-By: Claude Fable 5 --- backend/app.py | 5 +- backend/data_loader.py | 28 +++++++++ backend/tests/test_sixth_form_flag.py | 81 +++++++++++++++++++++++++++ backend/utils.py | 2 + 4 files changed, 115 insertions(+), 1 deletion(-) diff --git a/backend/app.py b/backend/app.py index 9b63008..7f017df 100644 --- a/backend/app.py +++ b/backend/app.py @@ -421,7 +421,10 @@ async def get_schools( if has_sixth_form in ("yes", "no"): if "has_sixth_form" in df_latest.columns: flag = df_latest["has_sixth_form"].eq(True) - else: # DB predates the pipeline re-run — fall back to age range + else: # Defensive fallback only — data_loader now always synthesizes + # has_sixth_form as NULL when the DB predates the pipeline re-run, + # so this branch shouldn't normally trigger. Falls back to age + # range if the column is somehow absent anyway. flag = df_latest["age_range"].str.contains("18", na=False) df_latest = df_latest[flag if has_sixth_form == "yes" else ~flag] diff --git a/backend/data_loader.py b/backend/data_loader.py index 6221461..9e50c73 100644 --- a/backend/data_loader.py +++ b/backend/data_loader.py @@ -3,11 +3,14 @@ Data loading module — reads from marts.* tables built by dbt. Provides efficient queries with caching. """ +import logging + import pandas as pd import numpy as np from typing import Optional, Dict, Tuple, List import requests from sqlalchemy import text +import sqlalchemy.exc from sqlalchemy.orm import Session from .config import settings @@ -215,11 +218,36 @@ _MAIN_QUERY = text(""" ORDER BY s.school_name, p.year """) +# Fallback used when marts.dim_school predates the has_sixth_form column +# (i.e. the nightly dbt pipeline hasn't rebuilt the mart yet on this DB). +# Keeps the column present as NULL so downstream code — including the +# app.py fallback branch — behaves as designed instead of KeyError-ing. +_MAIN_QUERY_NO_SIXTH_FORM = text( + str(_MAIN_QUERY).replace("s.has_sixth_form,", "NULL AS has_sixth_form,") +) +assert "NULL AS has_sixth_form" in str(_MAIN_QUERY_NO_SIXTH_FORM), ( + "expected replacement of 's.has_sixth_form,' to have taken effect" +) + def load_school_data_as_dataframe() -> pd.DataFrame: """Load all school + KS2 data as a pandas DataFrame.""" try: df = pd.read_sql(_MAIN_QUERY, engine) + except sqlalchemy.exc.ProgrammingError as exc: + if "has_sixth_form" not in str(exc): + print(f"Warning: Could not load school data from marts: {exc}") + return pd.DataFrame() + logging.getLogger(__name__).warning( + "marts.dim_school is missing has_sixth_form (pipeline hasn't " + "rebuilt the mart yet on this DB) — retrying without it: %s", + exc, + ) + try: + df = pd.read_sql(_MAIN_QUERY_NO_SIXTH_FORM, engine) + except Exception as exc2: + print(f"Warning: Could not load school data from marts: {exc2}") + return pd.DataFrame() except Exception as exc: print(f"Warning: Could not load school data from marts: {exc}") return pd.DataFrame() diff --git a/backend/tests/test_sixth_form_flag.py b/backend/tests/test_sixth_form_flag.py index fe1782a..fb7c34c 100644 --- a/backend/tests/test_sixth_form_flag.py +++ b/backend/tests/test_sixth_form_flag.py @@ -95,3 +95,84 @@ def test_detail_payload_includes_flag(client): resp = client.get("/api/schools/100002") assert resp.status_code == 200, resp.text assert resp.json()["school_info"]["has_sixth_form"] is True + + +def test_detail_payload_serializes_numpy_bool(monkeypatch): + """Once the pipeline has run, has_sixth_form is a real bool dtype column + (dbt not_null test guarantees no NULLs), so row access yields + numpy.bool_ rather than a Python bool. convert_to_native must handle it — + otherwise FastAPI's jsonable_encoder raises ValueError and the detail + endpoint 500s (C2).""" + from backend import app as app_module + + df = _schools_df() + # Drop the row with a None flag — this fixture models the post-pipeline + # state where the column is a genuine, fully-populated bool dtype. + df = df[df["has_sixth_form"].notna()].reset_index(drop=True) + df["has_sixth_form"] = df["has_sixth_form"].astype(bool) + assert df["has_sixth_form"].dtype == bool + + monkeypatch.setattr(app_module, "load_school_data", lambda: df) + monkeypatch.setattr(app_module, "get_supplementary_data", lambda db, urn: {}) + client = TestClient(app_module.app, raise_server_exceptions=False) + + resp = client.get("/api/schools/100002") + assert resp.status_code == 200, resp.text + assert resp.json()["school_info"]["has_sixth_form"] is True + + +class _FakeProgrammingError(Exception): + """Stand-in for sqlalchemy.exc.ProgrammingError wrapping psycopg2's + UndefinedColumn, without needing a real DB connection to construct one.""" + + +def test_load_school_data_survives_missing_has_sixth_form_column(monkeypatch): + """Real prod state until the nightly pipeline first rebuilds the mart: + marts.dim_school lacks has_sixth_form entirely. The first query raises + UndefinedColumn; load_school_data_as_dataframe must retry without the + column (synthesizing it as None) rather than swallow the error and + return (and then have load_school_data cache) an empty DataFrame (C1).""" + import sqlalchemy.exc + from backend import data_loader + + data_loader._df_cache = None + data_loader._df_latest_cache = None + + good_df = pd.DataFrame( + [ + { + "urn": 1, + "school_name": "Fallback School", + "school_type": "Academy", + "has_sixth_form": None, + } + ] + ) + calls = [] + + def fake_read_sql(query, con): + calls.append(query) + if len(calls) == 1: + raise sqlalchemy.exc.ProgrammingError( + "SELECT ...", + None, + Exception( + "(psycopg2.errors.UndefinedColumn) column s.has_sixth_form " + "does not exist" + ), + ) + return good_df.copy() + + monkeypatch.setattr(data_loader.pd, "read_sql", fake_read_sql) + + try: + df = data_loader.load_school_data_as_dataframe() + finally: + data_loader._df_cache = None + data_loader._df_latest_cache = None + + assert len(calls) == 2, "must retry with the no-sixth-form query variant" + assert calls[1] is data_loader._MAIN_QUERY_NO_SIXTH_FORM + assert not df.empty + assert "has_sixth_form" in df.columns + assert df["has_sixth_form"].iloc[0] is None diff --git a/backend/utils.py b/backend/utils.py index 18fb37e..01f885c 100644 --- a/backend/utils.py +++ b/backend/utils.py @@ -11,6 +11,8 @@ def convert_to_native(value: Any) -> Any: """Convert numpy types to native Python types for JSON serialization.""" if pd.isna(value): return None + if isinstance(value, np.bool_): + return bool(value) if isinstance(value, (np.integer,)): return int(value) if isinstance(value, (np.floating,)): From 4d226fd616f24f8e3663181b0774ea9aa5738a35 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 13:36:01 +0100 Subject: [PATCH 09/10] test: drop unused fake exception helper Co-Authored-By: Claude Fable 5 --- backend/tests/test_sixth_form_flag.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/backend/tests/test_sixth_form_flag.py b/backend/tests/test_sixth_form_flag.py index fb7c34c..af32799 100644 --- a/backend/tests/test_sixth_form_flag.py +++ b/backend/tests/test_sixth_form_flag.py @@ -121,11 +121,6 @@ def test_detail_payload_serializes_numpy_bool(monkeypatch): assert resp.json()["school_info"]["has_sixth_form"] is True -class _FakeProgrammingError(Exception): - """Stand-in for sqlalchemy.exc.ProgrammingError wrapping psycopg2's - UndefinedColumn, without needing a real DB connection to construct one.""" - - def test_load_school_data_survives_missing_has_sixth_form_column(monkeypatch): """Real prod state until the nightly pipeline first rebuilds the mart: marts.dim_school lacks has_sixth_form entirely. The first query raises From f1388ff5bd0af1409823a1e047b7ba84246a0f70 Mon Sep 17 00:00:00 2001 From: Tudor Date: Tue, 7 Jul 2026 14:05:10 +0100 Subject: [PATCH 10/10] fix(pipeline): normalize GIAS OfficialSixthForm comparison with lower(trim()) Matches the phase derivation's guard against casing/whitespace variants in raw GIAS data; an unmatched variant previously fell through silently to the statutory-age fallback. Co-Authored-By: Claude Fable 5 --- pipeline/transform/models/marts/dim_school.sql | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/pipeline/transform/models/marts/dim_school.sql b/pipeline/transform/models/marts/dim_school.sql index a09a016..0de54fa 100644 --- a/pipeline/transform/models/marts/dim_school.sql +++ b/pipeline/transform/models/marts/dim_school.sql @@ -55,9 +55,12 @@ select -- Authoritative sixth-form flag (spec §3): GIAS OfficialSixthForm. -- "Not applicable" (nurseries, primaries, PRUs) => false. Blank GIAS -- value (rare, new establishments) falls back to the statutory age range. + -- lower(trim()) guards against casing/whitespace variants in raw GIAS + -- data, same as the phase derivation above — an unmatched variant would + -- otherwise silently fall through to the age-range fallback. case - when s.official_sixth_form = 'Has a sixth form' then true - when s.official_sixth_form in ('Does not have a sixth form', 'Not applicable') then false + when lower(trim(s.official_sixth_form)) = 'has a sixth form' then true + when lower(trim(s.official_sixth_form)) in ('does not have a sixth form', 'not applicable') then false else coalesce(s.statutory_high_age >= 18, false) end as has_sixth_form, s.capacity,