Replaces the fragile age_range contains "18" sixth-form heuristic with the DfE's authoritative GIAS OfficialSixthForm flag, end to end: Singer tap → dbt staging → marts.dim_school.has_sixth_form → API payloads/filter → Next.js badge, note, row tag and filter labels.
16–19 sixth-form colleges (e.g. age range 16-19) are now correctly classified as having a sixth form; 11–18-on-paper schools without registered post-16 provision no longer are.
Filter labels are now "With sixth form" / "Without sixth form" (age-range parentheticals dropped).
Compatibility notes
API contract unchanged: GET /api/schools?has_sixth_form=yes|no, same values.
The DB column only exists after the next nightly pipeline run. Until then the backend detects the missing column and retries with NULL AS has_sixth_form (new regression test), the filter falls back to the old age-range heuristic, and the UI treats a missing flag as "no sixth form" — no outage window.
convert_to_native now handles numpy.bool_ so the detail endpoint serializes the real boolean column once populated (new regression test).
No e2e journey change: no journey exercises the sixth-form filter and the contract is unchanged.
Spec deviation noted: blank-GIAS fallback rows are not logged per-URN (dbt has no per-row logging); the fallback is statutory_high_age >= 18.
Tests: 8/8 backend pytest, 36/36 Jest, tsc --noEmit clean.
Replaces the fragile `age_range contains "18"` sixth-form heuristic with the DfE's authoritative GIAS `OfficialSixthForm` flag, end to end: Singer tap → dbt staging → `marts.dim_school.has_sixth_form` → API payloads/filter → Next.js badge, note, row tag and filter labels.
Spec: `docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md` (§3) · Plan: `docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md`
**Key behaviour changes**
- 16–19 sixth-form colleges (e.g. age range `16-19`) are now correctly classified as having a sixth form; 11–18-on-paper schools without registered post-16 provision no longer are.
- Filter labels are now "With sixth form" / "Without sixth form" (age-range parentheticals dropped).
**Compatibility notes**
1. API contract unchanged: `GET /api/schools?has_sixth_form=yes|no`, same values.
2. The DB column only exists after the next nightly pipeline run. Until then the backend detects the missing column and retries with `NULL AS has_sixth_form` (new regression test), the filter falls back to the old age-range heuristic, and the UI treats a missing flag as "no sixth form" — no outage window.
3. `convert_to_native` now handles `numpy.bool_` so the detail endpoint serializes the real boolean column once populated (new regression test).
4. No e2e journey change: no journey exercises the sixth-form filter and the contract is unchanged.
5. Spec deviation noted: blank-GIAS fallback rows are not logged per-URN (dbt has no per-row logging); the fallback is `statutory_high_age >= 18`.
Tests: 8/8 backend pytest, 36/36 Jest, `tsc --noEmit` clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
- data_loader.load_school_data_as_dataframe now catches a ProgrammingError
whose message mentions has_sixth_form (psycopg2 UndefinedColumn) and
retries with a NULL-AS-has_sixth_form query variant, so the API keeps
serving data (and the app.py column-fallback branch stays reachable)
even before the nightly pipeline has rebuilt marts.dim_school.
- utils.convert_to_native now handles numpy.bool_ so GET /api/schools/{urn}
doesn't 500 once has_sixth_form is a populated bool-dtype column.
- Update the now-stale comment on the app.py age-range fallback branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This PR replaces the fragile 'age_range contains 18' sixth-form heuristic with an authoritative GIAS OfficialSixthForm flag, threading it through the Singer tap, dbt staging/mart models, backend query/schema/filter logic, and the Next.js UI, with a defensive fallback (and tests) for the transition window before the pipeline has rebuilt the mart. The change is well tested and handles the missing-column/NULL/numpy-bool edge cases carefully; only minor robustness concerns remain.
🟡 Minor
pipeline/transform/models/marts/dim_school.sql: The has_sixth_form CASE does exact, case-sensitive string equality against 'Has a sixth form' / 'Does not have a sixth form' / 'Not applicable', with no lower()/trim() normalization — unlike the phase derivation a few lines above in the same model, which explicitly does lower(trim(s.phase)) not in ('not applicable', ...) to guard against casing variants seen in real GIAS data. If GIAS's raw value ever differs in case/whitespace from these literals, affected schools silently fall through to the statutory-age-range fallback instead of being correctly classified, with no error or log — a quiet data-quality regression rather than a crash.
backend/data_loader.py: The no-sixth-form fallback query is generated by string-replacing 's.has_sixth_form,' inside the compiled text of _MAIN_QUERY, guarded only by a module-level assert that runs at import time. This is fragile: any future edit to _MAIN_QUERY's formatting/whitespace/column ordering near that line that isn't an exact match would make the assert fail on import, taking down the entire backend at startup rather than failing gracefully for just this feature. A dedicated fallback query string (or a parameterized column list) would be more robust than deriving one query from another via text substitution.
## 🤖 AI Code Review (Claude Code)
This PR replaces the fragile 'age_range contains 18' sixth-form heuristic with an authoritative GIAS OfficialSixthForm flag, threading it through the Singer tap, dbt staging/mart models, backend query/schema/filter logic, and the Next.js UI, with a defensive fallback (and tests) for the transition window before the pipeline has rebuilt the mart. The change is well tested and handles the missing-column/NULL/numpy-bool edge cases carefully; only minor robustness concerns remain.
### 🟡 Minor
- **pipeline/transform/models/marts/dim_school.sql**: The has_sixth_form CASE does exact, case-sensitive string equality against 'Has a sixth form' / 'Does not have a sixth form' / 'Not applicable', with no lower()/trim() normalization — unlike the phase derivation a few lines above in the same model, which explicitly does `lower(trim(s.phase)) not in ('not applicable', ...)` to guard against casing variants seen in real GIAS data. If GIAS's raw value ever differs in case/whitespace from these literals, affected schools silently fall through to the statutory-age-range fallback instead of being correctly classified, with no error or log — a quiet data-quality regression rather than a crash.
- **backend/data_loader.py**: The no-sixth-form fallback query is generated by string-replacing 's.has_sixth_form,' inside the compiled text of _MAIN_QUERY, guarded only by a module-level `assert` that runs at import time. This is fragile: any future edit to _MAIN_QUERY's formatting/whitespace/column ordering near that line that isn't an exact match would make the assert fail on import, taking down the entire backend at startup rather than failing gracefully for just this feature. A dedicated fallback query string (or a parameterized column list) would be more robust than deriving one query from another via text substitution.
Matches the phase derivation's guard against casing/whitespace variants in
raw GIAS data; an unmatched variant previously fell through silently to the
statutory-age fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This PR replaces the fragile age_range-contains-"18" sixth-form heuristic with an authoritative GIAS OfficialSixthForm flag, threading it through the dbt staging/mart models, backend SQLAlchemy model/query/API payloads, and frontend components. It includes a defensive fallback query for when the DB predates the pipeline rebuild, a numpy bool JSON-serialization fix, and thorough backend/frontend regression tests covering the exact edge cases the change is meant to fix (16-19 colleges, NULL flags, missing columns).
✅ No issues found.
## 🤖 AI Code Review (Claude Code)
This PR replaces the fragile age_range-contains-"18" sixth-form heuristic with an authoritative GIAS OfficialSixthForm flag, threading it through the dbt staging/mart models, backend SQLAlchemy model/query/API payloads, and frontend components. It includes a defensive fallback query for when the DB predates the pipeline rebuild, a numpy bool JSON-serialization fix, and thorough backend/frontend regression tests covering the exact edge cases the change is meant to fix (16-19 colleges, NULL flags, missing columns).
✅ No issues found.
tudor
merged commit 45c68b60b4 into main2026-07-07 13:48:12 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Replaces the fragile
age_range contains "18"sixth-form heuristic with the DfE's authoritative GIASOfficialSixthFormflag, end to end: Singer tap → dbt staging →marts.dim_school.has_sixth_form→ API payloads/filter → Next.js badge, note, row tag and filter labels.Spec:
docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md(§3) · Plan:docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.mdKey behaviour changes
16-19) are now correctly classified as having a sixth form; 11–18-on-paper schools without registered post-16 provision no longer are.Compatibility notes
GET /api/schools?has_sixth_form=yes|no, same values.NULL AS has_sixth_form(new regression test), the filter falls back to the old age-range heuristic, and the UI treats a missing flag as "no sixth form" — no outage window.convert_to_nativenow handlesnumpy.bool_so the detail endpoint serializes the real boolean column once populated (new regression test).statutory_high_age >= 18.Tests: 8/8 backend pytest, 36/36 Jest,
tsc --noEmitclean.🤖 Generated with Claude Code
- data_loader.load_school_data_as_dataframe now catches a ProgrammingError whose message mentions has_sixth_form (psycopg2 UndefinedColumn) and retries with a NULL-AS-has_sixth_form query variant, so the API keeps serving data (and the app.py column-fallback branch stays reachable) even before the nightly pipeline has rebuilt marts.dim_school. - utils.convert_to_native now handles numpy.bool_ so GET /api/schools/{urn} doesn't 500 once has_sixth_form is a populated bool-dtype column. - Update the now-stale comment on the app.py age-range fallback branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>🤖 AI Code Review (Claude Code)
This PR replaces the fragile 'age_range contains 18' sixth-form heuristic with an authoritative GIAS OfficialSixthForm flag, threading it through the Singer tap, dbt staging/mart models, backend query/schema/filter logic, and the Next.js UI, with a defensive fallback (and tests) for the transition window before the pipeline has rebuilt the mart. The change is well tested and handles the missing-column/NULL/numpy-bool edge cases carefully; only minor robustness concerns remain.
🟡 Minor
lower(trim(s.phase)) not in ('not applicable', ...)to guard against casing variants seen in real GIAS data. If GIAS's raw value ever differs in case/whitespace from these literals, affected schools silently fall through to the statutory-age-range fallback instead of being correctly classified, with no error or log — a quiet data-quality regression rather than a crash.assertthat runs at import time. This is fragile: any future edit to _MAIN_QUERY's formatting/whitespace/column ordering near that line that isn't an exact match would make the assert fail on import, taking down the entire backend at startup rather than failing gracefully for just this feature. A dedicated fallback query string (or a parameterized column list) would be more robust than deriving one query from another via text substitution.🤖 AI Code Review (Claude Code)
This PR replaces the fragile age_range-contains-"18" sixth-form heuristic with an authoritative GIAS OfficialSixthForm flag, threading it through the dbt staging/mart models, backend SQLAlchemy model/query/API payloads, and frontend components. It includes a defensive fallback query for when the DB predates the pipeline rebuild, a numpy bool JSON-serialization fix, and thorough backend/frontend regression tests covering the exact edge cases the change is meant to fix (16-19 colleges, NULL flags, missing columns).
✅ No issues found.