# GIAS OfficialSixthForm Flag Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Ingest GIAS's authoritative `OfficialSixthForm` flag into `marts.dim_school.has_sixth_form` and replace every `age_range contains "18"` heuristic in the backend and frontend with it. **Architecture:** Data flows tap → raw → dbt staging → dbt mart → backend SQL → API payload → Next.js components. The GIAS Singer tap must declare the new CSV column (target-postgres only persists declared columns); the dbt staging model renames it; `dim_school` derives a boolean (with a statutory-age fallback for blank GIAS values); the backend exposes it on list + detail payloads and uses it for the `has_sixth_form=yes|no` filter; the frontend badge/note/filter-labels switch from the age-range substring check to the flag. **Tech Stack:** Singer SDK (tap), dbt (Postgres), FastAPI + pandas, Next.js + TypeScript, pytest, Jest/RTL. **Spec:** `docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md` §3. ## Global Constraints - A school **has a sixth form** iff GIAS `OfficialSixthForm (name)` = `"Has a sixth form"`. `"Does not have a sixth form"` and `"Not applicable"` → false. Blank/NULL (rare) → fall back to `statutory_high_age >= 18`. - The public API filter parameter stays `has_sixth_form=yes|no` (unchanged contract). - Filter dropdown labels must drop the age-range parentheticals: "With sixth form" / "Without sixth form" (sixth form ≠ age range). - Never push to `main`; work stays on branch `feat/gias-sixth-form-flag` (create from `docs/exam-phase-taxonomy` so the spec is included, or from `main` if that branch has merged). - The dbt models cannot be run locally (no pipeline DB); dbt changes are verified by review + `python -c` schema asserts + existing CI. Do NOT attempt to start a local server. - The backend marts tables are dbt `table` materializations — rebuilt on every pipeline run, so **no ALTER TABLE migration is needed** for `marts.dim_school`. - Deployment ordering: the tap must run before dbt on the first pipeline run after deploy (this is already the DAG order: extract → transform). Until that run happens, `has_sixth_form` is absent from the DB; the backend must treat a missing column as "flag false / fallback", never crash. --- ### Task 1: Ingest `OfficialSixthForm (name)` — tap schema + dbt staging **Files:** - Modify: `pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py:31-66` (Singer schema) - Modify: `pipeline/transform/models/staging/stg_gias_establishments.sql` (add renamed column) **Interfaces:** - Produces: raw column `"OfficialSixthForm (name)"` in `raw.gias_establishments`; staging column `official_sixth_form` (text: `Has a sixth form` / `Does not have a sixth form` / `Not applicable` / NULL) consumed by Task 2. - [ ] **Step 1: Add the property to the Singer schema** In `tap.py`, inside `GIASEstablishmentsStream.schema = th.PropertiesList(...)`, add after the `th.Property("PhaseOfEducation (name)", th.StringType),` line: ```python th.Property("OfficialSixthForm (name)", th.StringType), ``` - [ ] **Step 2: Verify the tap module still imports and declares the column** Run: ```bash cd /Users/tudor/projects/school_compare/pipeline/plugins/extractors/tap-uk-gias && \ python3 -c " import ast, sys src = open('tap_uk_gias/tap.py').read() ast.parse(src) assert '\"OfficialSixthForm (name)\"' in src.replace(\"'\", '\"') print('OK: tap declares OfficialSixthForm (name)') " ``` Expected: `OK: tap declares OfficialSixthForm (name)` (Uses `ast.parse` instead of importing because `singer_sdk` is not installed locally.) - [ ] **Step 3: Add the column to the staging model** In `stg_gias_establishments.sql`, in the `renamed` CTE, add after the `"PhaseOfEducation (name)" as phase,` line: ```sql nullif(trim("OfficialSixthForm (name)"), '') as official_sixth_form, ``` - [ ] **Step 4: Sanity-check the SQL edit** Run: ```bash grep -n "official_sixth_form" /Users/tudor/projects/school_compare/pipeline/transform/models/staging/stg_gias_establishments.sql ``` Expected: one line showing the new column inside the `renamed` CTE (before `from source`). - [ ] **Step 5: Commit** ```bash git add pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py pipeline/transform/models/staging/stg_gias_establishments.sql git commit -m "feat(pipeline): ingest GIAS OfficialSixthForm into staging Co-Authored-By: Claude Fable 5 " ``` --- ### Task 2: Derive `dim_school.has_sixth_form` (dbt mart + schema tests + SQLAlchemy model) **Files:** - Modify: `pipeline/transform/models/marts/dim_school.sql` (add derived column) - Modify: `pipeline/transform/models/marts/_marts_schema.yml` (document + test the column) - Modify: `backend/models.py:13-38` (`DimSchool` — add column) **Interfaces:** - Consumes: `official_sixth_form` text column from Task 1's staging model. - Produces: `marts.dim_school.has_sixth_form boolean not null`, and `DimSchool.has_sixth_form = Column(Boolean)` for the backend. Task 3 selects it as `s.has_sixth_form`. - [ ] **Step 1: Add the derived column to `dim_school.sql`** In the `select`, add after the `s.age_range` line (`s.statutory_low_age || '-' || s.statutory_high_age as age_range,`): ```sql -- Authoritative sixth-form flag (spec §3): GIAS OfficialSixthForm. -- "Not applicable" (nurseries, primaries, PRUs) => false. Blank GIAS -- value (rare, new establishments) falls back to the statutory age range. case when s.official_sixth_form = 'Has a sixth form' then true when s.official_sixth_form in ('Does not have a sixth form', 'Not applicable') then false else coalesce(s.statutory_high_age >= 18, false) end as has_sixth_form, ``` - [ ] **Step 2: Add schema documentation + tests in `_marts_schema.yml`** Under `- name: dim_school` → `columns:`, add after the `phase` column block: ```yaml - name: has_sixth_form description: > Authoritative sixth-form flag from GIAS OfficialSixthForm. "Has a sixth form" => true; "Does not have a sixth form" and "Not applicable" => false; blank GIAS value falls back to statutory_high_age >= 18. Replaces the age_range-contains-"18" heuristic (spec 2026-07-07 §3). tests: - not_null - accepted_values: values: [true, false] ``` - [ ] **Step 3: Add the column to the `DimSchool` SQLAlchemy model** In `backend/models.py`, in `class DimSchool`, add after `age_range = Column(String(20))`: ```python has_sixth_form = Column(Boolean) ``` - [ ] **Step 4: Verify SQL/YAML/Python all parse** Run: ```bash cd /Users/tudor/projects/school_compare && \ python3 -c " import yaml y = yaml.safe_load(open('pipeline/transform/models/marts/_marts_schema.yml')) dim = [m for m in y['models'] if m['name'] == 'dim_school'][0] cols = [c['name'] for c in dim['columns']] assert 'has_sixth_form' in cols, cols print('OK: schema yml documents has_sixth_form') " && \ grep -c "has_sixth_form" pipeline/transform/models/marts/dim_school.sql && \ python3 -c "import ast; ast.parse(open('backend/models.py').read()); print('OK: models.py parses')" ``` Expected: `OK: schema yml documents has_sixth_form`, grep count `>= 1`, `OK: models.py parses`. - [ ] **Step 5: Commit** ```bash git add pipeline/transform/models/marts/dim_school.sql pipeline/transform/models/marts/_marts_schema.yml backend/models.py git commit -m "feat(pipeline): derive dim_school.has_sixth_form from GIAS flag Co-Authored-By: Claude Fable 5 " ``` --- ### Task 3: Backend — expose `has_sixth_form` and replace the filter heuristic **Files:** - Modify: `backend/data_loader.py:117-215` (`_MAIN_QUERY` — select the column) - Modify: `backend/schemas.py:536-553` (`SCHOOL_COLUMNS` — include in list payloads) - Modify: `backend/app.py:419-422` (filter) and `backend/app.py:589-610` (detail `school_info`) - Test: `backend/tests/test_sixth_form_flag.py` (new) **Interfaces:** - Consumes: `marts.dim_school.has_sixth_form` (Task 2). - Produces: `has_sixth_form: bool | null` field on `GET /api/schools` items and on `GET /api/schools/{urn}` → `school_info`. Filter `GET /api/schools?has_sixth_form=yes|no` now driven by the flag. Frontend (Task 4) reads `school.has_sixth_form`. - [ ] **Step 1: Write the failing tests** Create `backend/tests/test_sixth_form_flag.py`: ```python """Tests for the GIAS-driven has_sixth_form flag (spec 2026-07-07 §3). The filter and payloads must use dim_school.has_sixth_form, not the old age_range-contains-"18" substring heuristic. The key regression case is a 16-19 sixth-form college: flag true, but "16-19" contains no "18". """ import numpy as np import pandas as pd import pytest from fastapi.testclient import TestClient def _schools_df() -> pd.DataFrame: """Latest-year snapshot rows as produced by load_latest_school_data.""" base = { "local_authority": "Testshire", "school_type": "Academy", "phase": "Secondary", "address": "1 Test Street", "town": "Testtown", "postcode": "TS1 1AA", "religious_denomination": None, "gender": "Mixed", "admissions_policy": None, "ofsted_grade": np.nan, "ofsted_date": None, "ofsted_framework": None, "latitude": 51.5, "longitude": -0.1, "year": 202425, "total_pupils": 1000, "rwm_expected_pct": np.nan, "attainment_8_score": 50.0, } return pd.DataFrame( [ # 11-18 school WITH a registered sixth form {**base, "urn": 100001, "school_name": "Alpha High", "age_range": "11-18", "has_sixth_form": True}, # 16-19 college: old heuristic said NO ("16-19" has no "18"), # GIAS flag says YES — must appear in the yes-filter results {**base, "urn": 100002, "school_name": "Beta Sixth Form College", "age_range": "16-19", "has_sixth_form": True}, # 11-18 age range on paper but NO registered sixth form: # old heuristic said YES, GIAS flag says NO {**base, "urn": 100003, "school_name": "Gamma Academy", "age_range": "11-18", "has_sixth_form": False}, # Missing flag (pipeline not yet re-run) — must not crash, # must not match the yes-filter {**base, "urn": 100004, "school_name": "Delta School", "age_range": "11-16", "has_sixth_form": None}, ] ) @pytest.fixture() def client(monkeypatch): from backend import app as app_module monkeypatch.setattr(app_module, "load_latest_school_data", _schools_df) monkeypatch.setattr(app_module, "load_school_data", _schools_df) monkeypatch.setattr(app_module, "get_supplementary_data", lambda db, urn: {}) return TestClient(app_module.app, raise_server_exceptions=False) def _urns(resp): return sorted(s["urn"] for s in resp.json()["schools"]) def test_filter_yes_uses_flag_not_age_range(client): resp = client.get("/api/schools?has_sixth_form=yes") assert resp.status_code == 200, resp.text # 16-19 college included; 11-18-without-sixth-form excluded assert _urns(resp) == [100001, 100002] def test_filter_no_uses_flag_not_age_range(client): resp = client.get("/api/schools?has_sixth_form=no") assert resp.status_code == 200, resp.text # Gamma (flag false) and Delta (flag missing => not true) assert _urns(resp) == [100003, 100004] def test_list_payload_includes_flag(client): resp = client.get("/api/schools") assert resp.status_code == 200, resp.text by_urn = {s["urn"]: s for s in resp.json()["schools"]} assert by_urn[100002]["has_sixth_form"] is True assert by_urn[100003]["has_sixth_form"] is False assert by_urn[100004]["has_sixth_form"] is None def test_detail_payload_includes_flag(client): resp = client.get("/api/schools/100002") assert resp.status_code == 200, resp.text assert resp.json()["school_info"]["has_sixth_form"] is True ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests/test_sixth_form_flag.py -v` Expected: FAIL — `test_filter_yes_uses_flag_not_age_range` asserts `[100001, 100002]` but the age-range heuristic returns `[100001, 100003]`; the payload tests fail with `KeyError: 'has_sixth_form'`. - [ ] **Step 3: Select the column in `_MAIN_QUERY`** In `backend/data_loader.py`, in `_MAIN_QUERY`, add after `s.age_range,`: ```sql s.has_sixth_form, ``` - [ ] **Step 4: Include it in list payloads** In `backend/schemas.py`, in `SCHOOL_COLUMNS`, add after `"age_range",`: ```python "has_sixth_form", ``` (`app.py` builds list responses from `SCHOOL_COLUMNS ∩ df.columns`, so a DB that predates the pipeline re-run simply omits the field — no crash.) - [ ] **Step 5: Replace the filter heuristic in `app.py`** Replace lines 419-422: ```python if has_sixth_form == "yes": df_latest = df_latest[df_latest["age_range"].str.contains("18", na=False)] elif has_sixth_form == "no": df_latest = df_latest[~df_latest["age_range"].str.contains("18", na=False)] ``` with: ```python # GIAS OfficialSixthForm flag (dim_school.has_sixth_form). NULL (flag not # yet populated by the pipeline) is treated as "no sixth form". if has_sixth_form in ("yes", "no"): if "has_sixth_form" in df_latest.columns: flag = df_latest["has_sixth_form"].eq(True) else: # DB predates the pipeline re-run — fall back to age range flag = df_latest["age_range"].str.contains("18", na=False) df_latest = df_latest[flag if has_sixth_form == "yes" else ~flag] ``` - [ ] **Step 6: Add the flag to the detail payload** In `backend/app.py` `school_info` dict (line ~598), add after `"age_range": latest.get("age_range", ""),`: ```python "has_sixth_form": latest.get("has_sixth_form"), ``` (`convert_to_native` already maps NaN/None → null and numpy bools → bool.) - [ ] **Step 7: Run the new tests** Run: `cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests/test_sixth_form_flag.py -v` Expected: 4 passed. - [ ] **Step 8: Run the full backend suite** Run: `cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests -v` Expected: all pass (the pre-existing `test_school_details.py` df has no `has_sixth_form` column — `latest.get()` returns None, serialized as null). - [ ] **Step 9: Commit** ```bash git add backend/data_loader.py backend/schemas.py backend/app.py backend/tests/test_sixth_form_flag.py git commit -m "feat(api): drive has_sixth_form filter and payloads from GIAS flag Co-Authored-By: Claude Fable 5 " ``` --- ### Task 4: Frontend — badge, note, row tag, and filter labels use the flag **Files:** - Modify: `nextjs-app/lib/types.ts:10-30` (`School` interface) - Modify: `nextjs-app/components/SecondarySchoolDetailView.tsx:101` (badge + coming-soon note) - Modify: `nextjs-app/components/SecondarySchoolRow.tsx:25-27` (row tag) - Modify: `nextjs-app/components/FilterBar.tsx:370-372` (labels only — param name unchanged) - Test: `nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx` (new) **Interfaces:** - Consumes: `has_sixth_form: boolean | null` on both list items and `school_info` (Task 3; both are typed as `School`). - Produces: no new exports — behavior change only. - [ ] **Step 1: Write the failing test** Create `nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx`: ```tsx /** * SecondarySchoolRow — sixth-form tag must come from the GIAS * has_sixth_form flag, not the age_range-contains-"18" heuristic. */ import '@testing-library/jest-dom'; import { render, screen } from '@testing-library/react'; import { SecondarySchoolRow } from '@/components/SecondarySchoolRow'; import type { School } from '@/lib/types'; const base = { urn: 100002, school_name: 'Beta Sixth Form College', local_authority: 'Testshire', school_type: 'Academy', phase: 'Secondary', gender: 'Mixed', attainment_8_score: 50.0, } as unknown as School; describe('SecondarySchoolRow sixth-form tag', () => { it('shows the tag for a 16-19 college with the GIAS flag set', () => { render( , ); expect(screen.getByText('Sixth form')).toBeInTheDocument(); }); it('hides the tag for an 11-18 school without a registered sixth form', () => { render( , ); expect(screen.queryByText('Sixth form')).not.toBeInTheDocument(); }); it('hides the tag when the flag is missing (pipeline not yet re-run)', () => { render( , ); expect(screen.queryByText('Sixth form')).not.toBeInTheDocument(); }); }); ``` - [ ] **Step 2: Run it to verify it fails** Run: `cd /Users/tudor/projects/school_compare/nextjs-app && npx jest __tests__/components/SecondarySchoolRow.test.tsx` Expected: FAIL — first test can't find "Sixth form" ("16-19" fails the substring check), second test finds an unexpected "Sixth form" tag. (If TS complains that `has_sixth_form` is not on `School`, that is the same failure — proceed.) - [ ] **Step 3: Add the field to the `School` type** In `nextjs-app/lib/types.ts`, in `export interface School`, add after `age_range: string | null;`: ```ts has_sixth_form?: boolean | null; ``` - [ ] **Step 4: Switch `SecondarySchoolRow` to the flag** Replace the helper at `SecondarySchoolRow.tsx:25-27`: ```ts function hasSixthForm(school: School): boolean { return school.age_range?.includes('18') ?? false; } ``` with: ```ts function hasSixthForm(school: School): boolean { // GIAS OfficialSixthForm flag; missing (pipeline not yet re-run) => false. return school.has_sixth_form ?? false; } ``` - [ ] **Step 5: Switch `SecondarySchoolDetailView` to the flag** Replace line 101: ```ts const hasSixthForm = schoolInfo.age_range?.includes('18') ?? false; ``` with: ```ts // GIAS OfficialSixthForm flag; missing (pipeline not yet re-run) => false. const hasSixthForm = schoolInfo.has_sixth_form ?? false; ``` (This drives both the header "Sixth form" badge at line ~230 and the "Post-16 destination data coming soon" note at line ~715 — no changes needed there.) - [ ] **Step 6: Fix the filter labels in `FilterBar.tsx`** Replace: ```tsx ``` with: ```tsx ``` - [ ] **Step 7: Run the new test and verify it passes** Run: `cd /Users/tudor/projects/school_compare/nextjs-app && npx jest __tests__/components/SecondarySchoolRow.test.tsx` Expected: 3 passed. - [ ] **Step 8: Run the full frontend checks** Run: `cd /Users/tudor/projects/school_compare/nextjs-app && npx tsc --noEmit && npx jest` Expected: typecheck clean, all Jest suites pass. - [ ] **Step 9: Verify no heuristic remains** Run: ```bash grep -rn "includes('18')\|contains(\"18\")" /Users/tudor/projects/school_compare/nextjs-app/components /Users/tudor/projects/school_compare/backend --include="*.tsx" --include="*.ts" --include="*.py" | grep -v test ``` Expected: only the documented fallback inside `app.py` (DB-predates-pipeline branch); no other hits. - [ ] **Step 10: Commit** ```bash git add nextjs-app/lib/types.ts nextjs-app/components/SecondarySchoolRow.tsx nextjs-app/components/SecondarySchoolDetailView.tsx nextjs-app/components/FilterBar.tsx nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx git commit -m "feat(ui): sixth-form badge, note and filter labels use GIAS flag Co-Authored-By: Claude Fable 5 " ``` --- ### Task 5: Update the spec status + PR **Files:** - Modify: `docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md` (§3 "Pipeline change (future work)" → implemented) **Interfaces:** - Consumes: everything above merged into the branch. - Produces: PR ready for review; e2e journeys are the promotion gate (no journey currently exercises the sixth-form filter, and the API contract is unchanged, so no e2e change is required — state this in the PR body). - [ ] **Step 1: Mark spec §3 pipeline change as implemented** In the spec, change the §3 heading `### Pipeline change (future work)` to `### Pipeline change (implemented 2026-07-07)` and append one line at the end of that subsection: ```markdown Implemented in `feat/gias-sixth-form-flag` — see `docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md`. ``` - [ ] **Step 2: Commit** ```bash git add docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md git commit -m "docs: mark sixth-form flag pipeline change implemented Co-Authored-By: Claude Fable 5 " ``` - [ ] **Step 3: Push and open the PR (Gitea)** Push the branch, then create the PR against `main` using the Gitea API via the git credential helper (token-header auth 401s on this Gitea; basic auth from `git credential fill` works): ```bash git push -u origin feat/gias-sixth-form-flag ``` PR title: `feat: drive sixth-form separation from GIAS OfficialSixthForm flag` PR body must note: (1) API contract unchanged (`has_sixth_form=yes|no`), (2) flag is NULL until the next pipeline run — backend and frontend degrade to "no sixth form" / age-range fallback, (3) no e2e journey change needed, and end with the standard generation footer. - [ ] **Step 4: Verify CI passes** Watch the PR checks (typecheck, tests, builds, AI review). All must pass before merge; merging deploys to staging automatically.