school_compare/docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md

GIAS OfficialSixthForm Flag Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Ingest GIAS's authoritative OfficialSixthForm flag into marts.dim_school.has_sixth_form and replace every age_range contains "18" heuristic in the backend and frontend with it.

Architecture: Data flows tap → raw → dbt staging → dbt mart → backend SQL → API payload → Next.js components. The GIAS Singer tap must declare the new CSV column (target-postgres only persists declared columns); the dbt staging model renames it; dim_school derives a boolean (with a statutory-age fallback for blank GIAS values); the backend exposes it on list + detail payloads and uses it for the has_sixth_form=yes|no filter; the frontend badge/note/filter-labels switch from the age-range substring check to the flag.

Tech Stack: Singer SDK (tap), dbt (Postgres), FastAPI + pandas, Next.js + TypeScript, pytest, Jest/RTL.

Spec: docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md §3.

Global Constraints

• A school has a sixth form iff GIAS OfficialSixthForm (name) = "Has a sixth form". "Does not have a sixth form" and "Not applicable" → false. Blank/NULL (rare) → fall back to statutory_high_age >= 18.
• The public API filter parameter stays has_sixth_form=yes|no (unchanged contract).
• Filter dropdown labels must drop the age-range parentheticals: "With sixth form" / "Without sixth form" (sixth form ≠ age range).
• Never push to main; work stays on branch feat/gias-sixth-form-flag (create from docs/exam-phase-taxonomy so the spec is included, or from main if that branch has merged).
• The dbt models cannot be run locally (no pipeline DB); dbt changes are verified by review + python -c schema asserts + existing CI. Do NOT attempt to start a local server.
• The backend marts tables are dbt table materializations — rebuilt on every pipeline run, so no ALTER TABLE migration is needed for marts.dim_school.
• Deployment ordering: the tap must run before dbt on the first pipeline run after deploy (this is already the DAG order: extract → transform). Until that run happens, has_sixth_form is absent from the DB; the backend must treat a missing column as "flag false / fallback", never crash.

---

Task 1: Ingest OfficialSixthForm (name) — tap schema + dbt staging

Files:

• Modify: pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py:31-66 (Singer schema)
• Modify: pipeline/transform/models/staging/stg_gias_establishments.sql (add renamed column)

Interfaces:

• Produces: raw column "OfficialSixthForm (name)" in raw.gias_establishments; staging column official_sixth_form (text: Has a sixth form / Does not have a sixth form / Not applicable / NULL) consumed by Task 2.

• Step 1: Add the property to the Singer schema

In tap.py, inside GIASEstablishmentsStream.schema = th.PropertiesList(...), add after the th.Property("PhaseOfEducation (name)", th.StringType), line:

        th.Property("OfficialSixthForm (name)", th.StringType),

• Step 2: Verify the tap module still imports and declares the column

Run:

cd /Users/tudor/projects/school_compare/pipeline/plugins/extractors/tap-uk-gias && \
python3 -c "
import ast, sys
src = open('tap_uk_gias/tap.py').read()
ast.parse(src)
assert '\"OfficialSixthForm (name)\"' in src.replace(\"'\", '\"')
print('OK: tap declares OfficialSixthForm (name)')
"

Expected: OK: tap declares OfficialSixthForm (name) (Uses ast.parse instead of importing because singer_sdk is not installed locally.)

• Step 3: Add the column to the staging model

In stg_gias_establishments.sql, in the renamed CTE, add after the "PhaseOfEducation (name)" as phase, line:

        nullif(trim("OfficialSixthForm (name)"), '')    as official_sixth_form,

• Step 4: Sanity-check the SQL edit

Run:

grep -n "official_sixth_form" /Users/tudor/projects/school_compare/pipeline/transform/models/staging/stg_gias_establishments.sql

Expected: one line showing the new column inside the renamed CTE (before from source).

• Step 5: Commit

git add pipeline/plugins/extractors/tap-uk-gias/tap_uk_gias/tap.py pipeline/transform/models/staging/stg_gias_establishments.sql
git commit -m "feat(pipeline): ingest GIAS OfficialSixthForm into staging

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

---

Task 2: Derive dim_school.has_sixth_form (dbt mart + schema tests + SQLAlchemy model)

Files:

• Modify: pipeline/transform/models/marts/dim_school.sql (add derived column)
• Modify: pipeline/transform/models/marts/_marts_schema.yml (document + test the column)
• Modify: backend/models.py:13-38 (DimSchool — add column)

Interfaces:

• Consumes: official_sixth_form text column from Task 1's staging model.

• Produces: marts.dim_school.has_sixth_form boolean not null, and DimSchool.has_sixth_form = Column(Boolean) for the backend. Task 3 selects it as s.has_sixth_form.

• Step 1: Add the derived column to dim_school.sql

In the select, add after the s.age_range line (s.statutory_low_age || '-' || s.statutory_high_age as age_range,):

    -- Authoritative sixth-form flag (spec §3): GIAS OfficialSixthForm.
    -- "Not applicable" (nurseries, primaries, PRUs) => false. Blank GIAS
    -- value (rare, new establishments) falls back to the statutory age range.
    case
        when s.official_sixth_form = 'Has a sixth form' then true
        when s.official_sixth_form in ('Does not have a sixth form', 'Not applicable') then false
        else coalesce(s.statutory_high_age >= 18, false)
    end as has_sixth_form,

• Step 2: Add schema documentation + tests in _marts_schema.yml

Under - name: dim_school → columns:, add after the phase column block:

      - name: has_sixth_form
        description: >
          Authoritative sixth-form flag from GIAS OfficialSixthForm.
          "Has a sixth form" => true; "Does not have a sixth form" and
          "Not applicable" => false; blank GIAS value falls back to
          statutory_high_age >= 18. Replaces the age_range-contains-"18"
          heuristic (spec 2026-07-07 §3).
        tests:
          - not_null
          - accepted_values:
              values: [true, false]

• Step 3: Add the column to the DimSchool SQLAlchemy model

In backend/models.py, in class DimSchool, add after age_range = Column(String(20)):

    has_sixth_form = Column(Boolean)

• Step 4: Verify SQL/YAML/Python all parse

Run:

cd /Users/tudor/projects/school_compare && \
python3 -c "
import yaml
y = yaml.safe_load(open('pipeline/transform/models/marts/_marts_schema.yml'))
dim = [m for m in y['models'] if m['name'] == 'dim_school'][0]
cols = [c['name'] for c in dim['columns']]
assert 'has_sixth_form' in cols, cols
print('OK: schema yml documents has_sixth_form')
" && \
grep -c "has_sixth_form" pipeline/transform/models/marts/dim_school.sql && \
python3 -c "import ast; ast.parse(open('backend/models.py').read()); print('OK: models.py parses')"

Expected: OK: schema yml documents has_sixth_form, grep count >= 1, OK: models.py parses.

• Step 5: Commit

git add pipeline/transform/models/marts/dim_school.sql pipeline/transform/models/marts/_marts_schema.yml backend/models.py
git commit -m "feat(pipeline): derive dim_school.has_sixth_form from GIAS flag

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

---

Task 3: Backend — expose has_sixth_form and replace the filter heuristic

Files:

• Modify: backend/data_loader.py:117-215 (_MAIN_QUERY — select the column)
• Modify: backend/schemas.py:536-553 (SCHOOL_COLUMNS — include in list payloads)
• Modify: backend/app.py:419-422 (filter) and backend/app.py:589-610 (detail school_info)
• Test: backend/tests/test_sixth_form_flag.py (new)

Interfaces:

• Consumes: marts.dim_school.has_sixth_form (Task 2).

• Produces: has_sixth_form: bool | null field on GET /api/schools items and on GET /api/schools/{urn} → school_info. Filter GET /api/schools?has_sixth_form=yes|no now driven by the flag. Frontend (Task 4) reads school.has_sixth_form.

• Step 1: Write the failing tests

Create backend/tests/test_sixth_form_flag.py:

"""Tests for the GIAS-driven has_sixth_form flag (spec 2026-07-07 §3).

The filter and payloads must use dim_school.has_sixth_form, not the old
age_range-contains-"18" substring heuristic. The key regression case is a
16-19 sixth-form college: flag true, but "16-19" contains no "18".
"""

import numpy as np
import pandas as pd
import pytest
from fastapi.testclient import TestClient


def _schools_df() -> pd.DataFrame:
    """Latest-year snapshot rows as produced by load_latest_school_data."""
    base = {
        "local_authority": "Testshire",
        "school_type": "Academy",
        "phase": "Secondary",
        "address": "1 Test Street",
        "town": "Testtown",
        "postcode": "TS1 1AA",
        "religious_denomination": None,
        "gender": "Mixed",
        "admissions_policy": None,
        "ofsted_grade": np.nan,
        "ofsted_date": None,
        "ofsted_framework": None,
        "latitude": 51.5,
        "longitude": -0.1,
        "year": 202425,
        "total_pupils": 1000,
        "rwm_expected_pct": np.nan,
        "attainment_8_score": 50.0,
    }
    return pd.DataFrame(
        [
            # 11-18 school WITH a registered sixth form
            {**base, "urn": 100001, "school_name": "Alpha High",
             "age_range": "11-18", "has_sixth_form": True},
            # 16-19 college: old heuristic said NO ("16-19" has no "18"),
            # GIAS flag says YES — must appear in the yes-filter results
            {**base, "urn": 100002, "school_name": "Beta Sixth Form College",
             "age_range": "16-19", "has_sixth_form": True},
            # 11-18 age range on paper but NO registered sixth form:
            # old heuristic said YES, GIAS flag says NO
            {**base, "urn": 100003, "school_name": "Gamma Academy",
             "age_range": "11-18", "has_sixth_form": False},
            # Missing flag (pipeline not yet re-run) — must not crash,
            # must not match the yes-filter
            {**base, "urn": 100004, "school_name": "Delta School",
             "age_range": "11-16", "has_sixth_form": None},
        ]
    )


@pytest.fixture()
def client(monkeypatch):
    from backend import app as app_module

    monkeypatch.setattr(app_module, "load_latest_school_data", _schools_df)
    monkeypatch.setattr(app_module, "load_school_data", _schools_df)
    monkeypatch.setattr(app_module, "get_supplementary_data", lambda db, urn: {})
    return TestClient(app_module.app, raise_server_exceptions=False)


def _urns(resp):
    return sorted(s["urn"] for s in resp.json()["schools"])


def test_filter_yes_uses_flag_not_age_range(client):
    resp = client.get("/api/schools?has_sixth_form=yes")
    assert resp.status_code == 200, resp.text
    # 16-19 college included; 11-18-without-sixth-form excluded
    assert _urns(resp) == [100001, 100002]


def test_filter_no_uses_flag_not_age_range(client):
    resp = client.get("/api/schools?has_sixth_form=no")
    assert resp.status_code == 200, resp.text
    # Gamma (flag false) and Delta (flag missing => not true)
    assert _urns(resp) == [100003, 100004]


def test_list_payload_includes_flag(client):
    resp = client.get("/api/schools")
    assert resp.status_code == 200, resp.text
    by_urn = {s["urn"]: s for s in resp.json()["schools"]}
    assert by_urn[100002]["has_sixth_form"] is True
    assert by_urn[100003]["has_sixth_form"] is False
    assert by_urn[100004]["has_sixth_form"] is None


def test_detail_payload_includes_flag(client):
    resp = client.get("/api/schools/100002")
    assert resp.status_code == 200, resp.text
    assert resp.json()["school_info"]["has_sixth_form"] is True

• Step 2: Run tests to verify they fail

Run: cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests/test_sixth_form_flag.py -v Expected: FAIL — test_filter_yes_uses_flag_not_age_range asserts [100001, 100002] but the age-range heuristic returns [100001, 100003]; the payload tests fail with KeyError: 'has_sixth_form'.

• Step 3: Select the column in _MAIN_QUERY

In backend/data_loader.py, in _MAIN_QUERY, add after s.age_range,:

        s.has_sixth_form,

• Step 4: Include it in list payloads

In backend/schemas.py, in SCHOOL_COLUMNS, add after "age_range",:

    "has_sixth_form",

(app.py builds list responses from SCHOOL_COLUMNS ∩ df.columns, so a DB that predates the pipeline re-run simply omits the field — no crash.)

• Step 5: Replace the filter heuristic in app.py

Replace lines 419-422:

    if has_sixth_form == "yes":
        df_latest = df_latest[df_latest["age_range"].str.contains("18", na=False)]
    elif has_sixth_form == "no":
        df_latest = df_latest[~df_latest["age_range"].str.contains("18", na=False)]

with:

    # GIAS OfficialSixthForm flag (dim_school.has_sixth_form). NULL (flag not
    # yet populated by the pipeline) is treated as "no sixth form".
    if has_sixth_form in ("yes", "no"):
        if "has_sixth_form" in df_latest.columns:
            flag = df_latest["has_sixth_form"].eq(True)
        else:  # DB predates the pipeline re-run — fall back to age range
            flag = df_latest["age_range"].str.contains("18", na=False)
        df_latest = df_latest[flag if has_sixth_form == "yes" else ~flag]

• Step 6: Add the flag to the detail payload

In backend/app.py school_info dict (line ~598), add after "age_range": latest.get("age_range", ""),:

            "has_sixth_form": latest.get("has_sixth_form"),

(convert_to_native already maps NaN/None → null and numpy bools → bool.)

• Step 7: Run the new tests

Run: cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests/test_sixth_form_flag.py -v Expected: 4 passed.

• Step 8: Run the full backend suite

Run: cd /Users/tudor/projects/school_compare && python3 -m pytest backend/tests -v Expected: all pass (the pre-existing test_school_details.py df has no has_sixth_form column — latest.get() returns None, serialized as null).

• Step 9: Commit

git add backend/data_loader.py backend/schemas.py backend/app.py backend/tests/test_sixth_form_flag.py
git commit -m "feat(api): drive has_sixth_form filter and payloads from GIAS flag

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

---

Task 4: Frontend — badge, note, row tag, and filter labels use the flag

Files:

• Modify: nextjs-app/lib/types.ts:10-30 (School interface)
• Modify: nextjs-app/components/SecondarySchoolDetailView.tsx:101 (badge + coming-soon note)
• Modify: nextjs-app/components/SecondarySchoolRow.tsx:25-27 (row tag)
• Modify: nextjs-app/components/FilterBar.tsx:370-372 (labels only — param name unchanged)
• Test: nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx (new)

Interfaces:

• Consumes: has_sixth_form: boolean | null on both list items and school_info (Task 3; both are typed as School).

• Produces: no new exports — behavior change only.

• Step 1: Write the failing test

Create nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx:

/**
 * SecondarySchoolRow — sixth-form tag must come from the GIAS
 * has_sixth_form flag, not the age_range-contains-"18" heuristic.
 */

import '@testing-library/jest-dom';
import { render, screen } from '@testing-library/react';
import { SecondarySchoolRow } from '@/components/SecondarySchoolRow';
import type { School } from '@/lib/types';

const base = {
  urn: 100002,
  school_name: 'Beta Sixth Form College',
  local_authority: 'Testshire',
  school_type: 'Academy',
  phase: 'Secondary',
  gender: 'Mixed',
  attainment_8_score: 50.0,
} as unknown as School;

describe('SecondarySchoolRow sixth-form tag', () => {
  it('shows the tag for a 16-19 college with the GIAS flag set', () => {
    render(
      <SecondarySchoolRow
        school={{ ...base, age_range: '16-19', has_sixth_form: true }}
      />,
    );
    expect(screen.getByText('Sixth form')).toBeInTheDocument();
  });

  it('hides the tag for an 11-18 school without a registered sixth form', () => {
    render(
      <SecondarySchoolRow
        school={{ ...base, age_range: '11-18', has_sixth_form: false }}
      />,
    );
    expect(screen.queryByText('Sixth form')).not.toBeInTheDocument();
  });

  it('hides the tag when the flag is missing (pipeline not yet re-run)', () => {
    render(
      <SecondarySchoolRow school={{ ...base, age_range: '11-18' }} />,
    );
    expect(screen.queryByText('Sixth form')).not.toBeInTheDocument();
  });
});

• Step 2: Run it to verify it fails

Run: cd /Users/tudor/projects/school_compare/nextjs-app && npx jest __tests__/components/SecondarySchoolRow.test.tsx Expected: FAIL — first test can't find "Sixth form" ("16-19" fails the substring check), second test finds an unexpected "Sixth form" tag. (If TS complains that has_sixth_form is not on School, that is the same failure — proceed.)

• Step 3: Add the field to the School type

In nextjs-app/lib/types.ts, in export interface School, add after age_range: string | null;:

  has_sixth_form?: boolean | null;

• Step 4: Switch SecondarySchoolRow to the flag

Replace the helper at SecondarySchoolRow.tsx:25-27:

function hasSixthForm(school: School): boolean {
  return school.age_range?.includes('18') ?? false;
}

with:

function hasSixthForm(school: School): boolean {
  // GIAS OfficialSixthForm flag; missing (pipeline not yet re-run) => false.
  return school.has_sixth_form ?? false;
}

• Step 5: Switch SecondarySchoolDetailView to the flag

Replace line 101:

  const hasSixthForm = schoolInfo.age_range?.includes('18') ?? false;

with:

  // GIAS OfficialSixthForm flag; missing (pipeline not yet re-run) => false.
  const hasSixthForm = schoolInfo.has_sixth_form ?? false;

(This drives both the header "Sixth form" badge at line ~230 and the "Post-16 destination data coming soon" note at line ~715 — no changes needed there.)

• Step 6: Fix the filter labels in FilterBar.tsx

Replace:

                    <option value="yes">With sixth form (11-18)</option>
                    <option value="no">Without sixth form (11-16)</option>

with:

                    <option value="yes">With sixth form</option>
                    <option value="no">Without sixth form</option>

• Step 7: Run the new test and verify it passes

Run: cd /Users/tudor/projects/school_compare/nextjs-app && npx jest __tests__/components/SecondarySchoolRow.test.tsx Expected: 3 passed.

• Step 8: Run the full frontend checks

Run: cd /Users/tudor/projects/school_compare/nextjs-app && npx tsc --noEmit && npx jest Expected: typecheck clean, all Jest suites pass.

• Step 9: Verify no heuristic remains

Run:

grep -rn "includes('18')\|contains(\"18\")" /Users/tudor/projects/school_compare/nextjs-app/components /Users/tudor/projects/school_compare/backend --include="*.tsx" --include="*.ts" --include="*.py" | grep -v test

Expected: only the documented fallback inside app.py (DB-predates-pipeline branch); no other hits.

• Step 10: Commit

git add nextjs-app/lib/types.ts nextjs-app/components/SecondarySchoolRow.tsx nextjs-app/components/SecondarySchoolDetailView.tsx nextjs-app/components/FilterBar.tsx nextjs-app/__tests__/components/SecondarySchoolRow.test.tsx
git commit -m "feat(ui): sixth-form badge, note and filter labels use GIAS flag

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

---

Task 5: Update the spec status + PR

Files:

• Modify: docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md (§3 "Pipeline change (future work)" → implemented)

Interfaces:

• Consumes: everything above merged into the branch.

• Produces: PR ready for review; e2e journeys are the promotion gate (no journey currently exercises the sixth-form filter, and the API contract is unchanged, so no e2e change is required — state this in the PR body).

• Step 1: Mark spec §3 pipeline change as implemented

In the spec, change the §3 heading ### Pipeline change (future work) to ### Pipeline change (implemented 2026-07-07) and append one line at the end of that subsection:

Implemented in `feat/gias-sixth-form-flag` — see
`docs/superpowers/plans/2026-07-07-gias-sixth-form-flag.md`.

• Step 2: Commit

git add docs/superpowers/specs/2026-07-07-exam-phase-taxonomy-design.md
git commit -m "docs: mark sixth-form flag pipeline change implemented

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

• Step 3: Push and open the PR (Gitea)

Push the branch, then create the PR against main using the Gitea API via the git credential helper (token-header auth 401s on this Gitea; basic auth from git credential fill works):

git push -u origin feat/gias-sixth-form-flag

PR title: feat: drive sixth-form separation from GIAS OfficialSixthForm flag PR body must note: (1) API contract unchanged (has_sixth_form=yes|no), (2) flag is NULL until the next pipeline run — backend and frontend degrade to "no sixth form" / age-range fallback, (3) no e2e journey change needed, and end with the standard generation footer.

• Step 4: Verify CI passes

Watch the PR checks (typecheck, tests, builds, AI review). All must pass before merge; merging deploys to staging automatically.