diff --git a/docs/superpowers/specs/2026-10-02-school-type-groups-and-faith-filter-design.md b/docs/superpowers/specs/2026-10-02-school-type-groups-and-faith-filter-design.md new file mode 100644 index 0000000..af02ad1 --- /dev/null +++ b/docs/superpowers/specs/2026-10-02-school-type-groups-and-faith-filter-design.md @@ -0,0 +1,218 @@ +# School Type Groups and a Faith Filter — Design + +**Date:** 2026-10-02 +**Status:** approved in conversation, awaiting spec review +**Scope:** search filters (`/` results toolbar and phone filter sheet), `/api/schools`, `/api/filters` + +## Goal + +Replace the School type filter's 34 GIAS establishment types with six groups a +parent recognises, and add a Faith filter. + +Since PR #169 the School type select offers the full GIAS list rather than the +types in the results. That fixed the trap where choosing a type left only that +type on offer, but it exposed the list itself: "Academy converter", "Academy +sponsor led", "Free schools", "Foundation school", "Voluntary controlled +school" and 29 more. These describe governance and funding. For a mainstream +state school they change almost nothing a parent experiences, and parents +cannot be expected to know the differences. + +What parents actually ask is: is it free, is it mainstream, is it for children +with special needs, is it a sixth form or college, and is it a faith school. +The first four are the type groups below. The fifth is the real meaning behind +"Voluntary aided" and "Voluntary controlled", but many academies are faith +schools too, so it gets its own filter rather than hiding inside type. + +## Non-goals + +- Changing what a school page or result row shows. They keep the precise GIAS + type ("Academy sponsor led"), where it is information, not a choice. +- Removing the "not offered" types from search results. They stay reachable + under "Any school type"; whether parents should see them at all is a + separate decision. +- Reordering the type options by phase (for example "Sixth form or college" + first for 16 plus). Fixed order is simpler. +- The rankings page, which has no type filter. +- Changing `result_filters`. Its `school_types` key is already unread + (docs/LEGACY_CODE.md). + +## Type groups + +Order is the order shown. Codes are GIAS `TypeOfEstablishment` codes +(`backend/gias_codes.py: SCHOOL_TYPE`). Counts are staging schools on +2026-10-02. + +| Key | Label | GIAS codes | Schools | +|---|---|---|---| +| `academy` | State school: academy or free school | 28 Academy sponsor led, 34 Academy converter, 35 Free schools, 40 University technical college, 41 Studio schools, 6 City technology college | 11,186 | +| `council` | State school: council-run | 1 Community school, 2 Voluntary aided school, 3 Voluntary controlled school, 5 Foundation school, 15 Local authority nursery school | 9,316 | +| `independent` | Independent (fee-paying) | 11 Other independent school | 1,585 | +| `special` | Special school (SEND) | 7 Community special, 12 Foundation special, 44 Academy special converter, 33 Academy special sponsor led, 36 Free schools special, 8 Non-maintained special, 10 Other independent special, 32 Special post 16 institution | 2,227 | +| `post16` | Sixth form or college | 18 Further education, 31 Sixth form centres, 45 Academy 16-19 converter, 46 Academy 16 to 19 sponsor led, 39 Free schools 16 to 19 | 302 | +| `alternative` | Alternative provision | 14 Pupil referral unit, 42 Academy AP converter, 43 Academy AP sponsor led, 38 Free schools AP | 331 | + +**Not offered** (no group, reachable only under "Any school type"): 29 Higher +education institutions, 27 Miscellaneous, 24 Secure units, 49 Online provider, +57 Academy secure 16 to 19, 56 Institution funded by other government +department. 238 schools. + +**Excluded upstream** (never in the marts): 25, 26, 30, 37, per +`non_england_school_type_codes` in `pipeline/transform/dbt_project.yml`. + +### Judgement calls + +- **Independent special schools are Special, not Independent.** They are + usually funded by the local authority through a child's EHCP; to a parent + they are SEND provision, not private school. +- **UTCs, studio schools and city technology colleges are academies.** They are + legally academies, they are few (66 together), and a family considering one + searches for it by name. +- **Special post 16 institutions are Special, not Sixth form or college.** The + defining fact for a parent is the SEND provision. +- **Alternative provision is last.** Parents do not apply to it; the local + authority places children there. + +## Faith groups + +Codes are GIAS `ReligiousCharacter` codes +(`backend/gias_codes.py: RELIGIOUS_CHARACTER`). A joint school belongs to every +faith its label names, so "Roman Catholic/Church of England" matches both +Church of England and Roman Catholic. A generic "Christian" alongside a named +denomination adds nothing ("Church of England/Christian" is Church of England +only). + +| Key | Label | GIAS codes | +|---|---|---| +| `none` | No religious character | 0 Does not apply, 6 None, 99 (blank), and a missing code | +| `church_of_england` | Church of England | 2, 31 Anglican, 34 Anglican/Church of England, 20 CofE/Christian, 32 Anglican/Christian, and the joint codes 9, 10, 11, 12, 13, 19, 30, 33, 41, 48 | +| `roman_catholic` | Roman Catholic | 3, 35 Catholic, and the joint codes 11, 13, 48 | +| `other_christian` | Other Christian | 4 Methodist, 8 Seventh Day Adventist, 14 Quaker, 15 Christian, 16 United Reformed Church, 17 Congregational Church, 18 Free Church, 22 Greek Orthodox, 26 Moravian, 28 Inter- / non- denominational, 37 Christian/Evangelical, 38 Christian Science, 39 Christian/Methodist, 40 Christian/non-denominational, 44 Plymouth Brethren Christian Church, 45 Protestant, 46 Protestant/Evangelical, 47 Reformed Baptist, and the joint codes 9, 10, 12, 19, 30, 33, 41 | +| `jewish` | Jewish | 5 Jewish, 36 Charadi Jewish, 43 Orthodox Jewish | +| `muslim` | Muslim | 7 Muslim, 42 Islam, 49 Sunni Deobandi | +| `other_faith` | Other faith | 21 Sikh, 24 Buddhist, 25 Hindu, 29 Multi-faith | + +The joint codes: 9 CofE/Methodist, 10 Methodist/CofE, 11 CofE/RC, 12 CofE/URC, +13 RC/CofE, 19 CofE/Free Church, 30 CofE/Methodist/URC/Baptist, +33 Anglican/Evangelical, 41 CofE/Evangelical, 48 RC/Anglican. + +28 "Inter- / non- denominational" is filed as Other Christian: GIAS uses it for +Christian schools that are not tied to one church. + +## Architecture + +Grouping lives in the backend, not dbt. The API already translates GIAS codes +to names when it loads the marts (`backend/data_loader.py: +translate_gias_code_columns`), and every filter is applied to that DataFrame. +Grouping at the same point needs no mart change, so there is no Airflow run +between merge and staging showing it. + +### `backend/school_groups.py` (new) + +- `TYPE_GROUPS`: ordered `(key, label, frozenset[int])` per type group. +- `UNOFFERED_TYPE_CODES`: the not-offered codes, so that "every code is + accounted for" is testable. +- `FAITH_GROUPS`: ordered `(key, label, frozenset[int])` per faith group. +- `type_group_for(code) -> str | None` and + `faith_groups_for(code) -> tuple[str, ...]` (a missing code gives + `("none",)`). + +No dependence on the generated GIAS dictionaries beyond their codes, so the +backend/pipeline dictionary parity test is untouched. + +### `backend/data_loader.py` + +In `translate_gias_code_columns`, before the code columns are replaced by +names, add two columns from the codes: + +- `school_type_group`: `type_group_for(school_type_code)`, or None. +- `faith_groups`: `faith_groups_for(religious_character_code)`, a tuple. + +The fallback query that reads name columns from older marts +(`_MAIN_QUERY_NO_EXTRA_COLS` and its replacements) has no codes. There, both +columns are derived from the names by reverse lookup through `SCHOOL_TYPE` and +`RELIGIOUS_CHARACTER`. + +### `/api/schools` + +- `school_type`: if the value is a type group key, filter on + `school_type_group`. Otherwise filter on the raw label exactly as today, so + an old `?school_type=Community+school` link keeps working. +- `faith` (new, optional, `max_length=40`, sanitised like the others): filter to + rows whose `faith_groups` contains the key. An unknown key returns no + schools rather than being ignored, so a typo does not silently show + everything. + +### `/api/filters` + +Two new keys: + +- `school_type_groups`: `[{value, label}]` in `TYPE_GROUPS` order, only groups + with at least one school. +- `faiths`: `[{value, label}]` in `FAITH_GROUPS` order, same rule. + +`school_types` stays as it is (the raw list), so nothing that reads it breaks. + +### Frontend + +- `lib/types.ts`: `Filters` gains optional `school_type_groups` and `faiths` + (`{ value: string; label: string }[]`). Optional, so the empty fallbacks in + `app/(frontend)/page.tsx` and `rankings/page.tsx` stay valid. +- `app/(frontend)/page.tsx`: reads `faith` from the search params, counts it + in `hasSearchParams`, and passes it to `fetchSchools`. `SchoolSearchParams` + in `lib/types.ts` gains `faith`. HomeView's load-more already forwards every + URL param. HomeView's `isSearchActive` is left as it is: like phase and + gender, faith narrows a search rather than starting one. +- `components/FilterBar.tsx`: + - The School type select's options become `filters.school_type_groups`; + "Any school type" stays first. With no groups (the API failed), the select + is left out, as Gender and Admissions already are. + - A new **Faith** select (`aria-label="Faith"`, "Any faith or none" first) + from `filters.faiths`. On desktop it goes in the More filters panel, after + Local authority. In the phone sheet it comes after School type. + - `faith` joins `FILTER_KEYS` (chip and Filters count) and the More filters + count, and `faith=` joins `filters_active` in the `search_submitted` + analytics event. + - Chip labels come from the option lists: "Special school (SEND)", + "Roman Catholic". An old raw-label `school_type` shows its raw label. +- `lib/utils.ts: isSpecialSchool` is unchanged. It reads a school's raw + `school_type`, which still arrives. + +## Testing + +**Backend (pytest):** + +- Every `SCHOOL_TYPE` code is in exactly one type group, in + `UNOFFERED_TYPE_CODES`, or in `non_england_school_type_codes`. A new DfE + code fails this test instead of silently vanishing from the filter. +- Every `RELIGIOUS_CHARACTER` code maps to at least one faith group. +- The joint codes map to each faith they name (11 and 48 → CofE and RC; + 9 → CofE and Other Christian) and 20 → CofE only. +- `/api/schools?school_type=special` returns only special-group schools; + `?school_type=Community+school` still filters by label. +- `/api/schools?faith=roman_catholic` returns only matching schools, a joint + school included; `?faith=nonsense` returns none. +- `/api/filters` lists the groups in order and leaves out an empty one. +- Run via uv, as the backend tests always are. + +**Frontend (Jest):** + +- School type offers the six group labels, not raw types. +- Faith offers its options, in the panel and in the sheet. +- A faith filter shows as a chip, counts on Filters and More filters, and + clears with Clear all. +- No groups or faiths in `filters` → the select is left out. + +**E2E (`e2e/tests/journeys.spec.ts`):** + +- A journey that picks "Special school (SEND)" and "Roman Catholic" from the + selects, then checks through `/api/schools` with the same params that every + returned school's `school_type` is a special type and its + `religious_denomination` names Catholic. Data-invariant: it asserts the + property, not a count. +- It can only pass after merge; the staging E2E gate runs post-merge. + +## Rollout + +One PR; backend and frontend deploy together on merge. The frontend tolerates +an API without the new keys (the selects are left out), so the order the +containers update in does not matter.