Beechwood College (URN 142458, Sully, CF64 5SE) survived the England-only
filter. GIAS types it a Special post 16 institution (32), not a Welsh
establishment (30), so filtering on establishment type alone left it behind —
the last Welsh school on the site, and the reason Vale of Glamorgan was still
in the authority list.
The earlier verification claimed the type codes mapped onto the Welsh
authorities in both directions. That was checked exhaustively for Cardiff and
by count for three others; Vale of Glamorgan was never checked, and it was the
one that did not hold.
LA code is the reliable discriminator: GIAS gives the 22 Welsh unitary
authorities the contiguous block 660-681, which English authorities never use.
Filtering on postcode would have been wrong — Redbrook, Tutshill, Wyedean and
two other Gloucestershire schools carry NP16/NP25 postcodes because Royal Mail
areas straddle the border, and they are English schools with English data. An
e2e test now pins those five so the fix cannot be simplified into a postcode
filter later.
The 660-681 range is documented GIAS structure this project cannot verify from
its own data, so the range filters and the authority NAME checks: if the range
is ever wrong, a Welsh authority reappears in assert_england_only_schools and
the pipeline fails loudly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
GIAS ships the whole UK plus overseas and offshore establishments. None of
them carry comparable DfE performance data — Wales does not publish on the
English measures at all — so every one of these pages rendered with null
results, null Ofsted and null phase. There were 2,036 of them: 1,569 Welsh,
123 offshore (Jersey, Guernsey, Isle of Man, Gibraltar), 316 British schools
overseas and 28 service children's schools. All 2,036 were being submitted to
search engines, alongside 29 local authorities that existed in the filters
purely to list them.
Filter at the mart boundary rather than the view layer. dim_school and
dim_location both exclude TypeOfEstablishment in {25, 26, 30, 37}, listed once
as vars.non_england_school_type_codes. Everything downstream reads those two
marts — search, the school page, /api/filters, rankings, Typesense and
build_sitemap() — so one filter removes them from the site and the sitemap
together, and Typesense drops them on its next rebuild since it recreates the
collection and swaps the alias rather than upserting in place.
coalesce rather than a bare NOT IN: a null type code would make the predicate
null and drop the row silently, and an unknown type is not grounds for
exclusion. No establishment has a null type today, but a future GIAS refresh
could ship one and the loss would be invisible.
assert_england_only_schools guards both directions: no excluded type survives
in dim_school, and dim_location holds no URN dim_school lacks — the API
inner-joins them, so the two filters drifting apart would silently shrink the
corpus.
Corpus goes from 27,229 schools to 25,193, and the authority list from 182 to
153. The 1,569 Welsh URLs now 404.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
These schools are still operating and publish results; they drop out
automatically when GIAS flips them to Closed since marts fully rebuild
each run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Remove optional flag from total_pupils (Typesense requires default
sorting field to be non-optional)
- Add latitude/longitude columns to dim_location computed from PostGIS
geom, for direct use by backend and Typesense sync
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostGIS extension lives in public schema; marts schema can't resolve
unqualified ST_MakePoint/ST_Transform calls.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GIAS grid references are the actual school location — far more accurate
than postcode centroids. Remove geocode_postcodes.py from the daily DAG
and the postcode-not-null filter from dim_location.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert GIAS British National Grid coordinates (EPSG:27700) to WGS84
(EPSG:4326) directly in the dbt model. The geocode script backfills
schools missing easting/northing via Postcodes.io.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
tudorandClaude Opus 4.6