feat(data): publish England only, dropping Welsh and overseas establishments #107

Merged
tudor merged 3 commits from feat/england-only-corpus into main 2026-08-20 21:20:13 +00:00
3 Commits
Author SHA1 Message Date
TudorandClaude Opus 5 69f2201244 docs(seo): correct W1 plan's sitemap child routes
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m2s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 10s
PR Checks / Build Frontend (no push) (pull_request) Successful in 44s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 35s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 9s
Next only treats a whole bracketed path segment as dynamic, so the planned
app/sitemap-[...parts]/route.ts would have been read as a literal static
folder and never matched. Children move under /sitemaps/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-20 22:09:00 +01:00
TudorandClaude Opus 5 e6048c9ca6 docs(seo): implementation plan for W1, crawl hygiene and sitemap
Five tasks: one canonical host, canonicals on every route, noindex on
parameterised comparisons, a sitemap that drops dataless schools and invented
priorities, and a per-family sitemap index.

Planning turned up a fault the spec had missed: the apex 301s to www, but
metadataBase, the school-page canonical, robots.txt's Sitemap: line and every
sitemap <loc> named the apex. Task 1 fixes it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-20 22:05:00 +01:00
TudorandClaude Opus 5 7650b16f62 feat(data): publish England only, dropping Welsh and overseas establishments
GIAS ships the whole UK plus overseas and offshore establishments. None of
them carry comparable DfE performance data — Wales does not publish on the
English measures at all — so every one of these pages rendered with null
results, null Ofsted and null phase. There were 2,036 of them: 1,569 Welsh,
123 offshore (Jersey, Guernsey, Isle of Man, Gibraltar), 316 British schools
overseas and 28 service children's schools. All 2,036 were being submitted to
search engines, alongside 29 local authorities that existed in the filters
purely to list them.

Filter at the mart boundary rather than the view layer. dim_school and
dim_location both exclude TypeOfEstablishment in {25, 26, 30, 37}, listed once
as vars.non_england_school_type_codes. Everything downstream reads those two
marts — search, the school page, /api/filters, rankings, Typesense and
build_sitemap() — so one filter removes them from the site and the sitemap
together, and Typesense drops them on its next rebuild since it recreates the
collection and swaps the alias rather than upserting in place.

coalesce rather than a bare NOT IN: a null type code would make the predicate
null and drop the row silently, and an unknown type is not grounds for
exclusion. No establishment has a null type today, but a future GIAS refresh
could ship one and the loss would be invisible.

assert_england_only_schools guards both directions: no excluded type survives
in dim_school, and dim_location holds no URN dim_school lacks — the API
inner-joins them, so the two filters drifting apart would silently shrink the
corpus.

Corpus goes from 27,229 schools to 25,193, and the authority list from 182 to
153. The 1,569 Welsh URLs now 404.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
2026-08-20 21:46:28 +01:00