100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
TudorandClaude Opus 5	5ad1cbfb53	fix(api): bound the search candidate set instead of draining Typesense ... Fetching every match kept scoped searches correct but left the number of round trips in the caller's hands: a one-letter query, or a deliberately broad one, walked the whole collection a page at a time. Cap the candidate set at 1,000 URNs — four pages — and return the relevance-ordered prefix when the ceiling is hit. That is still far more than one page, so the API's own authority, phase and postcode filters keep the matches they need, while latency and upstream load stay bounded. A capped query is logged so a genuinely truncated search is visible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-15 11:21:45 +01:00
TudorandClaude Opus 5	0c901cd0d1	feat(ci): gate promotion on the image set that actually passed E2E ... Staging health polling asked only whether something answered HTTP 200 at the base URL. It could not tell the new deployment from the old one, so journeys could pass against the previous release, and concurrent merges could move the staging tags underneath a run in flight. Each staging run now mints a build ID and stamps all three images with the commit and that ID, as labels and — for frontend and backend — as a build-time JSON file that environment overrides cannot rewrite. /release.json reports both identities uncached, and scripts/ci/release.py polls for the expected pair before and after the journeys. Only then are the captured build digests tagged verified-<sha>. Promotion resolves those verified tags to immutable digests, revalidates their labels, and refuses a mixed or incomplete set before any :prod tag moves. The whole staging workflow shares one concurrency group with cancellation disabled, so releases serialise. The scripts are stdlib-only and unit-tested against mocked registry and HTTP behaviour; PR checks now run the pipeline and CI suites too. The runbook records what this cannot prove locally, and that the first rollout needs a commit built by this workflow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-15 10:17:50 +01:00
TudorandClaude Opus 5	7b41218e6e	fix(web): show an outage as an outage, and drop superseded fetches ... The home page caught every fetch failure and rendered its empty state, so a backend outage looked like a site with no schools in it. School pages turned any error into notFound(), which told visitors — and crawlers — that a real school had ceased to exist. Place fetches did the same by returning [] and null. Failures now reach a retryable error boundary; only a genuine 404 still calls notFound(). "Load more" and the map fetch resolved against whatever state existed when they returned, so results from an abandoned search appended themselves to the new ones. Each fetch now carries an AbortController and checks that its search scope is still current before touching state. The map only records its cache key on success, so a failed load retries instead of pinning the stale marker set. Jest ignored .next/, whose build output otherwise shadowed real suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-15 10:17:18 +01:00
TudorandClaude Opus 5	9b75f54206	fix(api): publish a validated dataset, and stop truncating search ... Reload cleared the caches first and rebuilt afterwards, so any failure left the API serving nothing, and requests arriving mid-reload saw a half-swapped state. It now builds and validates the replacement frames, place registry, reverse index and sitemaps off the request loop, then publishes them in one synchronous step under a lock. A failed reload returns 503 and keeps the previous data. Sitemap regeneration takes the same path rather than clearing the live registry up front. Typesense search returned at most one page of hits and used an empty list for both "no matches" and "search is down", so a genuine empty result silently fell back to substring matching. It now pages through every candidate and returns None only on failure; the fallback matches literally, since a query containing regex metacharacters used to throw. Empty datasets answer 503 rather than 200-with-nothing or a misleading 404, so callers can tell an outage from an absent school. Adds /api/release, which reports the build identity baked into the image. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-15 10:16:58 +01:00
TudorandClaude Opus 5	38bc17cab3	feat(search): validate the index before the alias points at it ... The old sync created a collection, imported batches without reading a single import response, and swapped the alias regardless. A partial import published a half-empty index, and two overlapping DAG runs could prune each other's collections. Publication now checks every import response and the final document count before upserting the alias, and holds a session-scoped advisory lock across the read and the publish so concurrent runs serialise. Cleanup keeps the previous collection as a rollback pointer and is best-effort: an uncertain alias response must never delete what might still be live. Also parses the Typesense URL properly instead of splitting on colons, which mangled any host carrying a scheme and a default port. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-15 10:16:36 +01:00
TudorandClaude Opus 5	1d8858fbda	chore: remove the code the legacy CSV importer left behind ... `backend/migration.py` and `scripts/migrate_csv_to_db.py` import `School`, `SchoolResult`, `init_db` and `set_db_schema_version` — names that no longer exist. `scripts/geocode_schools.py` imports the same removed ORM model. None of the three can be imported against the current backend, so they were not dormant utilities anyone could fall back on; they were files that would fail on the first line. `backend/version.py` existed only to hand `SCHEMA_VERSION` to that importer, and the FastAPI lifespan performs no version-triggered import. Three symbols go with them, each confirmed to have no caller: the unvectorised `haversine_distance`, superseded by the inline NumPy calculation in search; `fetcher`, an SWR helper for a dependency this project does not install; and `kmToMiles`. `calculateDistance` stays — CutoffMapPanel uses it. Two comments pointed at `migrate_csv_to_db.py --drop` to explain why Payload owns its own schema. The reason survives the script: blog content must stay clear of the school marts and Airflow's metadata. Reworded rather than deleted, so the constraint keeps its justification. docs/LEGACY_CODE.md records what was removed and where to find it in history. It also records what was deliberately *not* removed, which is the more useful half: unused UI components awaiting a design decision, manual data utilities whose operators a repository search cannot see, and fallbacks that look obsolete but are load-bearing — `data_loader.py`'s older-mart branches, the generated GIAS dictionary copies, and the `legacy`-named dbt models that annual DAG selectors explicitly include. A zero-import count is evidence, not a verdict. The scripts that fetch DfE CSVs are marked historical and kept, pending confirmation that nobody runs them by hand. Checked: 190 backend tests, 429 frontend tests, `tsc --noEmit` clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016y2J6bs8gbuSJbH18w7Tan	2026-09-14 23:01:22 +01:00
TudorandClaude Opus 5	eaf5e5d180	docs: describe the system that exists, not the one we started with ... The README still opened on "Primary School Compass", a KS2 tool for Wandsworth and Merton served by FastAPI and vanilla JavaScript with Chart.js. Every layer of that sentence is now wrong: coverage is England-wide across KS2, KS4, all-through and post-16, Next.js owns the public UI, and school data comes from dbt-built `marts.*` rather than CSVs loaded at startup. The setup instructions walked a reader into a virtualenv and a CSV import that cannot build the current schema, so following the docs produced an empty database and a wrong mental model at the same time. Replace the narrative docs with two reference documents that were checked against the code: docs/ARCHITECTURE.md for request flow, data ownership, the backend/frontend module boundaries and the real publication sequence, and docs/DEVELOPMENT.md for the checks that actually run, including the container and CI version skew that makes "just run pytest" misleading. The env examples drifted the same way. ALLOWED_ORIGINS is a JSON array, not a comma-separated list; the frontend needs FASTAPI_URL, DATABASE_URL and PAYLOAD_SECRET, none of which were documented; and RATE_LIMIT_BURST, DEFAULT_PAGE_SIZE and MAX_PAGE_SIZE were presented as tuning controls the routes do not consult. Each is now stated as it behaves. MIGRATION_SUMMARY.md keeps its content but gains a banner, because it reads like setup instructions and is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016y2J6bs8gbuSJbH18w7Tan	2026-09-14 23:01:15 +01:00
TudorandClaude Opus 5	b6c2cd5116	fix(seo): declare the share card, which the route group stopped inheriting ... The staging E2E gate's one failure. Every link to the site pasted into a chat has been rendering bare. Staging serves og:title, og:description, og:url, og:site_name, og:type and twitter:card, and no og:image at all. So metadata from the layout reaches the page; only the file convention does not. app/opengraph-image.tsx does work — _not-found, which lives in the app root segment, carries an og:image from it in the build output. It does not reach the site's pages, which live in the (frontend) route group whose own layout.tsx is the root layout. The icon conventions are not affected: /icon.png and /apple-icon.png are both linked correctly on the same page, verified against staging. The asymmetry is the whole bug, and it arrived with the route-group split that Payload required. The file stays at the app root. Moving metadata files into a route group is what drops /robots.txt and hashes /icon.png, which CLAUDE.md records and which this must not undo — the build still emits all four of /robots.txt, /icon.png, /apple-icon.png and /opengraph-image. The root layout points at the route instead, and metadataBase makes it absolute, which the journey needs since it calls new URL() on the value. twitter.images is set for the same reason: the card is declared summary_large_image, and claiming a large-image card while supplying no image is worse than claiming a summary card. Checked before fixing that og:image was the only broken assertion in that journey: the test aborts at line 911, so its apple-touch-icon and maskable-icon assertions had never run. All four of those assets return 200 image/png from staging, so this does not simply move the failure further down the test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DXnXQKnPpZBBP61fBQiFkq	2026-09-14 21:54:32 +01:00
TudorandClaude Opus 5	b0d5334e06	perf(places): index the reverse lookup, and isolate the registry in tests ... Two review findings, both confirmed before fixing. The client fixture in test_school_details.py patched load_school_data but not _place_registry, which is a module-level cache. A probe settled it rather than an argument: poisoning the global with a registry built from data the fixture never saw, then issuing the fixture's own request, returned that other dataset's places. So the new `places == []` assertion was satisfied by a stale registry exactly as well as by the fixture's own data, and proved nothing. Every other test that touches place data already reset it; the fixture predates places existing and was never updated. It resets it now. places_for_urn walked every place in the registry and did a tuple membership test against each, on /api/schools/{urn}, the site's highest-traffic endpoint. It now reads a dict built once per registry. Measured against a synthetic corpus of 27k schools in 1,650 places: 0.118ms per request becomes 0.0001ms, with the index built once in 21ms. Production carries ~5,000 places, so the scan there is larger again. The absolute saving per request is small; the point is that it is repeated on every school page view and costs nothing to remove. The index is cached against the registry by identity rather than behind a second flag. Anything that drops _place_registry — every test that touches place data does — gets a fresh registry object, which no longer matches what the index was built from, so the index rebuilds with it. A separate _place_index = None would be one more thing to forget, and a stale reverse index is precisely the first finding's bug wearing a different hat. That invalidation has its own test, and the test was checked by breaking the identity check: five tests fail without it, so three existing ones were already relying on it too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DXnXQKnPpZBBP61fBQiFkq	2026-09-14 21:22:39 +01:00
TudorandClaude Opus 5	d65eb58883	fix(seo): build the phase links the docstring already promised ... Review caught _places_payload documenting a `phase_url` the function never returned. The docstring was not stray prose: the approved design included the phase variant — "Primary schools in Beccles" was one of its four example links — and it was dropped during implementation without being mentioned. Deleting the sentence would have closed the report while losing the feature, so the links are built instead. These are the pages that most needed them. ~950 phase variants were once reachable by nothing at all: absent from every sitemap and unlinked from the place page. "Primary schools in brentwood" is the query they exist to answer. Membership is read from the registry's own `phase_urns` rather than re-derived from the school's phase string. The registry already decides which phases a place publishes and which schools are listed on each, so asking it is both shorter and the only way the link cannot disagree with the page it points at. It also means outcodes need no special case: they carry empty `phase_urns` by design, because nobody searches "primary schools in SW11", so they report no phase links on their own. `phases` is a list rather than a single url. An all-through school is listed on both the primary and secondary pages, so there is no tie to break and no reason to invent one. Each entry renders directly after its own place, so "22 primary schools in Brentwood" reads as part of Brentwood rather than as an unrelated link further along the row. The e2e journey now follows a phase link where the town publishes one and asserts it resolves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DXnXQKnPpZBBP61fBQiFkq	2026-09-14 20:54:59 +01:00
TudorandClaude Opus 5	7f5f0fb676	feat(seo): link school pages into the location layer ... W2 shipped ~5,000 place pages and nothing linked into them. The location layer pointed down at school pages; school pages pointed nowhere on the site. Their only anchor was the school's own website, so the ~27k pages carrying most of the site's inbound authority passed it straight off-site, and the new corpus was reachable mainly through the sitemap. Three things close the loop. A reverse index over the place registry, places_for_urn, answers which published places contain a school. Derived from the registry rather than stored beside it, so the two cannot disagree about which places exist: a place below the publish threshold is absent from the registry and therefore never offered as a link. A test asserts that invariant across every place in a built registry. GET /api/schools/{urn} gains a `places` array carrying the name, count and canonical path for each. It rides on the request the page already makes, so the school page costs no extra round trip. The frontend types it optional and defaults it to empty, because the two images deploy separately and a frontend ahead of the API must render without it. The page gains a "More schools near here" module and a BreadcrumbList. The module orders narrowest first, because a reader on a school page wants its town before its county, while the API orders widest first for the trail. Anchors state their destination's size — "37 schools in Brentwood" — which is worth more to a reader and a crawler than "see more". With no published places it renders nothing rather than an empty heading. The trail is rooted at the homepage, not /schools. There is no /schools index page; the location layer lives only at /schools/[place], /schools/authority/[la] and /schools/near/[outcode]. Rooting it at the bare path would have opened every breadcrumb with a link to a 404. Outcodes are omitted from the trail: "schools near CM15" is a real query and a useful link, but nobody navigates Essex to CM15 to a school, and a breadcrumb claiming that describes a hierarchy the site does not have. School pages also now declare the School type rather than EducationalOrganization, the parent type that covers universities and nurseries alike. The e2e journey asserts the round trip in both directions, following a place page's own first school so the pair is genuinely related rather than hardcoded. A one-way link is what already existed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DXnXQKnPpZBBP61fBQiFkq	2026-09-14 20:45:50 +01:00
TudorandClaude Opus 5	6fc7fce948	feat(flags): put /about and /blog behind flags, dark by default ... Both features ship dark. Neither is reachable in an environment where its flag is off, and every flag in this system starts off, so a deploy of this commit makes both disappear until someone turns them on deliberately. Two independent flags rather than one, which makes blog-on-about-off a reachable state. That state is the whole reason the change is larger than four notFound() calls: the blog leans on the About page for its author identity. The Person entity is anchored at /about#tudor, and that URL 404s while about_page is dark, so a post published in that state would claim an author resolving to nothing. Worse than having no named author. Both bylines fall back to unlinked text and the BlogPosting attributes to the publisher instead, so every combination of the two flags renders something correct. Gated: /about, /blog, /blog/[slug], the RSS feed, both footer links, and the matching content-sitemap entries. A sitemap must never advertise a URL that 404s. With both dark it emits a valid empty urlset rather than a 404, because robots.txt names it unconditionally. Not gated: /admin. Posts have to be writable before the blog is worth switching on, so flagging the panel would make the flag unflippable. getFlags takes a revalidate rather than always using the 300s constant. Reading a flag pins the calling route to the lowest revalidate among its fetches, and the footer links live in the root layout, so a naive gate there would have dropped every school and place page from a weekly cache to a 5-minute one. The layout passes 604800, the floor those routes already declare, and the build confirms all four SSG route families still prerender. The cost is one-way latency: pages follow a flip in minutes, footer links within a week. The e2e journeys follow the existing paired shape from the admission_distance flag: a lit journey and a dark one for each flag, reading state from whether /about and /blog respond rather than from /api/flags, which another journey asserts is not publicly reachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DXnXQKnPpZBBP61fBQiFkq	2026-09-08 17:12:25 +01:00
TudorandClaude Opus 5	d47ac71c47	fix(cms): regenerate the import map so the Content field renders ... Creating a post in the admin panel showed no Content editor, and saving failed validation on the field the writer was never shown. The admin panel does not import field components. The server hands the client a path per field, and resolves it through the generated map at app/(payload)/admin/importMap.js. A richText field's path is @payloadcms/richtext-lexical/rsc#RscEntryLexicalField. The committed map held one entry, @payloadcms/next/rsc#CollectionCards, generated before the blog collections existed and never re-run. A path missing from the map is not an error the panel reports: the field simply does not render, while required is still enforced server-side on save. next build does not regenerate the map, so the stale copy shipped in the image and the editor was equally broken on staging and production. Regenerated with payload generate:importmap, which adds the lexical RSC field, cell and diff components, BlocksFeatureClient for the Callout block, and the default toolbar features. Two things stop it drifting again. There was no script to run, so package.json gets generate:importmap. And a test asserts the map carries an entry for each thing the config asks for, in the source-reading style of the other payload suites; against the old map all five fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DXnXQKnPpZBBP61fBQiFkq	2026-09-02 20:44:05 +01:00
TudorandClaude Opus 5	e2c63a9905	fix(cms): ship the initial migration so a container finds its tables ... Staging failed on boot with 42P01, relation "payload.users" does not exist. The schema was empty because no migration existed, and the adapter cannot create tables itself: db-postgres/connect.js gates push on NODE_ENV !== 'production', so it is inert in a deployed container regardless of config. The generated migration is schema-qualified to "payload" throughout but does not create that schema — schemaName says where tables go, it does not create anything. It only worked against the throwaway database used to generate it because the schema was created there by hand, so every real environment would have failed on the first statement. CREATE SCHEMA IF NOT EXISTS is hand-added at the top of up(), which makes it exactly the kind of edit a regeneration discards silently; a test asserts it is present and ordered before the first CREATE TABLE. payload-types.ts is now committed rather than ignored. Ignoring it meant CI typechecked against looser types than a developer with a generated copy, which is how a Record<string, unknown> cast passed CI and then failed locally the moment the file appeared. The post page uses the generated Post and Media types instead, and narrows heroImage rather than asserting it, since the field is an id at shallow depth. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 18:33:27 +01:00
TudorandClaude Opus 5	3f3c5953f6	style(copy): remove em dashes from the site's prose ... The em dash is one of the clearest tells of machine-written text, which is the exact impression this work exists to remove. Rewritten rather than substituted: where a dash was carrying a real aside the sentence is split or recast, not patched with a comma. Covers the About page, the two Callout labels an editor sees in the admin panel, and PUBLISHING.md, which defines the house style and should follow it. The rule is now recorded in that house style and in the spec's voice rules, so it survives this branch. Code comments are left alone: they are not copy, and the surrounding codebase uses the same punctuation throughout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 18:33:27 +01:00
TudorandClaude Opus 5	e25722d9ab	fix(blog): hide drafts at the access layer, and back the --drop claim ... Review findings on #140. Drafts were reachable. Posts granted unconditional public read and the _status filter lived only in the pages that query the collection — which is a convenience, not a control. Payload's documentation is explicit: "The `draft` argument alone does not restrict documents with _status: 'draft' from being returned by the API." A direct GET /cms-api/posts would have handed every unpublished draft to any visitor. Read access now returns a query constraint for anonymous callers, which is the documented mechanism. The --drop claim was asserted across four files while the spec still listed it as an open question. Now verified rather than assumed: run_full_migration drops exactly ["school_results", "schools"] by name, there is no drop_all() or DROP SCHEMA anywhere in backend/, the only other drop is schema-qualified to marts, and nothing sets search_path. The guarantee is stronger than schema isolation alone — those two table names do not exist in Payload — so the claim stands, but it now rests on cited code. The spec records the evidence and closes the open item. findPost is wrapped in React's cache(): Next calls generateMetadata and the page separately for one request, so every post view ran the same query against Postgres twice. The bare .lede rule was dead — .prose p scores (0,1,1) and outranks it — so only .prose .lede ever applied. Removed, with the specificity noted so the surviving selector is not "simplified" back into a silent regression. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:48:59 +01:00
TudorandClaude Opus 5	07d586d0ad	docs(blog): how to publish, and why the app has two route groups ... PUBLISHING.md carries the house style with the posts, so the standard survives without the design doc to hand — including the rule that a post states what a metric does not show, which is the strongest signal a human wrote it. CLAUDE.md gains the two constraints that are invisible from the code and expensive to rediscover: metadata file conventions break if moved into a route group, and the build must keep succeeding with DATABASE_URL unset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:30:09 +01:00
TudorandClaude Opus 5	b793640507	feat(blog): add the blog index, post pages, RSS and content sitemap ... The rendering split is dictated by CI building with no database. /blog, /blog/rss.xml and /content-sitemap.xml have no dynamic params, so Next prerenders them at build time and the build fails on a missing Payload secret — caught here, not on staging. They are force-dynamic instead: one indexed query against Postgres on the same Docker network, and a newly published post appears immediately rather than waiting on a revalidation. /blog/[slug] keeps ISR, because with no generateStaticParams there is nothing to prerender; it is generated on first request and cached, which is exactly what the collection's afterChange hook exists to invalidate. RichText takes `converters`, not `blocks`, in Payload 3.88, and the default converters must be spread or every paragraph and heading loses its renderer and the body comes out empty. BlogPosting references the Person and Organization by @id rather than repeating them, so every post and the About page resolve to one author entity instead of declaring several people with the same name. /sitemap.xml is proxied from FastAPI, which knows nothing about Payload, so the Next-owned URLs get their own sitemap and robots.txt lists both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:28:58 +01:00
TudorandClaude Opus 5	21a5d18f59	feat(blog): add the posts and media collections ... Drafts are on so a post can be written across sittings without saving being publishing. afterChange and afterDelete revalidate every path a post appears on. Blog pages are ISR because CI builds with no database, so without these a published post would not appear until the revalidate window expired — up to an hour of a writer concluding that publishing is broken. Payload runs in the same process as Next, so these are direct revalidatePath calls with no webhook and no shared secret. Media writes to an absolute /app/media matching the compose mount; a mismatch would write into the container filesystem, where the next redeploy silently discards it. Alt text is required rather than optional. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:24:15 +01:00
TudorandClaude Opus 5	f614414070	feat(about): give the site a named author ... The site had no author, no statement of why it exists and nobody accountable for its numbers, which is most of why it reads as machine generated. The page states plainly that its author is not an education expert. The credibility claim is lived experience — a parent going through primary admissions — plus stated provenance for every figure, which is true and cannot be undermined by someone noticing there is no teaching qualification behind it. First name only: the Person JSON-LD carries no familyName, worksFor or affiliation, and a test asserts it stays that way. The footer gains a fourth column, with a tablet breakpoint so four columns pair up rather than crushing before the 768px collapse. The nav is deliberately untouched — its mobile tab bar already carries four items. public/brand/tudor.jpg is NOT in this commit. The page references it and will show a broken image until the photograph is supplied; a stock portrait would defeat the entire point of the work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:22:52 +01:00
TudorandClaude Opus 5	310b63b0cb	build(cms): wire Payload into the Docker image and both stacks ... Uploads go to a named volume at /app/media. The directory is created in the image before the mount and covered by the existing chown, because Docker seeds a fresh named volume from the image path — a missing or root-owned directory there fails every upload with EACCES at runtime, long after the build passed. PAYLOAD_SECRET uses the same :? form as AIRFLOW_ADMIN_PASSWORD: refuse to start rather than boot with an empty secret and accept forged sessions. Staging's must differ from production's, which the header comment now says explicitly. Portainer prefixes volume names per stack, so payload_media isolates itself. prodMigrations is not wired yet — generating the initial migration needs a reachable Postgres. Follows in its own commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:19:27 +01:00
TudorandClaude Opus 5	c2c76c5817	feat(cms): keep the admin panel out of the index ... X-Robots-Tag rather than the robots.txt Disallow alone, for the same reason the staging rule uses one: a Disallow blocks crawling, not indexing, so a URL found from an external link can be indexed without ever being fetched — and blocking the crawl means the noindex is never seen. Both mechanisms are applied to /admin and /cms-api. The existing CSP is frame-ancestors only, which restricts who may embed the site rather than what a page may load, so it cannot break the panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:17:55 +01:00
TudorandClaude Opus 5	c5a4d106da	feat(cms): install Payload and serve the admin panel ... Payload 3.88 runs inside the Next app against the existing Postgres, in its own 'payload' schema so no pipeline operation on public — the app tables, Airflow's metadata, migrate_csv_to_db.py --drop — can reach blog content. Its REST API is mounted at /cms-api. /api is the FastAPI proxy's catch-all, which would swallow every admin call and forward it to the backend with no error. The mount points live in lib/payloadRoutes.ts so there is one definition and a test can assert it without importing Payload: it is ESM-only, next/jest will not transform it, and appending transformIgnorePatterns cannot un-ignore a package. Forcing it through transpilePackages would change how the production build bundles Payload to serve a test, so the live proof that /api still reaches FastAPI stays where it belongs — the e2e journeys, which call /api/schools. The package becomes ESM ("type": "module"), which Payload's CLI requires: richtext-lexical has top-level await and the config cannot be require()d. Only two files needed renaming, jest.config.cjs and a build script. The build is verified to succeed with DATABASE_URL and PAYLOAD_SECRET both unset, which is how CI builds it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:17:01 +01:00
TudorandClaude Opus 5	2437ffce42	refactor(app): move site routes into a (frontend) route group ... Payload's admin panel ships its own root layout rendering html/body. Next allows multiple root layouts only when no app/layout.tsx exists, so the site's routes move into their own group. Route groups are invisible to routing: every public URL is unchanged, verified against the build's route table. The metadata file conventions deliberately stay at the app/ root. Moving them into the group renamed /icon.png to /icon-4usi79.png (likewise apple-icon and opengraph-image) and dropped /robots.txt altogether, which would have broken the /icon.png cache-control rule, the outputFileTracingIncludes entry for the share card, and robots.txt. darkThemeSafety reads app/globals.css off disk rather than importing it, so it needed its own path fix — a grep for import specifiers misses it, and it fails as an unrunnable suite rather than a failed assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:11:48 +01:00
TudorandClaude Opus 5	eb648f3f76	build(next): convert the config to ESM so Payload can wrap it ... withPayload() is ESM-only, so next.config.js has to become .mjs. That file also carries the rule that keeps staging out of Google's index, so the conversion goes in on its own, behind a test that asserts the rule survived — along with the standalone output, the opengraph-image font tracing and the analytics frame-ancestors CSP. Jest resolves the .mjs config without extra configuration, so jest.config.js is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 16:08:58 +01:00
TudorandClaude Opus 5	74e5fffc10	docs(about-blog): implementation plan for the About page and Payload blog ... Nine tasks, each ending in an independently testable deliverable. Two structural findings that the spec did not anticipate, both recorded in the plan. Payload's admin panel ships its own root layout rendering html/body, and Next allows multiple root layouts only when no app/layout.tsx exists — so every existing route moves into an app/(frontend) route group first, on its own, with the full suite as the gate. Route groups are invisible to routing, so no public URL changes. The second finding corrects the spec: adding /cms-api to the FastAPI proxy's exclusion list would be dead code, because that catch-all only ever matches /api/*. The route remap alone is sufficient. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017YmbBhr8s7GusjDE12hrZM	2026-09-02 15:59:44 +01:00
TudorandClaude Opus 5	748ef32180	docs(about-blog): design for a named author, an About page and a Payload blog ... The site reads as synthetic because nobody is accountable for the numbers, no editorial judgement is visible, and the voice is institutional third person. This designs the fix: a named author (first name, photo, explicitly not an education expert), a coded /about page, and a blog backed by Payload CMS running inside the existing Next app. Also fills a hole in the SEO programme, which has eight workstreams and no E-E-A-T or authorship signal on a YMYL corpus. Records two collisions found while designing, both of which fail badly if missed: Payload's default /api route fights the existing FastAPI catch-all proxy, and withPayload() is ESM-only so next.config.js — which carries staging's noindex header — has to become next.config.mjs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FT1Ls4GbgLDXoQX7NAuHGT	2026-09-02 14:43:20 +01:00
TudorandClaude Opus 5	e236669fde	fix(destinations): school rows and the England reference are different grains ... The annual DAG died with a BrokenPipeError from Meltano's log writer, which is several frames from the cause: target-postgres exited first and the tap saw its stdout close. The tap declared primary_keys = [urn, ...] while emitting urn=None for the national rows, and target-postgres turns primary_keys into a NOT NULL constraint. The first national row of the run failed the insert and took the loader with it. Every other tap in this repo keys on non-null columns. Carrying two grains in one stream was the actual mistake, so the fix is to separate them rather than paper over the null: four streams now, with ees_ks4/ks5_destinations_national carrying no urn column at all — a school identifier that is null in every row is a grain mismatch, not a column. The staging models split the same way and the national mart reads the new pair instead of filtering `where urn is null`. Verified against the live API: the school stream yields 135,240 rows over 4,508 schools with no duplicate keys, no null key columns and all 31,382 suppression sentinels intact; the national streams yield 30 and 33 rows with no urn column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-31 21:36:43 +01:00
TudorandClaude Opus 5	264edd2e3a	fix(airflow): a login that survives a container restart ... The simple auth manager generates a random password on first start and writes it to a file, so every restart of the api-server invalidated the last one and the password had to be dug out of the container logs again. The stack now writes that file itself from AIRFLOW_ADMIN_PASSWORD before exec'ing the api-server. Airflow generates nothing when the file already exists, so the login is whatever the stack environment says it is. Written with python rather than echo, so json.dumps escapes a password containing quotes, backslashes or non-ASCII correctly — verified against `p@ss "wo\rd' £5`, which round-trips intact. An unset AIRFLOW_ADMIN_PASSWORD raises KeyError and the container exits. Falling back to a generated password would silently undo the point of the change, and a compose-level `:?` gives the same refusal a readable reason. This does mean the variable MUST be set in Portainer before the next deploy of either stack. Not affected by the two Docker gotchas in the upstream docs: this image has no USER directive so it runs as root, and the file is rewritten from the environment on every start rather than persisted on a volume. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-31 17:38:42 +01:00
TudorandClaude Opus 5	cd2cbe7be6	build(pipeline): install the destinations tap in the image ... meltano install would resolve it from pip_url, but five of the six custom taps are also installed explicitly and a new plugin failing to appear is not something you want to debug from a deploy log. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-31 17:37:46 +01:00
TudorandClaude Opus 5	cbe3a9a772	fix(destinations): the table said 'withheld' for a category that just doesn't apply ... The Share column keyed off `percentage === null`, which is true for not_applicable as well as suppressed, so a destination that does not apply to the school was labelled as one DfE withheld — while the Pupils column in the same row rendered blank. Two columns, one row, disagreeing about what the row was, and one of them making a claim about DfE that wasn't true. Both columns now derive from `status`, which is the distinction the mart, the SQLAlchemy model and the serialiser all preserve deliberately: published shows the figure, suppressed shows the withheld badge, not_applicable shows an em-dash with a title saying so. A published count with no published percentage now derives its share from the cohort rather than falling through to a marker — both halves are published, so nothing withheld is involved, and it is the same derivation the bar widths already use. Verified the new tests fail against the old logic before keeping them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-30 21:48:39 +01:00
TudorandClaude Opus 5	2e9b5c83c5	fix(destinations): the masking pass can no longer exit unsafely ... Review found _mask_for_disclosure could return with its invariant broken and say nothing. add_companion only ever withheld a *published* cell, so a group with one suppressed category and every other one not_applicable — routine in special schools and AP, where few categories apply — left the loop with the lone suppressed cell still solvable. Reproduced on a nine-pupil cohort: one hidden cell, cohort served, residual intact. A disclosure-control pass that fails silently is worse than none, because everything downstream trusts it. The loop now runs until the invariant holds and escalates when no companion exists: the pupil group is dropped from the payload, and an empty block serialises as None so the section is absent rather than an empty shell. disclosure_invariant_holds() is exported so tests assert it directly instead of re-deriving it, and an exhaustive test sweeps all 81 suppression patterns of a four-category group. Also fixes a test that set up six measures and checked one: the loop was `for measure in ["school_sixth_form"]`. It now checks every measure, and against the real invariant — none hidden, or at least two, rather than "at least two", which the five published measures would have failed. No regression on real data: 262 mainstream secondaries, all-pupils bar still drawable on 94%, zero invariant violations, one disadvantaged group dropped by the new escalation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-30 21:35:03 +01:00
TudorandClaude Opus 5	102397fe69	fix(destinations): withhold at the API, not just in the chart ... Code review found the disclosure the whole design was meant to prevent. R1 was written as a rendering rule and implemented as one: canRenderBar stopped the bar being drawn, but GET /api/schools/{urn} still carried the cohort and every published category. cohort - sum(published) returned Whitley Bay's withheld further-education figure exactly — 18 pupils — to any caller, and the RSC payload put it in the browser too. app.py already stated the principle for admission_distance: this endpoint is public and unauthenticated, so a field left in the payload is a published field. The same reasoning applies here and did not get applied. _mask_for_disclosure now closes both identities before serialisation — categories sum to the cohort, and disadvantaged + other = all — by adding secondary suppression until every row and column hides none or at least two. My first attempt picked the smallest published cell as the companion and a new test caught it choosing a zero, which protects nothing: the residual still resolved to 18. The companion must carry pupils. DfE's own aggregates are no longer served. Nothing rendered them, and one spanning a single suppressed component names it. Cost, measured over 262 mainstream secondaries: the all-pupils bar survives on 94% rather than 100%. Zero lone-suppressed groups remain. The e2e helper now tells a missing feature apart from missing data: it fails if the API serves no destinations key at all, and skips if the key is served but the annual DAG has not populated the marts. Failing on the second would redden the staging gate for unrelated commits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 19:58:02 +01:00
Tudor	68a192e430	Merge remote-tracking branch 'origin/main' into feat/ks4-destinations ... # Conflicts: # nextjs-app/__tests__/components/darkThemeSafety.test.ts	2026-08-28 18:45:27 +01:00
TudorandClaude Opus 5	ccd5074c90	test(e2e): destination journeys, including the no-bar rule ... The helper throws rather than skipping when no school returns a destinations block: a silent skip would let a real regression in the sections ride along unnoticed, which is why the distance journeys were changed the same way in 4f01fbd. The disadvantaged journey computes the residual itself and asserts it appears nowhere on the page — the one number the section must never state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 16:16:25 +01:00
TudorandClaude Opus 5	2b4cf20d75	feat(destinations): the post-16 section, replacing the placeholder ... The 'Post-16 destination data coming soon' note is deleted rather than reworded: for a school with no sixth form the truthful statement is that the question does not apply, and a placeholder there implies something is missing. hasSixthForm and .sixthFormNote go with it — nothing else used them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 16:15:21 +01:00
TudorandClaude Opus 5	cef2f77149	feat(destinations): the After Year 11 section ... Question cards over one bar, with the cards acting as a lens on the bar rather than a summary beside it — focusing a card dims everything it is not made of, so the grouping we chose is inspectable rather than asserted. The bar renders only when canRenderBar allows it. Where a category is withheld the section says so and shows the table instead: the categories sum to the cohort, so a bar drawn from the published segments leaves a gap whose width is the withheld figure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 16:14:13 +01:00
TudorandClaude Opus 5	68b6417149	feat(destinations): types and secondary section flags ... Destinations are secondary-only, so the flags go on computeSecondaryFlags rather than computeSchoolFlags. A phase counts as present only when some pupil group carries categories — an empty block would otherwise open a nav entry pointing at a section that never renders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 16:11:37 +01:00
TudorandClaude Opus 5	c5719ef362	feat(destinations): serve destinations without closing the gaps ... The serialiser carries status through and computes no totals of its own. The only aggregates in the payload are ones DfE published itself; whether showing one is safe depends on how many of its components are suppressed, which the frontend decides. The batch guard grows from six tables to eight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 16:10:29 +01:00
TudorandClaude Opus 5	5e5b61987a	feat(destinations): marts, with R3 masking applied at the boundary ... Disadvantaged and other-pupils partition the whole and the all-pupils figure is published, so publishing both halves recovers the suppressed one. The mask is applied in the mart rather than the API so no consumer added later can reach an unmasked combination. The R1 test is a warn, not an error: DfE publishes the recoverable combination and the mart's job is to carry it faithfully. Refusing to close the gap is the API's job and the frontend's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 16:08:20 +01:00
TudorandClaude Opus 5	c564566432	feat(destinations): staging models that keep 'withheld' distinct from 'absent' ... safe_numeric maps every EES sentinel to NULL, which is right for attainment and wrong here: one of those states has to print 'withheld' and the other has to print nothing. A status column carries the difference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 16:07:17 +01:00
TudorandClaude Opus 5	9188626051	feat(destinations): a tap that preserves the suppression sentinel ... EES writes 'c' where a figure is withheld and the categories sum to the cohort, so counts and percentages are emitted as text with the sentinel intact. safe_numeric must never be pointed at them. School rows and the England reference need different establishment pins: at national level selective schools, studios and UTCs are separate populations rather than labels, so leaving establishment open multiplies 30 rows into 190. Two queries per period, each keeping its own level. Verified against the live API for 2022/23: 135,240 school records over 4,508 schools, exactly 30 each, no duplicate keys, 31,382 sentinels kept. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 16:06:37 +01:00
TudorandClaude Opus 5	7ae9ecdc36	feat(destinations): colour tokens, with the absence hatched not coloured ... Activity not captured includes independent schools and moving abroad, so a red segment would be a factual error. The hatch doubles as the secondary encoding that rescues the neutral/blue pair, which separates at only dE 7.6 as flat fills. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 15:58:25 +01:00
TudorandClaude Opus 5	1980d79eee	feat(destinations): the disclosure rules, as executable guards ... The destination categories sum to the cohort and DfE publishes the cohort total, so a lone suppressed cell is recoverable by subtraction. canAggregate, canRenderPublishedAggregate and canRenderBar are what stop a consumer doing that; toBarSegments throws rather than leaving a readable gap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 15:57:46 +01:00
TudorandClaude Opus 5	9423f11567	docs(destinations): implementation plan, ten tasks ... Ordered so the disclosure guards land first and everything downstream consumes them: lib/destinations.ts, tokens, tap, staging, marts, API, then the two sections and the journeys. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 15:56:36 +01:00
TudorandClaude Opus 5	576013d627	docs(destinations): design for KS4 and post-16 destination measures ... The published files suppress individual cells, not whole cohorts, and the categories sum to the cohort — so on 22% of mainstream secondaries the withheld figure can be recovered by subtraction. Three disclosure rules fall out of that, and the rest of the design is downstream of them. Verified against the EES API rather than assumed: both datasets carry school-level rows keyed by URN, with the disadvantage split and every destination category the display needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob	2026-08-28 15:28:09 +01:00
TudorandClaude Opus 5	a7829d591a	fix(map): the popup never took the dark theme ... leaflet.css paints `background: white; color: #333` on the popup card and its tip. LeafletMapInner binds themed content into it — the school name and the headline figure are var(--text-primary) — so in dark mode #E9EEF0 landed on #FFFFFF at 1.17:1. The two things the popup exists to say were the two least readable things on the page. Every other foreground in that popup failed too, from the same cause: the muted phase line at 2.90:1, the vs-national delta at 1.94:1, the Ofsted badge at 1.74:1. Moving the surface onto --bg-card fixes all of them at once — 13.52, 5.45, 8.14 and 9.11:1 respectively. In light mode --bg-card is #FFFFFF, so the popup renders exactly as it did. globals.css already pulls the rest of Leaflet's chrome onto the tokens, and says why: "this matters most in dark mode, where Leaflet's white attribution bar would otherwise sit on a near-black page." The popup was simply missed. The View Details button needed its own fix. It pairs background:var(--status- above) with a literal white label, which theming the card does not reach: --status-above is #36743F in light but #7FCB8A in dark, taking the label from 5.63:1 to 1.94:1. --text-inverse is the token for ink on a saturated fill, and the popup's own Ofsted badge already uses it. darkThemeSafety already guards this defect class, but only inside .module.css. Neither half of this one lives there — the surface is a third party's, the text is inline in a TSX template — so it scanned clean throughout. Two rules added for the layer it could not see. Fixing the grouped-selector blind spot in its rules() helper was needed to write them: taking only a selector's last line discarded every selector in a grouped rule but the final one, which makes a safety guard fail open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FuPUioHpxtaiDNagQvjxyM	2026-08-27 22:34:18 +01:00
TudorandClaude Opus 5	7a16b1b52f	fix(admissions): flag-off pages must not speak for the council ... The secondary admissions section words the absence of a cut-off distance: "<LA> has not published a cut-off distance for this school." That sentence is true when the authority publishes nothing. It is false when the authority does publish and the admission_distance flag is simply off — and off is the current state, so every secondary page with an EES admissions row has been making a claim about a council on our behalf. The backend already draws the distinction the copy needs. /api/schools/{urn} omits the admission_distance key entirely while the flag is dark rather than sending null, precisely so that "we are not publishing cut-offs" stays distinguishable from "this school has no cut-off"; lib/types.ts says so in as many words. The page then collapsed the two with `?? null` before the section ever saw them. So stop collapsing it: thread the raw field to SecondarySchoolSections and word the absence only when the feature is on. Null still gets the sentence naming the authority — that case is unchanged and still tested. Primary pages are unaffected: AdmissionsSection carries no absence copy and renders nothing when there is no figure. DistanceSection already treated absent and null alike; only its type widens. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FuPUioHpxtaiDNagQvjxyM	2026-08-27 21:17:16 +01:00
TudorandClaude Opus 5	e820e7fecd	fix(analytics): the funnel source read a referrer that never changes ... The staging E2E gate has been red since #132 merged (run 1064, and 1066 after it): "a school reached from a location page is attributed to it, not to direct" expects `place`, receives `direct`. #132 fixed a real bug — `/schools/` had no case and fell through to `direct` — but the mechanism underneath it never worked. getNavigationSource read document.referrer, which the browser writes only when a *document* loads. Every internal navigation here is an App Router soft navigation: history.pushState, no new document, so document.referrer goes on naming whatever opened the tab for the whole session. Verified on staging: load /schools/brentwood, click a school, the URL becomes /school/… and document.referrer is still "". So `from` reported `direct` for essentially every in-app journey, not just the ones through the location layer — search, rankings, compare and detail were all being counted as "typed the URL". The unit suite passed throughout because every case set document.referrer directly, which only happens on a full page load. The fix is a module-level trail written by RouteTrail, a render-nothing client component in the root layout. Its lifetime is exactly right: it survives soft navigation, and it dies on a real document load — which is precisely when document.referrer becomes meaningful again, so the two cover each other with no overlap. Reading it skips entries equal to the current path rather than taking the second-to-last. That makes the answer independent of whether the layout effect or the page effect ran first — React orders those by tree position, which is not a contract worth resting a measurement on — and it gives the right answer both when the user returns to a page they came from and on a hard load of a school page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FuPUioHpxtaiDNagQvjxyM	2026-08-27 09:22:37 +01:00
TudorandClaude Opus 5	9a1f56c431	feat(places): say what each school is, not only how it scored ... The location tables carried one column: a percentage. A parent shortlisting from a town page is asking a different question first — does it take my child's age, is it a faith school, does it have a nursery — and the page could not answer any of it. Primary tables gain Ages, Religious character, Nursery and Constituency; secondary tables the same minus Nursery, which is a question about a different intake. An all-through school renders in both groups, so its nursery shows under primary alone. The measure moves to the second column rather than the last. Six columns overflow a phone and .tableWrap turns that into a horizontal swipe; with the measure last, the one number the page exists for is the one scrolled off the screen. Cell rules are the ones the school page already uses, so the two surfaces cannot disagree about the same school: "Does not apply", "None" and "Not applicable" all read as no religious character, and the en-dash age normalisation moves into formatAgeSpan, which formatAgeRange now delegates to. Backend: nursery_provision and parliamentary_constituency were not in the place response. Both are optional GIAS mart columns that data_loader degrades to NULL, and the `in rows.columns` guard keeps a mart the pipeline has not rebuilt working. Also fixes a live bug on the same line: SCHOOL_COLUMNS already ends with latitude and longitude, and the endpoint concatenated them again, so pandas dropped one of every duplicated pair and warned "columns are not unique" on each request. Ordered de-duplication removes the warning and the silent drop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FuPUioHpxtaiDNagQvjxyM	2026-08-27 08:49:14 +01:00
TudorandClaude Opus 5	d1a8596208	feat(analytics): measure the location layer, and stop calling it direct ... The location pages were only half-tracked. Umami counts a pageview for each of the ~3,900 URLs automatically, but nothing else: components/ places contained no track() call, and place_viewed was not even a declared event name. The part that mattered was worse than a gap. getNavigationSource mapped a same-origin referrer to a funnel source and had no case for /schools/, so every school view arriving through the location layer fell through to 'direct' — the bucket you read as "typed the URL, no referrer". W2's whole purpose is funnelling search traffic onto school pages, so the one measurement that says whether it worked was reporting the wrong answer, and reporting it confidently. Verified live against staging: expected "place", received "direct". /schools/ is checked before /school/. They differ by one letter and mean different things — the location layer versus a single school — and a prefix test in the wrong order silently merges them. place_viewed carries kind, slug, phase and school_count. kind is the reason it exists: whether to keep investing in these pages turns on which sort earns engagement, and a pageview cannot say, because all four families share the /schools/ prefix and only the registry knows which is which. It is a client component because PlaceView is a server component; one line in PlaceView covers all four families, since they all render through it. Both E2E journeys were verified failing against staging first — one because place_viewed does not exist there, the other on the exact "place" vs "direct" mismatch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-27 08:23:21 +01:00
TudorandClaude Opus 5	0804566736	fix(test): drop a committed scratch probe, and close a hole in the guard ... Code review, all three findings valid. e2e/tests/__m.spec.ts was a throwaway probe used to measure the mobile hero geometry. It asserts nothing, so it could never fail; it carried a leftover `pick('form').constructor === Object ? null : null` that is null either way and throws if no form matches; and it should never have been committed. Deleted. It survived because `rm -f e2e/tests/__m.spec.ts` ran with the shell already inside e2e/, so the path resolved to e2e/e2e/tests/... — which does not exist, and rm -f is silent about that. `git add -A` then swept it in. I checked `git diff --stat` before committing, which lists only tracked modifications and never shows an untracked file; `git status --short` would have. The scoping guard compared the last line of a rule's prelude against the literal '.filterBar', so a regression written as a selector list — `.filterBar, .other { padding }`, or the same split across two lines — would have walked straight past the test meant to catch it. Selectors are now split on commas and matched individually, and comments are stripped first so a brace inside one cannot desynchronise the parse. Verified against all three shapes: bare, inline comma list, and multi-line comma list. Each is caught; each passes again once reverted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 21:49:48 +01:00
TudorandClaude Opus 5	55363cbd18	fix(suggest): the dropdown reopened on top of the search results ... Three staging-gate failures, two of them one real bug. After a search, the results-page bar still holds the term in its input, so on every render the query was >= 2 characters and the suggestion list opened again — on top of the very results the search had just produced. Playwright reported it as "<li role=option ...> intercepts pointer events" while trying to click the first result; a reader would simply have found their first result unclickable. Both the school-detail and hero-map journeys failed on it, and neither is about autosuggest. Suggestions now answer typing, not the mere presence of a value: `hasTyped` gates the hook, is set on change, and is cleared when a search is submitted or a suggestion is chosen. A pre-filled input makes no request and shows no list. Third failure was my test, not the product. An unphased place page renders one table per phase, and an all-through school legitimately appears in both — so the page's school links were never one alphabetical run. The assertion collected them all together and only passed because no town it picked had held an all-through school. When the data gave Abbots Langley one, Breakspeare School appeared in the primary table and again in the secondary, and the test failed on correct behaviour. It now checks each table separately, and passes against the data that broke it. Guards: a jest test that a pre-filled input neither fetches nor opens (verified by reverting — it is the only one that fails), and an E2E journey that submits a search and then requires the first result to be clickable, which is the reader-facing version of the same thing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 21:38:16 +01:00
TudorandClaude Opus 5	0b15497c09	fix(search): the mobile hero search was indented by a card's padding ... Measured at 390px: the headline and lede sit at x=34, while the search box, the hint and the location link all sat at x=48 and the field was 28px narrower than the copy above it. The 14px came from `@media (max-width: 768px) { .filterBar { padding: 0.875rem } }`. That rule is for the results filter bar, which is a card — background, border, shadow — and needs inner padding. The hero search is not a card: .heroMode strips all of it, padding included. Both selectors are specificity (0,1,0), so source order decides, and .heroMode only wins because it is declared right after .filterBar. A bare .filterBar rule inside a media query comes later and silently wins instead. The two rules directly below this one in the same block were already written as `.filterBar:not(.heroMode)`; this one was missed. Scoping it aligns the search box, hint and location link to the same left edge as the headline and gives the field back its 28px. The location link also carried its own 6px of button padding, so its label started further right than the hint even once the boxes agreed. Pulled back with a negative margin, which keeps the tap target. The guard is a stylesheet test: the failure is a plausible-looking layout rather than a broken one, so nothing short of measuring or looking would catch it. Verified by reverting: it names ".filterBar sets padding". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 21:32:41 +01:00
TudorandClaude Opus 5	3236efa846	fix(map): the hero map's fade to the header was hardcoded white ... The fade between the map band and the school header ramped through rgba(255,255,255,...) and landed on var(--bg-card). In the light theme that is white into white and invisible, as designed. In the dark theme it climbed to 95% WHITE and then met a near-black card, putting a bright band across the full width exactly where the map should dissolve into the title. Fading to the colour the gradient lands on is the whole trick, and it only works if that colour is a token — so --bg-card-rgb now exists in both theme blocks, matching the --hero-ground-rgb precedent. Two more defects in the same file, same cause, found while in there: The controls floating over the map paired a hardcoded white background with color: var(--text-primary), which resolves to #E9EEF0 in dark — near-white text on a near-white button. These deliberately do NOT follow the theme, because the map tiles are light in both, so the ink is now literal too and says why. A themed token is the wrong tool for a surface that never changes. The loading skeleton swept 50% white across var(--bg-secondary), which is a bright flash every 1.4s on a dark page. It now sweeps toward the card colour, a shade lighter than the ground in both themes. The guard is a stylesheet test rather than a render test, because the bug is invisible in the theme it was written for. Verified by reverting each fix in turn: it names .fade and .openHint exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 21:20:57 +01:00
TudorandClaude Opus 5	d5a6db289d	fix(suggest): let the dropdown out of the hero panel ... .heroPanel had overflow: hidden to clip its artwork and scrim to the rounded corners. It clipped the suggestion dropdown too. Measured on staging with the flag on: the list runs 482 to 802, the panel ends at 624 — so 178px of 320 was cut off, about half the options, with nothing on screen to say anything was missing. The two things that actually needed clipping now round themselves: .heroArt gets border-radius: inherit plus its own overflow, and the ::before scrim inherits the radius. Below 860px the artwork is a band flush with the top of the panel rather than a layer covering it, so it takes the top two corners only — inheriting all four would leave it floating with rounded corners against the copy. Nothing else depended on the panel clipping: .valueProps below it is entirely static, so a positioned dropdown paints above it without a z-index fight. The regression test asserts the LAST option is the element actually painted at its own coordinates. toBeVisible() would not have caught this — it checks for a non-empty box and visibility, and an ancestor's overflow clips neither. elementFromPoint catches clipping and occlusion alike. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 21:10:32 +01:00
TudorandClaude Opus 5	0fa1a292c7	fix(api): bound what a forged CF-Connecting-IP can buy ... Code review, both findings valid. The design doc claimed Cloudflare "replaces the header, so a browser cannot forge it", and that only the X-Forwarded-For fallback was forgeable. That is true only for traffic that actually passed through Cloudflare, and nothing in this process can verify that it did. Reaching the origin directly, both headers are equally attacker-controlled — and rotating CF-Connecting-IP mints a fresh rate-limit bucket per request, defeating per-client limits on every endpoint including the DataFrame-heavy /api/schools. Against abuse that is worse than the shared bucket it replaced, which at least capped everyone together. So the ceiling comes back. I dropped it earlier arguing it belonged at Cloudflare; that argument assumed the keying was sound, and it is not. GlobalRateLimitMiddleware counts all /api/ traffic in a fixed window against a total, independent of client identity, outermost so it refuses before any work happens. Written by hand because slowapi cannot express a global cap: default_limits and application_limits are both keyed by key_func, and the latter needs middleware this app does not install. It does not make the header trustworthy — it makes trusting it survivable. The real fix is Authenticated Origin Pulls or an origin firewall, now documented in DEPLOY.md as the open gap it is. 127.0.0.1 is exempt: the healthcheck curls localhost from inside the container, and starving it would restart the container and turn a load spike into an outage loop. Keyed on the peer address, never the Host header, which the caller sets. Second finding: suggest_schools_typesense promised "never raises" while the parsing loop sat outside the try, so int(None) on a malformed document would have made a keystroke a 500. The loop now skips bad rows rather than dropping the whole list — and a hit with no document no longer becomes a suggestion pointing at /school/0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:56:32 +01:00
TudorandClaude Opus 5	d2115364ae	test(e2e): autosuggest journeys, gated on the flag ... Feature state is read from its observable effect — whether the search box is a combobox — because /api/flags is denied to the public on purpose. Same approach as the distance journeys. The three endpoint tests are ungated: /api/suggest is live whether or not the UI is, which is what lets it be smoke-tested in an environment where the feature is still dark. The flag-off journey asserts the plain search still works, not just that the combobox is absent. Verified against staging, where the flag is off: it passes and the flag-on journey correctly skips. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:39:33 +01:00
TudorandClaude Opus 5	28cf0a342c	feat(suggest): wire autosuggest into the search box behind a flag ... Off means off — no combobox role, no listener, no fetch. A test asserts the absence of the request, not just the absence of the dropdown, because a hidden-but-fetching control would still be spending the rate limit on a feature nobody can see. Enter with no active option falls through to the form's submit handler and searches the typed text exactly as before. The existing behaviour is preserved, not replaced, and that has its own test. Suppressed once the value parses as a postcode: the box takes a name OR a postcode, and suggesting schools during postcode entry fights the user. .omniBoxContainer gains position: relative — the dropdown is absolutely positioned and without it would have anchored to the page instead. Four render sites, all wired: page.tsx renders HomeView in the success path AND the catch fallback, and HomeView renders FilterBar as hero AND sticky. Missing any one would make the flag silently do nothing somewhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:38:52 +01:00
TudorandClaude Opus 5	d88e77f459	feat(suggest): the dropdown, with combobox ARIA ... Presentational only — it fetches nothing and owns no state, so the fetching rules and the ARIA rules can be read separately. onMouseDown, not onClick. The input's blur handler closes the list and blur fires before click, so a click handler never runs: the classic bug where a dropdown works perfectly by keyboard and is dead to the mouse. The plan's CSS guessed at token names like --color-surface. The real tokens are --bg-card, --border, --text-muted, --bg-secondary and --shadow-soft, and all five are redefined in the dark theme — invented names would have silently fallen back to hardcoded light values and broken dark mode. Local authority is rendered because there are many schools called 'St Mary's'; a list without it is unusable for exactly the query autosuggest exists to serve, which is what the test asserts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:36:39 +01:00
TudorandClaude Opus 5	06eb433db5	feat(suggest): debounced, abortable suggestion hook ... The AbortController is correctness, not economy. Without it a slow response for 'st' can land after the fast one for 'st marys' and replace a correct list with a stale one — the classic autosuggest race. No cache: 'no-store', unlike the compare modal's search. This is the one endpoint where prefix queries repeat most across users, so discarding the browser cache and the backend's ETag 304s would be throwing away the cheapest win available. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:35:27 +01:00
TudorandClaude Opus 5	1a6d349dad	feat(suggest): GET /api/suggest, cacheable and DataFrame-free ... A dedicated endpoint rather than a mode of /api/schools, because that path filters and sorts 25,000 pandas rows per query while holding the GIL — affordable once per search, not once per keystroke. A test asserts the distinction directly by making load_school_data raise and requiring the endpoint to answer anyway. Nothing errors on ordinary input: a short query, no matches, or Typesense being down are all 200 with an empty list. Cached deliberately. Prefix queries repeat enormously across users and school names change once a year, so s-maxage plus the existing ETag middleware turns most keystrokes into 304s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:34:34 +01:00
TudorandClaude Opus 5	75d3534d82	feat(suggest): Typesense rows for autosuggest, no DataFrame ... search_schools_typesense returns URNs, which forces the caller to hydrate from the 25,000-row in-memory frame. Every field a suggestion needs is already in the Typesense document, so this returns documents and the caller needs no pandas at all — the difference between a query that can run per keystroke and one that cannot. Never raises. Typesense unreachable or erroring gives an empty list, because a dropdown that quietly stops appearing is the right failure for a keystroke path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:33:42 +01:00
TudorandClaude Opus 5	ff041544f2	fix(api): rate-limit per caller, not per proxy ... The limiter keyed on request.client.host, which in staging and prod is the Next container — the backend has no published ports and nothing else can reach it. So every browser user on the site shared one 60/minute bucket per route. Measured against staging: 70 concurrent requests to /api/schools returned exactly 60 OK and 10 refused, from one machine. CF-Connecting-IP first. Cloudflare fronts both environments and overwrites any client-supplied value, which a parsed X-Forwarded-For chain does not guarantee. The XFF fallback is forgeable only from inside the Docker network. Named rather than hidden: the shared bucket was an accidental global throttle on a single-process backend, and correct per-user keying removes it. A real global ceiling belongs at Cloudflare, which is already in the path; slowapi cannot express one without a second Limiter and middleware this app does not install. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:32:58 +01:00
TudorandClaude Opus 5	59265f78b6	docs(flags): Unleash does not create flags by itself ... The runbook said a flag 'appears in the Unleash UI after the backend has evaluated it once'. That is wrong. SDKs read definitions from the server and never register anything, and metrics for an unknown flag are discarded — so a declared flag is evaluated on every request, stays False forever, and never shows up until someone creates it by hand. Found the way these things usually are: staging had been running the flag code for a while and the UI was still empty. Also names the environment trap while here — each stack's token is scoped to one environment, so toggling the other does nothing visible and looks like the flag is broken. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 20:11:45 +01:00
TudorandClaude Opus 5	6e0a278340	docs(suggest): implementation plan, eight tasks ... Self-review caught three defects in the plan. .omniBoxContainer, the wrapper the dropdown positions against, does not declare position: relative — without it the list anchors to the page. The postcode suppression test typed character by character, so it would have asserted no request while 'NW1' legitimately fires one; it now sets the value in one go. And the Enter-submits-search assertion needed waitFor, because updateURL pushes inside startTransition. Task 1 is the one to review hardest: it is the only unflagged change and it alters rate limiting for every endpoint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 19:48:39 +01:00
TudorandClaude Opus 5	e651dd0d65	docs(suggest): the in-app global ceiling would not have worked ... Reading slowapi rather than assuming: default_limits and application_limits are both evaluated with the same key_func, so they are per-client across routes, not global. And application_limits only apply 'if in_middleware' — this app installs no SlowAPIMiddleware, so they would never have fired at all. A genuine global cap would need a second Limiter with a constant key plus that middleware. Cloudflare is already in the path on both environments and does this at the right layer, so the ceiling is named as a follow-up there rather than built badly here. The risk that leaves is stated plainly in the risks section instead of being papered over with a mechanism that does not do the job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 19:44:21 +01:00
TudorandClaude Opus 5	22c113fc29	docs(suggest): design for school autosuggest ... The load-bearing finding is not about autosuggest. The rate limiter keys on request.client.host, which in staging and prod is the Next container — so all browser users share one 60/min bucket per route. Measured against staging: 70 concurrent requests gave exactly 60 x 200 and 10 x 429. Eight concurrent searchers would 429 the site once each keystroke costs a request, so the keying fix is part of this work. Both environments are behind Cloudflare, which sets CF-Connecting-IP and overwrites any client-supplied value — trustworthy in a way a parsed X-Forwarded-For chain is not, and the backend is unreachable except through the Next proxy. Named honestly: the shared bucket has been an accidental global throttle on a single-process backend, so correct per-user keying removes a protection. A global ceiling ships with it rather than instead of it. Suggestions come from Typesense alone. The existing search path filters a 25,000-row DataFrame per query, which is exactly the cost a keystroke endpoint cannot pay, so there is deliberately no DataFrame fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-26 19:41:00 +01:00
TudorandClaude Opus 5	413d86cc3c	chore(flags): wire UNLEASH_URL and the SDK cache volume into the stacks ... Both variables default to empty, so an environment without Unleash has every flag off — the correct dark state rather than a boot failure. The cache volume is the mitigation for the one real regression risk in this design: the SDK evaluates everything False until it syncs, so a backend cold-starting with an empty cache while Unleash is unreachable would make a *released* feature disappear. On a named volume the disk cache survives a restart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:58:29 +01:00
TudorandClaude Opus 5	4f01fbdedb	test(e2e): make the distance journeys fail loudly, not skip quietly ... The existing distance journeys all skip when no school has a published figure, which is right when the feature is off — and wrong when it is supposed to be on and is silently broken, because that shows up as a green run full of skips. The new gate fails in exactly that case. Feature state is read from the data, not from /api/flags: the public proxy denies that path on purpose, since it names unreleased features. Presence of the admission_distance key is the observable effect. Verified against staging, where the feature is currently on: the on-gate passes, the off-gate skips, the existing eight distance journeys are unaffected. One honest caveat — the /api/flags check passes on staging today because that image predates the endpoint, not because the denylist works. The denylist itself is covered by the jest unit test; this is defence in depth and becomes a real assertion once deployed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:57:34 +01:00
TudorandClaude Opus 5	c3ba7aae0d	feat(flags): ship last-distance-offered dark behind a flag ... One gate, at the source. The frontend needs no change: DistanceSection already returns null when distance_m is missing, and the admissions block already conditions on (admissions || admissionDistance). Only 57 local authorities publish cut-offs, so the off-path is the commonest path on the site and is well covered already. Absent, not null. /api/schools/ is public and unauthenticated, so a field left in the payload is a published field — the reasoning already recorded in c9a1892 when history was withheld. The two are also different claims: null says this school has no cut-off, absent says cut-offs are not being published at all. The frontend type now says so. The feature is on main and live on staging and has never reached production, which is what makes it the right first consumer: the flag lets the code promote without the feature appearing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:56:33 +01:00
TudorandClaude Opus 5	54a30de0d8	feat(flags): server-side getFlags for the frontend ... Ships without a consumer, deliberately. The first flag needs none — the backend withholds the field and the page follows — but 'UI elements on existing pages' is one of the three surfaces this capability exists for, and a flag layer that cannot gate one is incomplete. Never throws: an unreadable flag is a dark one, which matches the backend's fail-closed default. A page that 500s because the flags endpoint blinked would be a worse outcome than a hidden feature. Reading flags pins the calling route to a 300s ISR floor, since Next takes the lowest revalidate among a route's fetches. That matches what /school/[slug] already sits at, and it is the same property that makes a flip propagate without a webhook. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:55:32 +01:00
TudorandClaude Opus 5	c30ad1db07	feat(flags): serve /api/flags, and keep the public proxy off it ... The endpoint and its exposure control ship together on purpose. The moment /api/flags exists, app/api/[...path] forwards it — and the response names every unreleased feature the codebase knows about, along with whether it is on. Publishing that is the opposite of shipping dark. Denied on an exact first-segment match, not a prefix, so /api/flagship does not go down with /api/flags. Next reads the endpoint server-side over the Docker network, which never transits the public proxy. jest.setup.js now guards its browser globals. It runs for every suite, including the one that declares @jest-environment node to exercise the route handler — NextRequest needs Fetch API globals jsdom lacks, and there is no window there to define matchMedia on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:54:26 +01:00
TudorandClaude Opus 5	7424cef7c6	feat(flags): the registry and a fail-closed Unleash client ... Unleash holds flag state; it does not hold the list of flags. REGISTRY is that list, because the SDK evaluates an unknown flag to False and without a registry that is an undeclared False — indistinguishable from a typo in a flag name. Fail-closed throughout, and never raises: an unset UNLEASH_URL, an unreachable server, a client that throws, an undeclared name — all False. A flag layer that can 500 a request path or stop the API booting is worse than one that is switched off. Every flag defaults to False, with no per-flag override, because a flag that defaults on is a kill switch and this is deliberately not one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:52:55 +01:00
TudorandClaude Opus 5	01ccbb8e82	feat(flags): add the Unleash stack and its runbook ... Its own Portainer stack, belonging to neither application stack: a staging redeploy must not be able to disturb production's flag state. One instance serves both. OSS Unleash ships development and production environments with environment-scoped client tokens, so the same flag holds independent state in each — which is what lets a feature be on in staging, where the E2E journeys exercise it, while production stays dark. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:51:35 +01:00
TudorandClaude Opus 5	c339c2f1a1	docs(flags): implementation plan, eight tasks ... Task 1 is the Unleash stack and ends with a human step — the Portainer deploy and the token generation cannot be automated from here. Nothing else blocks on it: an unset UNLEASH_URL means every flag is False, which is what local development and CI get, so the whole suite runs without a flag server existing. Self-review caught three defects in the plan itself. get_supplementary_data takes (db, urn), not (urn), and the test DataFrame was minimised to the point where the endpoint would have failed for reasons unrelated to flags — both now copy the known-good shape from test_school_details.py. The proxy test needs the node jest environment, since NextRequest wants Fetch API globals jsdom does not provide. And the e2e off-state check hardcoded a URN, so a 404 page would have satisfied it without proving anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:46:15 +01:00
TudorandClaude Opus 5	e2ca3d79f9	docs(flags): drop the webhook — the seven-day premise was wrong ... Next uses the LOWEST revalidate among a route's fetches, not the segment value. School pages fetch school details at 300s and place pages fetch national averages at 3600s, so the effective ISR period is five minutes and one hour respectively — not the seven days the segment declares. A flag flip therefore propagates on its own, well inside the monthly, by-hand cadence these flags are for. That deletes two webhook integrations, a revalidate route, a secret-in-query-string scheme, an idempotency requirement, and the rule that every fetch carry a cache tag — which was the part most likely to rot as fetches are added. Two constraints survive: a flag must never gate content on a force-static page, because app/admissions never revalidates; and a route-family flag must rebuild the sitemap, deferred with the route case since no flag in scope touches it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 10:40:35 +01:00
TudorandClaude Opus 5	c2364bf09e	docs(flags): design for a ship-dark feature flag layer ... Unleash self-hosted in its own Portainer stack, with FastAPI holding the only SDK and Next reading flags through a tagged fetch. The two hard parts are consequences of putting flag state in a service rather than the repo: main stops being the whole truth about what is on, and a flag can now change without the deploy that would have cleared the caches. A code-declared registry bounds the first; webhook-driven revalidateTag handles the second. Cache tagging is deliberately coarse — every server fetch carries the flags tag, not just the flags fetch itself. The first consumer proves why: admission_distance changes the shape of /api/schools/{urn}, so a narrow purge would leave ~25,000 school pages serving the pre-flip render for a week, invisibly. First consumer is the last-distance-offered feature, which is on main and staging and has never reached production. It needs one gate, at the API, because the frontend already no-ops on a missing field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-23 09:47:58 +01:00
TudorandClaude Opus 5	d1358cc00f	fix(places): phase links must stay in their own namespace ... Every place page built its phase links as /schools/[slug]/[phase], the shape that belongs to towns alone. On an authority page that pointed into the town namespace. For 87 of the 151 authorities the target does not exist and the link 404s; for the other 64 it resolves to the town of the same name — a different set of schools, which is precisely the near-duplicate the two namespaces were introduced to prevent. On an outcode page it 404s outright. Two causes behind it, both a rule written twice and inherited by only one of the places that needed it. The authority phase route was in the spec and dropped by the plan, which built the three bare routes and no fourth. The sitemap is generated from the place registry, which was right about them all along, so 302 authority phase URLs have been submitted to Google and every one 404s. Adding the route makes the sitemap true and serves a real query — admissions are authority-run, so "primary schools in Kent" is how a parent searches before they have settled on a town. The outcode variants were the opposite: the registry computed phases for outcodes although the spec gives them no route, and the sitemap knew to skip them while the API did not. The registry now decides alone, and the sitemap's duplicate of that rule is gone. Also: an authority under the five-school threshold has no page, so the API sends a null slug for it and the page names it without linking. Two English authorities are in that position. It was unreachable in today's data — verified across the EC and TR outcodes — but the thin place redirect would have sent a reader to a 404 the year it isn't. The e2e journey now walks every /schools link a page of each family emits and requires a 200, which is the check that was missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-22 17:22:11 +01:00
TudorandClaude Opus 5	9cc87c41bb	fix(places): a phase page needs results, not merely publishable schools ... Asked where schools with no results should sit in an alphabetical list, and found that some pages were almost entirely made of them. The per-phase threshold counted schools that were publishable — a result OR an Ofsted grade — while a phase page exists for its results column. /schools/kent/primary published with none of its five rows carrying a result; Minehead had one of seven, Buntingford one of five. Forty-four phase pages were majority-blank. It is the same rule as "no page without a local average", which was written into the spec as a thin-page control and never extended per phase. The threshold now counts schools with a result for that phase. It gates whether the page exists; it does not filter rows — a page that publishes still lists every school of the phase, because someone looking up a school by name has to find it whether or not it published results. 126 of 1,012 variant pages stop publishing: 62 primary, 64 secondary. Every one of them was a table with too little in it to be worth a page. The ordering itself is unchanged: pure A-Z, blanks interleaved. A school sits where its name says it does, and at roughly a tenth of rows that reads fine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-22 00:14:31 +01:00
TudorandClaude Opus 5	8967966eef	feat(places): list schools alphabetically on place pages ... Someone on a place page is usually looking for a school they can name, so the order should serve scanning for it rather than ranking. /api/rankings keeps its league-table ordering; this is a place-page decision, not a site-wide one. Sorted case-insensitively, or a capitalised name would sort ahead of every lowercase one. The change made five pieces of copy untrue, so they go with it. The phase variant titled itself "— Ranked", and all four route families described themselves as "ranked by SATs and GCSE results". A page that opens by claiming an order it does not keep is worse than one that claims nothing. The ItemList markup carried `position` with no declared order, which reads as a ranking. It now declares ItemListOrderAscending, so the structured data says what the table does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-22 00:09:58 +01:00
TudorandClaude Opus 5	4e82e6c916	fix(e2e): three assertions that were wrong about correct behaviour ... The staging gate was red on three journeys. All three were faults in the tests; the site was behaving correctly in each case. Next normalises canonical URLs against trailingSlash:false, so the homepage ships "https://www.schoolcompare.co.uk" with no slash while every other route keeps its path. Both address the same document. The test hardcoded the slash and so failed only on the root — /rankings and /admissions passed throughout, which is what made it look like a homepage bug rather than a test bug. Compared with trailing slashes stripped from both sides. The robots.txt assertion matched "Disallow: /" anywhere in the file and tripped over the AI-crawler groups Cloudflare injects — ClaudeBot, GPTBot, Amazonbot and six others all carry a blanket disallow, deliberately, and none of them is Googlebot. It now parses the file into user-agent groups and checks only the "*" group, which is also the thing the test was always trying to say: Google may crawl the page, so it can see the noindex header. Both were the same mistake as the doubled brand: asserting a naive string rather than the semantics, and asserting against what the code assembles rather than what the page renders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 23:51:34 +01:00
TudorandClaude Opus 5	bb2f7a5841	fix(places): address review, and merge places GIAS spells more than one way ... Two findings from review on #121, plus a third the review prompted. The cap at three authorities silently dropped the fourth in exactly the case where the information matters most — a genuinely fragmented place — and contradicted the stated goal of naming every authority a place sits in. It is gone. The share rule was always the real limit and already bounds the list at ten. Measured against the live corpus, one town would have been truncated today: LONDON, split evenly between Hackney, Lambeth, Westminster and Lewisham. parent_authority used mode() while authorities used value_counts(), and on an exact tie pandas does not guarantee the two pick the same name, so the 301 could have pointed somewhere other than the authority named first on the page. The parent is now derived from authorities[0]: one computation, one answer. It also inherits the sentinel filter, so a place can no longer redirect to /schools/authority/does-not-apply. Chasing the truncation case surfaced a worse bug. Places were grouped by raw town value, but the registry is keyed by slug, and GIAS spells the same place several ways. Five town slugs come from more than one spelling: "London" (1,819 schools) and "LONDON" (12) both slugify to `london`, so the later group simply overwrote the earlier one — /schools/london could have shown twelve schools, silently, depending on row order. Weston-super-Mare was split 14/19 across two spellings and Newcastle-under-Lyme across three. Grouping is now by slug, and the display name is the most common spelling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 22:42:18 +01:00
TudorandClaude Opus 5	1cb5314c53	feat(places): name every authority a place sits in ... SW19 is mostly Merton but partly Wandsworth, and the page said only Merton. The cause was one field doing two jobs: _parent_authority takes the modal authority, which is right for a 301 target and wrong as a statement about where a place is. This is not a corner case. A quarter of viable outcodes (425 of 1,760) and a third of viable towns (263 of 783) cross an authority boundary — Bedford the town spans Bedford and Central Bedfordshire. Place now carries `authorities`, every authority holding at least a tenth of the schools and at least two of them, largest first. parent_authority stays single and unchanged, because a redirect still needs one target. The share threshold exists because GIAS carries postcode errors: EN6 lists two Shropshire schools among fourteen in Hertfordshire, and a bare "any authority present" rule would print those as though they were real. A place too small or too fragmented to clear the threshold still names its largest, so the page never goes silent about where it is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 22:40:58 +01:00
TudorandClaude Opus 5	dbb74d9b60	fix(places): align the measure column's heading with its values ... The heading sat on the right edge of the column and every value on the left. A specificity collision, not a layout problem: the two were aligned by different selectors and only one of them won. .table td (0,1,1) text-align: left <- won for the value .num (0,1,0) text-align: right <- lost .table th:last-child (0,2,1) text-align: right <- won for the heading The heading and the value cell now share one class and one rule, so they cannot drift apart again whatever else changes around them. The column also stretched to half the table. It now hugs its content with width:1% and nowrap, so the school name takes the remaining width — which is what made the gap read as misalignment on a wide screen, and what crowded the name column on a narrow one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 22:15:02 +01:00
Tudor	3365ebcb3a	fix(places): phase-grouped tables, plain-English measures, styled links ... Three presentation faults on the place pages, all found by looking at a rendered page rather than at a test. An unphased place page showed one primary-only measure for a list holding both phases: 8 of 27 rows on /schools/brentwood were blank, because secondaries have no reading-writing-maths score. Picking the other measure would only have inverted which rows were empty, and putting both in one column would have mixed a percentage with a 0-90 score. Each phase now gets its own table, so a blank cell means the school genuinely has no published result — which is worth saying, and now says "Not published" rather than a bare dash. "RWM expected" was invented here. The site already names the measure in METRIC_DEFINITIONS, surfaced at /api/metrics: "Reading, Writing & Maths Combined %". The heading now reads "Reading, writing & maths" with the full definition in the tooltip. Links carried no class at all, so they rendered as default blue underlined browser links beside a site that styles table links as body colour with a brand hover. They now follow RankingsView's convention, and running-copy links take the brand colour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 21:56:06 +01:00
TudorandClaude Opus 5	24e114dee7	fix(places): stop the place titles doubling the brand ... Every place page shipped as 'Schools in Brentwood - Compare 27 Schools | schoolcompare | schoolcompare'. The root layout's title template appends '| schoolcompare' to any plain-string title, and all four place routes already carried the brand. W8 opted the other routes out with an absolute title; the place routes were written afterwards and did not inherit the lesson. ~2,600 titles affected, and the repetition pushed them past Google's truncation point, so the doubled brand displaced real words in the result. An e2e journey now asserts no title repeats the brand, across the static routes and a place page, so this cannot come back on a route added later. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 21:43:32 +01:00
TudorandClaude Opus 5	6f749ed21f	fix(places): submit and link the phase variants ... /schools/[place]/[phase] shipped as routes but reached nothing. The sitemap emitted one URL per registry entry and the registry had no phase dimension, so ~950 pages were absent from every sitemap — and PlaceView did not link them either, leaving them reachable by nothing at all. That is the query shape the baseline actually showed: 'primary schools in beccles', 'secondary schools in brentwood'. Publishing the routes without a path in meant building for the demand and then hiding from it. Place now carries phase_urns so the per-phase threshold can be applied without re-querying, the sitemap emits a variant wherever a phase clears the threshold on its own, and the API exposes the qualifying phases so the place page links only variants that exist. Outcodes are excluded: nobody searches 'primary schools in SW11' and those routes do not exist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 20:43:07 +01:00
TudorandClaude Opus 5	d3c63ccc6d	fix(places): a locality collision must not break the sitemap ... Sitemap regeneration failed on staging. 'richmond' in the curated locality list collides with the GIAS town Richmond in North Yorkshire (37 schools), the registry raised, and the admin endpoint 500d — taking down sitemap generation for all 25,000 school pages over one bad row of curated data. The guard now skips the colliding locality and logs an error. Skipping still achieves what the guard was for — a locality never silently shadows a town — without letting curated data break the site. That matters beyond this bug: GIAS town names change with no code change here, so a raise could fire spontaneously in production later. Also removes four localities that were London boroughs rather than districts. Hackney, Islington, Greenwich and Ealing are local authorities with 104, 72, 108 and 115 schools and already have authority pages; a locality defined by two or three outcodes would have been a partial near-duplicate of one — the thin-content failure the two-namespace design exists to avoid. A test now guards the whole borough list. Validated against the live corpus: 15 localities, no town collisions, no authority duplicates, all 15 clear the threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 20:20:52 +01:00
TudorandClaude Opus 5	6b871ce1e9	feat(places): ItemList and BreadcrumbList, and the e2e gate ... ItemList tells Google the page is a ranked set rather than prose; BreadcrumbList puts the place in a hierarchy. School URLs in the markup are absolute on the canonical host, since a relative URL in JSON-LD is ambiguous. Eight journeys covering all four families, the two-namespace guarantee, the threshold, the canonical, the sitemap and the local-versus-England line — the last because that comparison is the reason these pages are not a name dropped into a template. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:19:08 +01:00
TudorandClaude Opus 5	c981d89137	feat(places): town, locality, authority and outcode routes ... Every generateStaticParams is gated behind PRERENDER_PLACES and wrapped in the same try/catch the school route uses. The plan claimed authority pages were 'few enough to always prebuild' — but few enough still means the API must be reachable at build time, and in CI it is not: the build failed with ECONNREFUSED rather than degrading to ISR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:17:57 +01:00
TudorandClaude Opus 5	de5e790112	feat(places): place page client and view component ... One component for all four families: they differ in what fills the registry, not in what the page shows, so a second would be a second place to forget the same change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:15:10 +01:00
TudorandClaude Opus 5	42138fc402	feat(places): submit place and outcode sitemaps ... Separate children per family so Search Console reports the location layer's indexation apart from the school pages' — which is the point of the index built in W1, and the number the stop condition watches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:13:55 +01:00
TudorandClaude Opus 5	c5af476213	feat(places): /api/places registry and place detail endpoints ... The registry is cached for the process and reset by the same admin endpoint that rebuilds the sitemaps, so places and sitemap always describe the same corpus rather than drifting apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:13:05 +01:00
TudorandClaude Opus 5	de853b90b3	feat(places): London localities and postcode districts ... The GIAS town field puts 1,819 London schools under the single value 'London', so it cannot answer 'schools in Battersea' — a query that appears in the baseline. No single field can: parliamentary constituency gives Battersea but not Canary Wharf, admin_ward gives Canary Wharf but not Battersea, and neither gives Clapham or Shoreditch. So a locality is curated, defined by the postcode districts it covers, which needs no new ingestion. A locality may not shadow a published town: the registry raises rather than silently costing a page that carries real demand. One below the threshold is logged rather than raising, because a locality can legitimately be too small. The pipeline seed mirrors the module, with a test guarding the drift — the same arrangement gias_codes has, and for the same reason: the backend image does not contain pipeline/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:11:56 +01:00
TudorandClaude Opus 5	759d9f5cea	feat(places): registry of towns and authorities ... Two namespaces because 67 town names collide with an authority name and neither set contains the other — postal towns cross authority boundaries, so Bedford the town holds 104 schools against the authority's 86. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:10:37 +01:00
TudorandClaude Opus 5	555d3f0a7d	docs(seo): implementation plan for the W2 location layer ... Seven tasks: the place registry, London localities and outcodes, the places API, per-family sitemaps, the shared place view, the four route families, and structured data plus the e2e gate. Two things the plan corrects against the spec. The backend image does not contain pipeline/, so the curated locality list cannot live only in a dbt seed — it follows the gias_codes.py precedent instead, canonical in backend with the seed as a mirror. And NationalAverages is nested by phase rather than flat, which the first draft read wrongly and would have rendered every page without its England comparison. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:09:35 +01:00
TudorandClaude Opus 5	ecc847091c	docs(seo): design for the W2 location layer ... Supersedes the original spec's W2. The Search Console baseline inverted its ordering: every measured location query is town or district level, none is an administrative area, and phase is part of the query rather than a filter. Two problems the original design did not anticipate. 67 viable towns share a name with a local authority, and the authority is the larger set in only 43 of them — postal towns cross authority boundaries, so neither can absorb the other. Two namespaces resolve it by construction. And the GIAS town field collapses 1,819 London schools into one value, which a curated locality-to-outcode seed solves without new ingestion. Sizing is measured against the live 25,185-school corpus rather than estimated: 783 viable towns, 1,760 outcodes, 154 authorities. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 18:09:35 +01:00
TudorandClaude Opus 5	c0547c45e5	feat(seo): rewrite the C1 snippets to earn the click (W8) ... The baseline says these pages already rank and are not clicked. 'compare school performance' sits at position 6.1 with 0.43% CTR; 'compare schools' at 7.2 with 0.87%. The brand query 'school compare' draws 9.16% from the same neighbourhood of the same results page, which rules out a ranking explanation — when the snippet gives a reason to click, it gets clicked. These SERPs are owned by the DfE's own 'Compare school performance' service. The old title put a lowercase brand nobody searches for in the most valuable pixels, then a near-paraphrase of that service's name. Beside the government's own result it read as a lookalike. Intent in the title, differentiator in the description. Titles now match what people type, and the descriptions carry the one fact gov.uk does not publish: how close you had to live to get a place. /compare deliberately takes the tool phrasing rather than the homepage's, so the two pages stop competing for one phrase. The root layout's default and Open Graph copy were saying something different again; they now agree. No hard school counts in any of it. The corpus moves with every data refresh and this repo has already shipped one copy bug of that kind. Tests guard the mechanics — SERP length, intent keyword, the differentiator, no brand-first title — and leave the wording free to iterate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 00:21:08 +01:00
TudorandClaude Opus 5	07c97a46c5	fix(seo): a school is publishable on any year's results, not the latest ... _school_sitemap_rows tested only the latest year's row, which quietly dropped every school with results in its history but a null row for the most recent year — a school that stopped reporting, or whose figures were suppressed for small-cohort disclosure. The Mallard Academy (150367) is the case that caught it: real KS2 results for 2015-16 through 2018-19, then null rows from 2022-23 on. Its detail page shows all four years; the sitemap omitted it. Sampling 40 of the 2,206 excluded schools found 4 like this, so roughly 220 real pages were being withheld. Publishable is now a property of the school, computed across every row, while lastmod still comes from the latest row so the most recent Ofsted date wins. The field list is a module constant shared with _has_publishable_data so the two checks cannot drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-21 00:04:32 +01:00