5 Commits

Include renames

Author	SHA1	Message	Date
TudorandClaude Opus 5	3aad5101a8	fix(data): drop the Welsh school GIAS does not type as Welsh ... Beechwood College (URN 142458, Sully, CF64 5SE) survived the England-only filter. GIAS types it a Special post 16 institution (32), not a Welsh establishment (30), so filtering on establishment type alone left it behind — the last Welsh school on the site, and the reason Vale of Glamorgan was still in the authority list. The earlier verification claimed the type codes mapped onto the Welsh authorities in both directions. That was checked exhaustively for Cardiff and by count for three others; Vale of Glamorgan was never checked, and it was the one that did not hold. LA code is the reliable discriminator: GIAS gives the 22 Welsh unitary authorities the contiguous block 660-681, which English authorities never use. Filtering on postcode would have been wrong — Redbrook, Tutshill, Wyedean and two other Gloucestershire schools carry NP16/NP25 postcodes because Royal Mail areas straddle the border, and they are English schools with English data. An e2e test now pins those five so the fix cannot be simplified into a postcode filter later. The 660-681 range is documented GIAS structure this project cannot verify from its own data, so the range filters and the authority NAME checks: if the range is ever wrong, a Welsh authority reappears in assert_england_only_schools and the pipeline fails loudly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-20 22:54:24 +01:00
TudorandClaude Opus 5	7650b16f62	feat(data): publish England only, dropping Welsh and overseas establishments ... GIAS ships the whole UK plus overseas and offshore establishments. None of them carry comparable DfE performance data — Wales does not publish on the English measures at all — so every one of these pages rendered with null results, null Ofsted and null phase. There were 2,036 of them: 1,569 Welsh, 123 offshore (Jersey, Guernsey, Isle of Man, Gibraltar), 316 British schools overseas and 28 service children's schools. All 2,036 were being submitted to search engines, alongside 29 local authorities that existed in the filters purely to list them. Filter at the mart boundary rather than the view layer. dim_school and dim_location both exclude TypeOfEstablishment in {25, 26, 30, 37}, listed once as vars.non_england_school_type_codes. Everything downstream reads those two marts — search, the school page, /api/filters, rankings, Typesense and build_sitemap() — so one filter removes them from the site and the sitemap together, and Typesense drops them on its next rebuild since it recreates the collection and swaps the alias rather than upserting in place. coalesce rather than a bare NOT IN: a null type code would make the predicate null and drop the row silently, and an unknown type is not grounds for exclusion. No establishment has a null type today, but a future GIAS refresh could ship one and the loss would be invisible. assert_england_only_schools guards both directions: no excluded type survives in dim_school, and dim_location holds no URN dim_school lacks — the API inner-joins them, so the two filters drifting apart would silently shrink the corpus. Corpus goes from 27,229 schools to 25,193, and the authority list from 182 to 153. The 1,569 Welsh URLs now 404. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj	2026-08-20 21:46:28 +01:00
TudorandClaude Opus 5	88c653215d	feat(admissions): show the last distance offered where councils publish it ... Adds the cut-off distance a parent actually asks about — "how close do we need to live?" — end to end: a Singer tap, dbt staging and mart models, an Airflow DAG, and a tile on both detail templates. 3,597 schools across 57 local authorities carry a figure; the rest are unchanged. There is no national source for this. Each LA publishes its own cut-offs in its own format, and the collected CSV is transcribed from PDFs, spreadsheets and web pages — so most of the work here is deciding what is safe to show. Data * tap-uk-school-distance loads the CSV verbatim into raw. Keyed on (urn, year, school_name), because school_name carries the admission route: (urn, year) alone collides on 118 keys and a reload would have silently dropped every band but one. * stg_school_distance applies a 25 m – 25 km plausibility band. The source contains 0.0-mile rows (published where a school filled on a higher criterion), 1-metre cut-offs, and one reading 533 miles — ~4% of rows, all of which would put a visibly wrong number on a live page. * fact_admission_distance collapses routes to one row per school per year using the furthest, and keeps route_count so the page can say the figure is the widest of several bands rather than the one for a given child. Serving * Kept out of fact_admissions: that mart is EES-derived and near-complete for England, this one covers 57 LAs, and the two refresh independently. * Latest year only. Coverage is ragged — a school may have 2021 and 2026 and nothing between — so a history array would invite a trend line drawn through gaps that are absences of publication, not of a cut-off. * The Admissions section now renders on either source. 3% of the schools that render have a cut-off and no EES admissions row, and gating on admissions alone would have hidden the figure on those pages. Interface * The year travels with the figure everywhere it appears; a cut-off detached from its admissions round is not a fact about anything. * "Not a fixed catchment — it moves every year" sits under every instance, because that is the inference a parent will otherwise draw. * Replaces a hardcoded "Historical distance cut-off data is not available for this school" that appeared on every secondary page, including the ones whose council does publish it. The absence is now stated only when it is real, and names the authority that would hold it. The tint costs the muted tokens their AA margin: measured on the composited backdrop (not the computed one, which reports the untinted card), --text-muted falls to 4.09:1 in dark theme. The tile uses --text-secondary instead — 6.50:1 dark, 6.60:1 light. The DAG is manual, like the other annual ones: councils publish on allocation day, each on its own timetable, so there is no date worth scheduling against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WDvkyqqHABm4bmth2kjAxE	2026-08-15 22:48:30 +01:00
TudorandClaude Fable 5	fa6c929a3a	feat(pipeline): dim_school/dim_location store GIAS codes; seed drift test ... Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 10:45:15 +01:00
tudorandClaude Opus 4.6	8f02b5125e	feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold ... Replaces the hand-rolled integrator with a production-grade ELT pipeline using Meltano (Singer taps), dbt Core (medallion architecture), and Apache Airflow (orchestration). Adds Typesense for search and PostGIS for geospatial queries. - 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI) - dbt project: 12 staging, 5 intermediate, 12 mart models - 3 Airflow DAGs (daily/monthly/annual schedules) - Typesense sync + batch geocoding scripts - docker-compose: add Airflow, Typesense; upgrade to PostGIS - Portainer stack definition matching live deployment topology Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-26 08:37:53 +00:00