4 Commits

Include renames

Author	SHA1	Message	Date
tudor	b6e48c4930	Merge pull request 'fix(pipeline): decode GIAS extracts as Windows-1252' (#182) from fix/gias-encoding into main ... Reviewed-on: #182	2026-10-05 06:31:30 +00:00
TudorandClaude Opus 5.5	94bfac9caf	fix(pipeline): decode GIAS extracts as Windows-1252 ... GIAS publishes its CSVs in Windows-1252 and sends no charset. The tap read resp.text, so requests guessed the codec, and the encoding="latin-1" passed to read_csv did nothing on already-decoded text. On 3 Oct 2026 the guess was windows-1250, and "St Thomas à Becket" (138950, 149557) was stored as "St Thomas ŕ Becket". A different guess on another day would garble other accented names. Both streams now decode the downloaded bytes themselves (gias_csv.py). A byte Windows-1252 leaves undefined becomes U+FFFD with a logged warning instead of failing the load, so one odd name cannot stop the daily refresh. None of the nine extracts checked (1 Jul to 3 Oct 2026) contains such a byte. Checked by running the tap on the real 3 Oct extract with .text forced to windows-1250: all 52,586 rows decode, with no "ŕ" and no replacement characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-04 10:09:39 +01:00
TudorandClaude Opus 5.5	65a2619e1d	fix(pipeline): keep the destinations marts out of the scheduled builds ... fact_ks4_destinations and fact_ks5_destinations join dim_school, so the daily build's stg_gias_establishments+ and the monthly Ofsted build's dim_school+ both selected them. They also read stg_ees_ks4/ks5_destinations, which only the manually triggered EES DAG builds. Where that DAG hasn't run since the destinations models landed, dbt_build fails with "relation staging.stg_ees_ks4_destinations does not exist", and sync_typesense and invalidate_cache never run. Production's register data has been stuck at about 25 Aug 2026. Both builds now exclude the descendants of the two EES staging models, as the daily build already does for the KS2/KS4 lineage models. The EES DAG still rebuilds the marts when their data changes. test_dag_selectors reads the model graph from the SQL (CI has no dbt) and checks that every scheduled build only reads models it or the daily build builds. It failed for the daily and monthly Ofsted builds before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-03 22:21:27 +01:00
TudorandClaude Opus 5	38bc17cab3	feat(search): validate the index before the alias points at it ... The old sync created a collection, imported batches without reading a single import response, and swapped the alias regardless. A partial import published a half-empty index, and two overlapping DAG runs could prune each other's collections. Publication now checks every import response and the final document count before upserting the alias, and holds a session-scoped advisory lock across the read and the publish so concurrent runs serialise. Cleanup keeps the previous collection as a rollback pointer and is best-effort: an uncertain alias response must never delete what might still be live. Also parses the Typesense URL properly instead of splitting on colons, which mangled any host carrying a scheme and a default port. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-15 10:16:36 +01:00