Commit Graph
100 Commits
Author SHA1 Message Date
tudor 47335fcda0 Merge pull request 'fix(frontend): proxy /api and /sitemap.xml at runtime, not via baked rewrites' (#14) from fix/runtime-api-proxy into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 13s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 48s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 13s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Failing after 52s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Has been skipped
Reviewed-on: #14
2026-07-06 10:11:39 +00:00
tudor 5c39131b50 Merge pull request 'chore: remove the Ofsted Parent View feature end to end' (#13) from chore/remove-parent-view into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 20s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 47s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m14s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Failing after 1m7s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Has been skipped
Reviewed-on: #13
2026-07-06 08:31:00 +00:00
tudor 694b6013b3 Merge pull request 'fix(compare): keep chart data when a client refetch fails' (#12) from fix/compare-chart-refetch-resilience into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 16s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 47s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Failing after 1m9s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Has been skipped
Reviewed-on: #12
2026-07-06 07:27:03 +00:00
tudor 18cd805c6c Merge pull request 'feat(compare): readable comparison chart on mobile' (#11) from feat/compare-chart-mobile-readability into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 13s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 49s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Failing after 1m6s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Has been skipped
Reviewed-on: #11
2026-07-05 21:26:01 +00:00
tudor 90f2a02e75 Merge pull request 'fix(school): make hero map fullscreen work on iOS Safari' (#10) from fix/hero-map-ios-fullscreen into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 13s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 52s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Successful in 34s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Successful in 8s
Reviewed-on: #10
2026-07-05 20:52:32 +00:00
tudor ff606dad71 Merge pull request 'fix(e2e): pick the latest explicit year in the rankings year test' (#8) from fix/e2e-rankings-year-pick into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 52s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Successful in 34s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Successful in 10s
Reviewed-on: #8
2026-07-05 13:54:55 +00:00
tudor 0a370e3b63 Merge pull request 'fix(api): accept academic-year codes in rankings year filter' (#7) from fix/rankings-year-validation into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 21s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 46s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Failing after 1m3s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Has been skipped
Reviewed-on: #7
2026-07-05 10:46:04 +00:00
tudor 23b4e1c453 Merge pull request 'fix(e2e): scroll detail-page chart into view before asserting' (#5) from fix/e2e-detail-chart-scroll into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 13s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 49s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Successful in 33s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Successful in 8s
2026-07-03 13:56:00 +00:00
tudor 515494dbf0 Merge pull request 'fix(analytics): restrict Umami to production hostnames' (#4) from fix/umami-prod-domains-only into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 13s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 50s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 0s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Failing after 1m7s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Has been skipped
2026-07-03 13:17:53 +00:00
tudor f5de745a8b Merge pull request 'fix(ci): AI review via Claude Code CLI; reuse REGISTRY_TOKEN' (#3) from fix/ai-review-claude-code into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 18s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 46s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Failing after 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Has been skipped
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Has been skipped
2026-07-03 08:50:26 +00:00
tudor df0bf1c4d6 Merge pull request 'feat(sdlc): staging environment + automated staging→prod pipeline' (#2) from feat/sdlc-staging-pipeline into main
Deploy (staging -> E2E gate -> production) / Build Backend (FastAPI) (push) Successful in 19s
Deploy (staging -> E2E gate -> production) / Build Frontend (Next.js) (push) Successful in 46s
Deploy (staging -> E2E gate -> production) / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Deploy (staging -> E2E gate -> production) / Deploy to Staging (push) Successful in 1s
Deploy (staging -> E2E gate -> production) / E2E Journeys against Staging (push) Failing after 1m10s
Deploy (staging -> E2E gate -> production) / Promote to Production (push) Has been skipped
2026-07-03 06:17:40 +00:00
tudor 8ad11f1728 Merge pull request 'feat(admissions): surface multi-year admissions trend on school detail' (#1) from feat/admissions-multi-year-trend into main
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 13s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 51s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 12s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Reviewed-on: #1
2026-06-19 17:42:20 +00:00
tudor f48faa1803 showing schools with no KS2 results
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 44s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
2026-03-30 18:14:43 +01:00
tudorandClaude Sonnet 4.6 6e5249aa1e refactor(phase): merge KS2+KS4 into fact_performance, fix all phase inconsistencies
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 50s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m24s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Root cause: the UNION ALL query in data_loader.py produced two rows per
all-through school per year (one KS2, one KS4), with drop_duplicates()
silently discarding the KS4 row. Fixes:

- New dbt mart `fact_performance`: FULL OUTER JOIN of fact_ks2_performance
  and fact_ks4_performance on (urn, year). One row per school per year.
  All-through schools have both KS2 and KS4 columns populated.
- data_loader.py: replace 175-line UNION ALL with a simple JOIN to
  fact_performance. No more duplicate rows or drop_duplicates needed.
- sync_typesense.py: single LATERAL JOIN to fact_performance instead of
  two separate KS2/KS4 joins.
- app.py: remove drop_duplicates (no longer needed); add PHASE_GROUPS
  constant so all-through/middle schools appear in primary and secondary
  filter results (were previously invisible to both); scope result_filters
  gender/admissions_policies to secondary schools only.
- HomeView.tsx: isSecondaryView is now majority-based (not "any secondary")
  and isMixedView shows both sort option sets for mixed result sets.
- school/[slug]/page.tsx: all-through schools route to SchoolDetailView
  (renders both SATs + GCSE sections) instead of SecondarySchoolDetailView
  (KS4-only). Dedicated SEO metadata for all-through schools.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 14:07:30 +01:00
tudorandClaude Sonnet 4.6 695a571c1f fix(filters): forward gender, admissions_policy, has_sixth_form to API
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
These params were present in the URL but never passed to fetchSchools on
the server side, so the backend never applied the filters. Also include
them in hasSearchParams so filter-only searches trigger a fetch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 10:45:09 +01:00
tudorandClaude Sonnet 4.6 bd4e71dd30 feat(sort): persist sort order in URL as ?sort= param
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Sort was local state, lost on navigation. Now reads from
searchParams.get('sort') and pushes to URL on change.
'default' removes the param to keep URLs clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 10:10:28 +01:00
tudorandClaude Sonnet 4.6 cd6a5d092c feat(map): make desktop map view taller using viewport height
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Replace fixed 480px height with calc(100vh - 280px) so the map
fills most of the viewport on any screen size, clamped between
520px and 800px.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 09:29:12 +01:00
tudorandClaude Sonnet 4.6 5aed055331 feat(map): add fullscreen button using browser Fullscreen API
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Button sits top-right of the map (matching Leaflet control style),
toggles expand/compress icon, and syncs state with Escape key via
the fullscreenchange event.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 09:21:44 +01:00
tudorandClaude Sonnet 4.6 d6a45b8e12 feat(map): fetch all schools for map view, add reference pin, cap radius at 5mi
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 45s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- Remove 10-mile radius option; cap backend radius max at 5 miles
- Raise backend page_size max to 500 so map can fetch all schools in one call
- HomeView: when map view is active, fetch all schools within radius
  (page_size=500) instead of showing only the paginated first page;
  falls back to initial SSR schools while loading
- SchoolMap/LeafletMapInner: accept referencePoint prop and render a
  distinctive coral circle pin at the search postcode location

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 09:13:14 +01:00
tudorandClaude Sonnet 4.6 daf24e4739 fix(search): fix load-more silently failing due to missing page_size param
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 47s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Load-more requests read URL params (postcode, radius, etc.) but page_size
is never in the URL — it's hardcoded in page.tsx. Without it the backend
received page_size=None, hit a TypeError on (page-1)*None, returned 500,
and the silent catch left the user stuck on page 1.

In a dense area (e.g. Wimbledon SW19) 50 schools fit within ~1.8 miles,
so page 1 never shows anything beyond that regardless of selected radius.

Fix:
- Backend: give page_size a safe default of 25 instead of None
- Frontend: explicitly pass initialSchools.page_size in load-more params

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 09:08:15 +01:00
tudorandClaude Sonnet 4.6 0c5bef34cf feat(ui): polish filter controls with pill styling and custom arrows
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- controlsRow selects: pill shape (border-radius: 999px), appearance:none,
  custom SVG chevron arrow, hover/focus transitions
- advancedToggle: pill button with border, matches select height
- filterSelect (advanced panel): appearance:none, custom arrow, 8px radius,
  focus ring consistent with search input
- clearButton: pill shape matching other controls
- filters panel: separator line and spacing when expanded

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 21:31:03 +01:00
tudorandClaude Sonnet 4.6 5615458223 feat(ui): consolidate search/filter area into cleaner 2-row layout
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Before: 7 visual rows (search, radius row, phase row, advanced toggle,
teal location banner, results count, filter chips).
After: 2 rows in card + 1 unified results toolbar.

- FilterBar: merge radius, phase, and Advanced toggle into a single
  .controlsRow below the search bar; removed orphaned stacked rows
- HomeView: remove separate teal location banner; merge location info
  into results heading ("16 schools within 1.0 miles of SW196AR");
  move List/Map toggle inline with sort in one results header row
- Remove "Near postcode (Xmi)" chip (now redundant with heading)
- Sort select hidden in map view (sorting is meaningless there)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 20:46:38 +01:00
tudorandClaude Sonnet 4.6 9c9528b51b fix(FilterBar): close fragment opened around phase filter and advanced section
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 20:21:11 +01:00
tudorandClaude Sonnet 4.6 1009d7c976 fix(chart): format year as 2022/23 instead of 202223 on performance chart
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Failing after 55s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 20:17:04 +01:00
tudor 790b12a7f3 changes to order of display and text
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Failing after 55s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
2026-03-29 20:14:42 +01:00
tudorandClaude Sonnet 4.6 8f4c052294 feat(filters): move phase filter out of advanced section to always-visible position
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Failing after 1m3s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
Phase is a common filter (primary vs secondary) so it now appears
between the search form and the Advanced filters toggle rather than
being hidden inside the collapsible section.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 18:33:45 +01:00
tudorandClaude Sonnet 4.6 b7bff7bf6b feat(seo): static sitemap generation job via Airflow
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 45s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m29s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
- Backend builds sitemap.xml from school data at startup (in-memory)
- POST /api/admin/regenerate-sitemap refreshes it after data updates
- New Airflow DAG (sitemap_generate) runs Sundays 05:00 and calls the endpoint
- Next.js proxies /sitemap.xml to the backend; removes the slow dynamic sitemap.ts
- docker-compose passes BACKEND_URL + ADMIN_API_KEY to Airflow env

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 15:15:41 +01:00
tudorandClaude Sonnet 4.6 748891ab31 fix(detail): restore admissions cut-off note on secondary school page
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 15:07:52 +01:00
tudorandClaude Sonnet 4.6 17b8873f0f fix(detail): add missing location map section to secondary school page
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Frontend (Next.js) (push) Has been cancelled
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 15:06:27 +01:00
tudorandClaude Opus 4.6 15c0055687 refactor(detail): align secondary school page with primary — scroll-to-section nav
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Replace tab-based show/hide with always-visible sections and anchor
link navigation, matching the primary school detail page behaviour.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 14:48:06 +01:00
tudorandClaude Opus 4.6 6315f366c8 fix(build): single-brace JSX for schoolUrl, migrate images.domains to remotePatterns
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 14:15:06 +01:00
tudorandClaude Opus 4.6 784febc162 feat(seo): add school name to URLs, fix sticky nav, collapse compare widget
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Failing after 57s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
- URLs now /school/138267-school-name instead of /school/138267
- Bare URN URLs redirect to canonical slug (backward compat)
- Remove overflow-x:hidden that broke sticky tab nav on secondary pages
- ComparisonToast starts collapsed — user must click to open

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 12:41:28 +01:00
tudorandClaude Opus 4.6 e2c700fcfc fix(ui): round admission percentages, fix mobile overflow on detail pages
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 11:22:13 +01:00
tudorandClaude Opus 4.6 77a0f5b674 fix(detail): enrich secondary overview tab — show Ofsted grades, admissions, SEN
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 37s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The overview tab was sparse for schools without parent view data, showing
only 2 cards. Now shows:
- Individual Ofsted grades when no overall effectiveness (post-Sept 2024)
- Admissions summary card (PAN, applications, 1st choice rate)
- School context card (pupils, capacity, SEN support, EHCP)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 10:59:30 +01:00
tudorandClaude Opus 4.6 63dfa22255 fix(detail): detect secondary schools by attainment_8_score, not just phase field
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Schools with phases like "All-through" or null phase but with GCSE data
were falling through to the primary SchoolDetailView, rendering only
partial content. Now checks yearly_data for attainment_8_score as well.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 10:12:07 +01:00
tudorandClaude Opus 4.6 1d22877aec feat(ux): 8 UX improvements — simpler home, advanced filters, phase tabs, 4-line rows
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 48s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m13s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
1. Simpler home page: only search box on landing, no filter dropdowns
2. Advanced filters: hidden behind toggle on results page, auto-open if active
3. Per-school phase rendering: each row renders based on its own data
4. Taller 4-line rows with context line (type, age range, denomination, gender)
5. Result-scoped filters: dropdown values reflect current search results
6. Fix blank filter values: exclude empty strings and "Not applicable"
7. Rankings: Primary/Secondary phase tabs with phase-specific metrics
8. Compare: Primary/Secondary tabs with school counts and phase metrics

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 08:57:06 +01:00
tudor e8175561d5 updates for secondary schools
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 46s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m15s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
2026-03-28 22:36:00 +00:00
tudorandClaude Sonnet 4.6 f3a8ebdb4b fix(dbt): deduplicate int_ks4_with_lineage predecessor rows
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
When multiple predecessor URNs exist for the same current school and
year, use DISTINCT ON to keep the one with the most pupils — matching
the same logic already in int_ks2_with_lineage.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 18:58:50 +00:00
tudorandClaude Sonnet 4.6 f0c76a1724 fix(dbt): fix stg_ees_ks4 breakdown filter: 'Total' not 'All pupils'
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 31s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m26s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
The EES KS4 performance CSV uses breakdown_topic='Total' for the
all-pupils aggregate, not 'All pupils' as the model assumed. This
caused 0 rows to pass the filter despite 40k rows in raw.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 18:35:00 +00:00
tudorandClaude Sonnet 4.6 3e787b395f chore(pipeline): add EES KS4 tap diagnostic script
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 2m28s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m11s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 18:26:15 +00:00
tudorandClaude Sonnet 4.6 3d1c4c61c9 fix(types): add missing phases field to Filters fallback in rankings page
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 15:03:03 +00:00
tudorandClaude Sonnet 4.6 250d1f7c77 fix(tap-uk-idaci): add openpyxl dependency for Excel file parsing
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 49s
Build and Push Docker Images / Build Frontend (Next.js) (push) Failing after 1m2s
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 15:00:00 +00:00
tudorandClaude Sonnet 4.6 5eff9af69c feat: add secondary school support with KS4 data and metric tooltips
Build and Push Docker Images / Build Frontend (Next.js) (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Backend (FastAPI) (push) Has been cancelled
- Backend: replace INNER JOIN ks2 with UNION ALL (ks2 + ks4) so primary
  and secondary schools both appear in the main DataFrame
- Backend: add /api/national-averages endpoint computing means from live
  data, replacing the hardcoded NATIONAL_AVG constant on the frontend
- Backend: add phase filter param to /api/schools; return phases from
  /api/filters; fix hardcoded "phase": "Primary" in school detail endpoint
- Backend: add KS4 metric definitions (Attainment 8, Progress 8, EBacc,
  English & Maths pass rates) to METRIC_DEFINITIONS and RANKING_COLUMNS
- Frontend: SchoolDetailView is now phase-aware — secondary schools show
  a GCSE Results section (Att8, P8, E&M, EBacc) instead of SATs; phonics
  tab hidden for secondary; admissions says Year 7 instead of Year 3;
  history table shows KS4 columns; chart datasets switch for secondary
- Frontend: new MetricTooltip component (CSS-only ⓘ icon) backed by
  METRIC_EXPLANATIONS — added to RWM, GPS, SEN, EAL, IDACI, progress
  scores and all KS4 metrics throughout SchoolDetailView and SchoolCard
- Frontend: METRIC_EXPLANATIONS extended with KS4 terms (Attainment 8,
  Progress 8, EBacc) and previously missing terms (SEN, EHCP, EAL, IDACI)
- Frontend: SchoolCard expands "RWM" to "Reading, Writing & Maths" and
  shows Attainment 8 / English & Maths Grade 4+ for secondary schools
- Frontend: FilterBar adds Phase dropdown (Primary / Secondary / All-through)
- Frontend: HomeView hero copy updated; compact list shows phase-aware metric
- Global metadata updated to remove "primary only" framing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 14:59:40 +00:00
tudorandClaude Sonnet 4.6 b0990e30ee fix(ui): retheme comparison toast to match site palette
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- Switch from dark (#1a1612) to site's warm cream background
- Clear all button now visible as a text button with muted/coral hover
- Remove scroll bar: no max-height cap needed since 5 schools max
- Compare Now button uses coral accent to match primary CTAs
- School items use bg-secondary (beige) consistent with site cards

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 22:09:59 +00:00
tudorandClaude Sonnet 4.6 1629a8f994 feat(pipeline): add DAGs for Parent View and IDACI deprivation
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
- school_data_monthly_parent_view: runs 1st of month, extracts Ofsted
  Parent View and builds fact_parent_view
- school_data_annual_idaci: manual trigger, extracts IDACI deprivation
  index and builds fact_deprivation

Both tables were missing, causing safe_query to fail and leave the
PostgreSQL transaction in an aborted state, silently killing all
subsequent supplementary data queries including fact_admissions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 22:08:12 +00:00
tudorandClaude Sonnet 4.6 55749bdfaf debug(backend): log safe_query exceptions and rollback on failure
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 45s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 22:00:27 +00:00
tudorandClaude Sonnet 4.6 cd1c649d0f fix(frontend): format 6-digit EES academic year codes correctly
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
formatAcademicYear now handles both 4-digit (2023→2023/24) and 6-digit
EES codes (202526→2025/26). Applied to all year displays: SATs, phonics,
admissions, finances, and the yearly results table.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 18:30:37 +00:00
tudorandClaude Sonnet 4.6 7724fe3503 fix(stg_ofsted_inspections): correctly filter NULL string inspection dates
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The string 'NULL' is not SQL NULL, so the WHERE in the renamed CTE
passed those rows through. Filter on the raw value using nullif in the
CTE and on the computed date in the outer SELECT.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 18:21:30 +00:00
tudorandClaude Sonnet 4.6 1d56eebe87 fix(stg_ofsted_inspections): filter out rows with no inspection date
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m24s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Schools in the MI file that have never been inspected have a null
inspection_date after parsing. Exclude them — they are not inspection
records.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 17:55:11 +00:00
tudorandClaude Sonnet 4.6 10720400fd fix(stg_ofsted_inspections): parse DD/MM/YYYY date format from Ofsted CSV
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m3s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 17:34:34 +00:00
tudorandClaude Sonnet 4.6 05cb22f1a5 fix(stg_ofsted_inspections): handle NULL strings from Ofsted CSV
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m26s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Use nullif+trim for date cast and safe_numeric for integer grades to
handle literal 'NULL' strings present in the new Report Card format CSV.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 17:23:46 +00:00
tudorandClaude Sonnet 4.6 26aa3c2d70 fix(tap-uk-ofsted): fix header row detection matching 'urn' inside 'turn'
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m40s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The preamble row in Ofsted CSVs contains 'turn off all filters' which
matched 'urn' in line.lower(), so header_idx was set to 0 instead of
the real header row. Use a regex that matches URN only as a CSV field.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 17:05:03 +00:00
tudorandClaude Sonnet 4.6 e56a63c59c debug(tap-uk-ofsted): log CSV column names to diagnose 0-record extraction
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 31s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m40s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 15:47:32 +00:00
tudorandClaude Sonnet 4.6 221923857d chore: remove integrator/kestra CI jobs, fix school website link protocol
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- Remove build-integrator and build-kestra-init jobs from Gitea Actions
- Update trigger-deployment needs to only depend on remaining three builds
- Fix school website href to prepend https:// when protocol is missing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 15:30:08 +00:00
tudorandClaude Sonnet 4.6 62284e7a94 chore: remove Kestra and integrator legacy services
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m11s
Build and Push Docker Images / Build Integrator (push) Failing after 30s
Build and Push Docker Images / Build Kestra Init (push) Failing after 29s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 30s
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
Migration to Airflow + Meltano pipeline is complete. Remove:
- kestra, kestra-init, integrator services from docker-compose.portainer.yml
- kestra_storage and supplementary_data volumes
- KESTRA_USER/KESTRA_PASSWORD env var references
- integrator/ directory (Kestra flows, scripts, Dockerfiles)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 15:03:34 +00:00
tudorandClaude Sonnet 4.6 668e234eb2 feat(census): add demographic columns to EES census tap and staging models
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m39s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
tap-uk-ees: EESCensusStream now declares 27 data columns (FSM %, EAL %,
ethnicity breakdowns, pupil counts) with clean Singer field names mapped
from the verbose CSV column names (e.g. '% of pupils known to be eligible
for free school meals' → fsm_pct) via a new _column_renames mechanism on
the base stream class.

stg_ees_census: materialised as table, applies safe_numeric to all
percentage/count columns, filters to numeric URNs.

int_pupil_chars_merged + fact_pupil_characteristics: pass all columns
through from staging (previously stubs with only 3 columns).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 14:07:48 +00:00
tudorandClaude Sonnet 4.6 4b02ab3d8a feat: wire Typesense search into backend, fix sync performance data bug
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 1m1s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
sync_typesense.py:
- Fix query string replacement: was matching 'ST_X(l.geom) as lng' but
  QUERY_BASE uses 'l.longitude as lng' — KS2/KS4 lateral joins were
  silently dropped on every sync run

backend:
- Add typesense_url/typesense_api_key settings to config.py
- Add search_schools_typesense() to data_loader.py — queries Typesense
  'schools' alias, returns URNs in relevance order with typo tolerance;
  falls back to empty list if Typesense is unavailable
- /api/schools: replace pandas str.contains with Typesense search;
  results are filtered from the DataFrame and returned in relevance order;
  graceful fallback to substring match if Typesense is down

requirements.txt: add typesense==0.21.0, numpy==1.26.4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 13:23:32 +00:00
tudorandClaude Sonnet 4.6 5d8b319451 fix(dbt): stub rc_* columns as NULL in stg_ofsted_inspections
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m23s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
tap-uk-ofsted schema only declares OEIF columns; rc_* (Report Card)
columns were never emitted so they don't exist in raw.ofsted_inspections.
Replace column references with NULL::text until the actual CSV column
names for the post-Nov 2025 Report Card framework are confirmed and
added to the tap schema.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 12:50:58 +00:00
tudorandClaude Sonnet 4.6 77f75fb6e5 fix(dbt): deduplicate predecessor KS2 rows and downgrade orphan test to warn
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m11s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
- int_ks2_with_lineage: use DISTINCT ON (current_urn, year) in predecessor_ks2
  to handle schools with multiple predecessors that both have KS2 data for the
  same year (e.g. two schools that merged). Keeps the predecessor with most pupils.
- dbt_project.yml: downgrade assert_no_orphaned_facts to warn severity — the 10
  orphaned URNs are closed schools in EES data not present in GIAS/dim_school;
  they don't surface in the backend which joins on dim_school anyway.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 12:16:36 +00:00
tudorandClaude Sonnet 4.6 b41e6c250e fix(dbt): filter non-numeric URNs and trim whitespace in EES staging models
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
- Filter school_urn/time_period to '^[0-9]+$' to exclude "n/a" and other
  non-numeric values that caused integer cast failures in fact_admissions
- Add trim() to all school_urn/time_period casts to prevent whitespace
  variants producing duplicate urn+year rows in fact_ks2_performance

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 12:00:30 +00:00
tudorandClaude Sonnet 4.6 6e720feca4 perf(dbt): collapse stg_ees_ks2 to single-pass pivot
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Previous version scanned ees_ks2_attainment (1.2M rows) 5 times via
separate CTEs (all_pupils, gender_boys, gender_girls, disadv, not_disadv)
plus 5 LEFT JOINs. Rewritten as one GROUP BY with conditional aggregation
— single scan, no self-joins.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 11:42:40 +00:00
tudorandClaude Sonnet 4.6 ae9fd26eba perf(dbt): materialize stg_ees_ks2 and stg_ees_ks4 as tables
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
KS2 attainment has 1.2M rows in long format. As a view, the pivot was
re-executed inline for every downstream model (intermediate → fact),
causing fact_ks2_performance CREATE TABLE to run for 18+ minutes.

Materializing as tables means the pivot runs once during staging, and
downstream models read from a pre-computed ~16k-row result.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 11:20:20 +00:00
tudorandClaude Sonnet 4.6 33b395d2bd fix(dbt): apply safe_numeric macro to fix EES suppression code 'c' errors
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m14s
Build and Push Docker Images / Build Integrator (push) Successful in 58s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Replace nullif(col, 'z') casts with safe_numeric macro across KS2, KS4,
and admissions staging models. The regex-based macro treats any non-numeric
string (z, c, x, q, u, etc.) as NULL without needing an explicit list.

Also fix FSM_eligible_percent column quoting in stg_ees_admissions — target-
postgres stores mixed-case column names quoted, so unquoted references were
being folded to fsm_eligible_percent by PostgreSQL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 10:41:27 +00:00
tudorandClaude Sonnet 4.6 8e8d1bd8c5 fix(ees-tap): filter out rows with null URN before emitting
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m47s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The admissions school-level file contains some rows with null school_urn
(LA/category aggregates that survive the geographic_level filter). These
cause a not-null constraint violation at target-postgres. Drop any row
where the URN column is null or empty before yielding records.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 10:13:17 +00:00
tudorandClaude Sonnet 4.6 c7357336e3 fix(ees-tap): fix BOM handling for admissions CSV
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m40s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Admissions file is UTF-8 with BOM, not Latin-1. Reading as latin-1
decoded the BOM bytes as '' which wasn't stripped. Change admissions
encoding to utf-8-sig (strips BOM automatically). Also update the manual
BOM strip fallback to handle the latin-1 decoded form.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 10:03:17 +00:00
tudorandClaude Sonnet 4.6 b8ecc5c58b fix(ees-tap): strip UTF-8 BOM from CSV column names
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m42s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Some DfE supporting-files CSVs have a UTF-8 BOM on the first column,
causing it to be named '\ufefftime_period' instead of 'time_period'.
This trips Singer schema validation ('time_period' is a required property).
Strip the BOM from all column names after read_csv.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 09:54:15 +00:00
tudorandClaude Sonnet 4.6 f4f0257447 fix(ees-tap): add latin-1 encoding for census/admissions, default utf-8 for others
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 52s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m8s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m40s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
DfE supporting-files CSVs (spc_school_level_underlying_data, AppsandOffers
SchoolLevel) are Latin-1 encoded. Add _encoding class attribute to base
stream class and override to 'latin-1' for census and admissions streams.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 09:41:40 +00:00
tudorandClaude Sonnet 4.6 ca351e9d73 feat: migrate backend to marts schema, update EES tap for verified datasets
Pipeline:
- EES tap: split KS4 into performance + info streams, fix admissions filename
  (SchoolLevel keyword match), fix census filename (yearly suffix), remove
  phonics (no school-level data on EES), change endswith → in for matching
- stg_ees_ks4: rewrite to filter long-format data and extract Attainment 8,
  Progress 8, EBacc, English/Maths metrics; join KS4 info for context
- stg_ees_admissions: map real CSV columns (total_number_places_offered, etc.)
- stg_ees_census: update source reference, stub with TODO for data columns
- Remove stg_ees_phonics, fact_phonics (no school-level EES data)
- Add ees_ks4_performance + ees_ks4_info sources, remove ees_ks4 + ees_phonics
- Update int_ks4_with_lineage + fact_ks4_performance with new KS4 columns
- Annual EES DAG: remove stg_ees_phonics+ from selector

Backend:
- models.py: replace all models to point at marts.* tables with schema='marts'
  (DimSchool, DimLocation, KS2Performance, FactOfstedInspection, etc.)
- data_loader.py: rewrite load_school_data_as_dataframe() using raw SQL joining
  dim_school + dim_location + fact_ks2_performance; update get_supplementary_data()
- database.py: remove migration machinery, keep only connection setup
- app.py: remove check_and_migrate_if_needed, remove /api/admin/reimport-ks2
  endpoints (pipeline handles all imports)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 09:29:27 +00:00
tudorandClaude Opus 4.6 d82e36e7b2 feat(ees): rewrite EES tap and KS2 models for actual data structure
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 31s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m8s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m45s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- Fix publication slugs (KS4, Phonics, Admissions were wrong)
- Split KS2 into two streams: ees_ks2_attainment (long format) and
  ees_ks2_info (wide format context data)
- Target specific filenames instead of keyword matching
- Handle school_urn vs urn column naming
- Pivot KS2 attainment from long to wide format in dbt staging
- Add all ~40 KS2 columns the backend needs (GPS, absence, gender,
  disadvantaged breakdowns, context demographics)
- Pass through all columns in int_ks2_with_lineage and fact_ks2

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 23:08:50 +00:00
tudorandClaude Opus 4.6 719f06e480 fix(pipeline): make total_pupils non-optional for Typesense, add lat/lng to dim_location
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m3s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m29s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
- Remove optional flag from total_pupils (Typesense requires default
  sorting field to be non-optional)
- Add latitude/longitude columns to dim_location computed from PostGIS
  geom, for direct use by backend and Typesense sync

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 22:45:02 +00:00
tudorandClaude Opus 4.6 5e44d88d23 fix(sync): use numeric default_sorting_field, dynamic KS2/KS4 joins, populate geopoints
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- Typesense requires numeric default_sorting_field — use total_pupils
- Dynamically include KS2/KS4 joins only if those tables exist
- Extract lat/lng from PostGIS geom and populate Typesense geopoint field

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 22:16:21 +00:00
tudorandClaude Opus 4.6 cc481aa00c fix(airflow): remove PostGIS init from airflow, rely on postgis image initdb
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The postgis/postgis image auto-enables PostGIS on fresh database creation.
No need to do it from airflow-init.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 22:11:00 +00:00
tudorandClaude Opus 4.6 613a030c95 fix(airflow): ensure PostGIS extension exists during init
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 22:08:12 +00:00
tudorandClaude Opus 4.6 72cbbf7778 fix(dbt): simplify search_path to just public for PostGIS
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 21:47:01 +00:00
tudorandClaude Opus 4.6 03256fed41 fix(dbt): add search_path to profile so PostGIS functions resolve in all schemas
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Integrator (push) Has been cancelled
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Frontend (Next.js) (push) Has been cancelled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 21:45:53 +00:00
tudorandClaude Opus 4.6 b7cc01f26f fix(dbt): schema-qualify PostGIS functions in dim_location
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Integrator (push) Has been cancelled
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Frontend (Next.js) (push) Has been cancelled
PostGIS extension lives in public schema; marts schema can't resolve
unqualified ST_MakePoint/ST_Transform calls.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 21:45:03 +00:00
tudorandClaude Opus 4.6 28ba2fd0a6 fix(dbt): cast easting/northing to double precision for ST_MakePoint
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 21:29:16 +00:00
tudorandClaude Opus 4.6 03cd1de6af fix(airflow): delete and reimport DAGs on init to clear stale task refs
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Integrator (push) Has been cancelled
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Frontend (Next.js) (push) Has been cancelled
When tasks are removed from a DAG, old serialized metadata in the DB
causes 'Task not found' errors. Delete all DAGs before reserializing
on each deploy to ensure a clean state.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 21:28:03 +00:00
tudorandClaude Opus 4.6 54df58746e feat(pipeline): use GIAS easting/northing for all geocoding, drop postcode step
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
GIAS grid references are the actual school location — far more accurate
than postcode centroids. Remove geocode_postcodes.py from the daily DAG
and the postcode-not-null filter from dim_location.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 21:18:59 +00:00
tudorandClaude Opus 4.6 d3e655abdb fix(dbt): compute geom from easting/northing in dim_location
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m2s
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Integrator (push) Has been cancelled
Convert GIAS British National Grid coordinates (EPSG:27700) to WGS84
(EPSG:4326) directly in the dbt model. The geocode script backfills
schools missing easting/northing via Postcodes.io.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 21:17:08 +00:00
tudorandClaude Opus 4.6 45f3e4d9fc fix(dbt): override generate_schema_name to use direct schema names
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
dbt default prepends the profile schema as prefix (public_staging,
public_marts). Override to use custom schema names directly (staging,
marts) so scripts can reference marts.dim_location correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 21:09:23 +00:00
tudorandClaude Opus 4.6 d25e333826 fix(dbt): remove invalid relationship test on map_school_lineage
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Lineage map includes predecessor URNs for closed schools, which are
correctly excluded from dim_school (status = 'Open').

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 20:59:29 +00:00
tudorandClaude Opus 4.6 7f82088d53 fix(pipeline): use to_date for DD-MM-YYYY GIAS dates, exclude EES models from daily DAG
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
GIAS CSV dates are DD-MM-YYYY format — use to_date() instead of cast().
Exclude int_ks2_with_lineage+ and int_ks4_with_lineage+ from daily DAG
selector since they depend on EES data not yet loaded.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 20:51:40 +00:00
tudorandClaude Opus 4.6 e7b1ab9f37 fix(pipeline): expand GIAS schema, handle empty strings, scope DAG selectors
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m8s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 34s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m39s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
- Declare all 34 columns needed by dbt in GIAS tap schema (target-postgres
  only persists columns present in the Singer schema message)
- Use nullif() for empty-string-to-integer/date casts in staging models
- Scope daily DAG dbt build to GIAS models only (stg_gias_establishments+
  stg_gias_links+) to avoid errors on unloaded sources
- Scope annual EES DAG similarly; remove redundant dbt test steps
- Make dim_school gracefully handle missing int_ofsted_latest table

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 20:43:24 +00:00
tudorandClaude Opus 4.6 24cfb83144 fix(dbt): fix GIAS source column quoting and remove tests on unloaded sources
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 2m39s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m8s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m27s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
GIAS tap emits uppercase URN column — add quote: true so dbt source tests
reference "URN" instead of urn. Remove source-level tests from tables not yet
loaded (ofsted, ees, parent_view, fbit, idaci) to prevent relation-not-found
errors during dbt build.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 20:25:56 +00:00
tudorandClaude Opus 4.6 72ef1b03b7 fix(airflow): use correct Airflow 3 env vars for multi-container JWT and Execution API
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Integrator (push) Successful in 54s
Build and Push Docker Images / Build Kestra Init (push) Successful in 30s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Replace Airflow 2.x env vars (CORE__SECRET_KEY, CORE__INTERNAL_API_URL) with
correct Airflow 3.x equivalents (API_AUTH__JWT_SECRET, API_AUTH__JWT_ISSUER,
CORE__EXECUTION_API_SERVER_URL) on all three Airflow services.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 20:11:06 +00:00
tudorandClaude Opus 4.6 ea160b53df fix(airflow): point scheduler to api-server via INTERNAL_API_URL
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m3s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 30s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 33s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
With separate containers, task workers in the scheduler need the
api-server's address for the Execution API. Defaults to localhost:8080
which fails across containers. Set INTERNAL_API_URL to the api-server's
Docker service name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 17:09:17 +00:00
tudorandClaude Opus 4.6 8a2503230f fix(airflow): split back to separate scheduler and api-server containers
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m1s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 29s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Running both in one container caused JWT secret key race conditions.
Separate containers with the same AIRFLOW__CORE__SECRET_KEY env var
ensures both processes use identical JWT signing keys. Shared
airflow_logs volume allows the api-server to read task logs written
by the scheduler.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 17:00:07 +00:00
tudorandClaude Opus 4.6 677e80ad70 fix(airflow): generate config before starting processes, set fixed secret key
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 31s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m3s
Build and Push Docker Images / Build Integrator (push) Successful in 54s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
The init container and airflow container have separate filesystems, so
airflow.cfg generated by db migrate is not available to the scheduler/
api-server. Without a config file, both processes race to generate
their own with different random JWT secret keys.

Fix by:
1. Running `airflow config list` first to generate airflow.cfg once
2. Setting a fixed SECRET_KEY via env var (>= 64 bytes for SHA512)
3. Adding sleep 3 so scheduler writes config before api-server starts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 16:57:22 +00:00
tudorandClaude Opus 4.6 1dbcc24434 fix(airflow): stop deleting airflow.cfg, let processes share config
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 31s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m2s
Build and Push Docker Images / Build Integrator (push) Successful in 54s
Build and Push Docker Images / Build Kestra Init (push) Successful in 30s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Deleting airflow.cfg at container start caused the scheduler and
api-server to each generate their own random JWT secret key, leading
to 'Signature verification failed' when task workers communicated
with the api-server. Let both processes share the config file
generated by db migrate (env vars still override where needed).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 16:49:18 +00:00
tudorandClaude Opus 4.6 b3e4769d82 fix(airflow): set shared internal API secret key
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 30s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m2s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 30s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
When scheduler and api-server run in the same container, both generate
independent JWT signing keys on startup. The scheduler's task workers
then fail with 'Invalid auth token: Signature verification failed'
when communicating with the api-server. Fix by setting a shared
INTERNAL_API_SECRET_KEY via env var.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 16:42:02 +00:00
tudorandClaude Opus 4.6 7a39f4cdb1 fix(ci): use correct mirror address 10.0.1.224:6000
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 30s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m3s
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 15:06:17 +00:00
tudorandClaude Opus 4.6 1a9dd49097 fix(ci): configure buildx to use local Docker Hub mirror
Build and Push Docker Images / Build Backend (FastAPI) (push) Failing after 44s
Build and Push Docker Images / Build Frontend (Next.js) (push) Failing after 50s
Build and Push Docker Images / Build Integrator (push) Failing after 41s
Build and Push Docker Images / Build Kestra Init (push) Failing after 41s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 29s
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
docker/setup-buildx-action creates a BuildKit builder that ignores
the host daemon's registry-mirrors setting. Configure buildkitd inline
to route docker.io pulls through the local pull-through cache at
172.17.0.1:6000 (Docker bridge gateway → host port 6000).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 14:59:16 +00:00
tudorandClaude Opus 4.6 0062a5eabe fix(tap-gias): declare numeric CSV columns as StringType
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Failing after 30s
Build and Push Docker Images / Build Kestra Init (push) Failing after 30s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Failing after 29s
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
CSV is read with dtype=str so all values arrive as strings. Declaring
LA (code) and EstablishmentNumber as IntegerType caused schema
validation failures in target-postgres. Use StringType for all columns
except URN (which is explicitly cast to int for the primary key).
Type casting happens in dbt staging models.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 14:03:26 +00:00
tudorandClaude Opus 4.6 84261f6125 fix(meltano): set default_environment, remove deprecated version field
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Build and Push Docker Images / Build Integrator (push) Has been cancelled
Meltano 4.x requires an environment to be specified. Set production as
the default. Also remove the deprecated 'version: 2' field.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 14:01:31 +00:00
tudorandClaude Opus 4.6 9eae6bffae fix(meltano): use 'database' not 'dbname' for meltanolabs target-postgres
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m35s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
The meltanolabs target-postgres variant expects 'database' as the
config key, not 'dbname' (which was the pipelinewise variant's key).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 13:53:49 +00:00
tudorandClaude Opus 4.6 c576bba06a fix(meltano): remove catalog capability and switch elt to run
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m26s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
The `catalog` capability forced Meltano to run --discover and generate
a catalog file (tap.properties.json) before each extraction. This fails
because our Singer SDK taps emit schemas inline and don't need external
catalog files. Removing the capability makes Meltano invoke taps
directly without catalog generation.

Also switch from deprecated `meltano elt` to `meltano run` for
Meltano 4.x compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 13:45:23 +00:00
tudorandClaude Opus 4.6 1c77a6c593 fix(pipeline): run meltano install in Dockerfile to generate catalogs
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m13s
Build and Push Docker Images / Build Integrator (push) Successful in 58s
Build and Push Docker Images / Build Kestra Init (push) Successful in 33s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Meltano elt requires catalog files (tap.properties.json) to exist.
These are generated by `meltano install` which discovers tap schemas
and installs the target-postgres loader. Without this step, `meltano
elt` fails with "catalog file is missing".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 12:28:59 +00:00
tudorandClaude Opus 4.6 07869738c0 fix(airflow): merge scheduler and api-server into single container
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m6s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
With LocalExecutor, tasks run in the scheduler process and logs are
written locally. Running api-server and scheduler in separate containers
meant the api-server couldn't read task logs (empty hostname in log
fetch URL). Combining them into one container eliminates the issue —
logs are always on the local filesystem.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 12:16:18 +00:00
tudorandClaude Opus 4.6 a3a50cc8d2 fix(airflow): remove generated airflow.cfg so env vars take effect
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
airflow db migrate generates airflow.cfg with default values that
shadow our env vars (DAGS_FOLDER, WORKER_LOG_SERVER_HOST, etc).
Delete the generated config file before starting each service so
Airflow falls through to env var configuration exclusively.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 12:12:32 +00:00