tudor/school_compare
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity
474 Commits 70 Branches 0 Tags
e188c2ff4be36fe84ca4bb915bf5597d6c41ad66
Commit Graph
7 Commits
Author SHA1 Message Date
TudorandClaude Fable 5 e188c2ff4b feat: GIAS code->name dictionaries generated from live bulk CSV 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-09 10:38:19 +01:00
tudorandClaude Sonnet 4.6 6e5249aa1e refactor(phase): merge KS2+KS4 into fact_performance, fix all phase inconsistencies
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 50s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m24s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details 
Root cause: the UNION ALL query in data_loader.py produced two rows per
all-through school per year (one KS2, one KS4), with drop_duplicates()
silently discarding the KS4 row. Fixes:

- New dbt mart `fact_performance`: FULL OUTER JOIN of fact_ks2_performance
  and fact_ks4_performance on (urn, year). One row per school per year.
  All-through schools have both KS2 and KS4 columns populated.
- data_loader.py: replace 175-line UNION ALL with a simple JOIN to
  fact_performance. No more duplicate rows or drop_duplicates needed.
- sync_typesense.py: single LATERAL JOIN to fact_performance instead of
  two separate KS2/KS4 joins.
- app.py: remove drop_duplicates (no longer needed); add PHASE_GROUPS
  constant so all-through/middle schools appear in primary and secondary
  filter results (were previously invisible to both); scope result_filters
  gender/admissions_policies to secondary schools only.
- HomeView.tsx: isSecondaryView is now majority-based (not "any secondary")
  and isMixedView shows both sort option sets for mixed result sets.
- school/[slug]/page.tsx: all-through schools route to SchoolDetailView
  (renders both SATs + GCSE sections) instead of SecondarySchoolDetailView
  (KS4-only). Dedicated SEO metadata for all-through schools.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-30 14:07:30 +01:00
tudorandClaude Sonnet 4.6 3e787b395f chore(pipeline): add EES KS4 tap diagnostic script
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 2m28s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m11s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-28 18:26:15 +00:00
tudorandClaude Sonnet 4.6 4b02ab3d8a feat: wire Typesense search into backend, fix sync performance data bug
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 1m1s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m7s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m25s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
sync_typesense.py:
- Fix query string replacement: was matching 'ST_X(l.geom) as lng' but
  QUERY_BASE uses 'l.longitude as lng' — KS2/KS4 lateral joins were
  silently dropped on every sync run

backend:
- Add typesense_url/typesense_api_key settings to config.py
- Add search_schools_typesense() to data_loader.py — queries Typesense
  'schools' alias, returns URNs in relevance order with typo tolerance;
  falls back to empty list if Typesense is unavailable
- /api/schools: replace pandas str.contains with Typesense search;
  results are filtered from the DataFrame and returned in relevance order;
  graceful fallback to substring match if Typesense is down

requirements.txt: add typesense==0.21.0, numpy==1.26.4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-27 13:23:32 +00:00
tudorandClaude Opus 4.6 719f06e480 fix(pipeline): make total_pupils non-optional for Typesense, add lat/lng to dim_location
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m3s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m29s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details 
- Remove optional flag from total_pupils (Typesense requires default
  sorting field to be non-optional)
- Add latitude/longitude columns to dim_location computed from PostGIS
  geom, for direct use by backend and Typesense sync

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 22:45:02 +00:00
tudorandClaude Opus 4.6 5e44d88d23 fix(sync): use numeric default_sorting_field, dynamic KS2/KS4 joins, populate geopoints
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 55s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m28s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
- Typesense requires numeric default_sorting_field — use total_pupils
- Dynamically include KS2/KS4 joins only if those tables exist
- Extract lat/lng from PostGIS geom and populate Typesense geopoint field

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 22:16:21 +00:00
tudorandClaude Opus 4.6 8f02b5125e feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.

- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 08:37:53 +00:00