TudorandClaude Opus 5.5
94bfac9caf
GIAS publishes its CSVs in Windows-1252 and sends no charset. The tap read resp.text, so requests guessed the codec, and the encoding="latin-1" passed to read_csv did nothing on already-decoded text. On 3 Oct 2026 the guess was windows-1250, and "St Thomas à Becket" (138950, 149557) was stored as "St Thomas ŕ Becket". A different guess on another day would garble other accented names. Both streams now decode the downloaded bytes themselves (gias_csv.py). A byte Windows-1252 leaves undefined becomes U+FFFD with a logged warning instead of failing the load, so one odd name cannot stop the daily refresh. None of the nine extracts checked (1 Jul to 3 Oct 2026) contains such a byte. Checked by running the tap on the real 3 Oct extract with .text forced to windows-1250: all 52,586 rows decode, with no "ŕ" and no replacement characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SchoolCompare
SchoolCompare compares schools across England: primary (KS2), secondary (KS4), all-through and post-16 provision, with coverage depending on the source dataset. It provides school search, postcode maps, comparisons, rankings, place pages, Ofsted information, admissions and destination measures. Editorial content lives in a Payload CMS blog.
Start here
- Architecture and data flow
- Development and validation
- Deployment and promotion
- Legacy and unused-code inventory
- Frontend conventions
- CMS publishing
Repository map
| Path | Responsibility |
|---|---|
backend/ |
FastAPI routes, cached school data, read-only SQLAlchemy mappings, feature flags |
nextjs-app/ |
Next.js App Router, React UI, Payload CMS, frontend tests |
pipeline/plugins/extractors/ |
Custom Singer taps for GIAS, EES, Ofsted and other datasets |
pipeline/transform/ |
dbt staging/intermediate models, marts, seeds and data tests |
pipeline/dags/ |
Airflow extraction, transformation and publication workflows |
pipeline/scripts/ |
Search indexing, code generation and operational diagnostics |
e2e/ |
Playwright journeys against a running environment |
.gitea/workflows/ |
PR checks, staging deployment and manual production promotion |
scripts/ |
CI review tooling and historical data utilities; see the legacy inventory |
docs/superpowers/, mockups/ |
Design history and prototypes, not application entry points |
Runtime
The public site is Next.js, not the FastAPI root page. Browser /api/*
requests pass through a Next.js route handler to FastAPI. Server-rendered pages
call FastAPI directly using FASTAPI_URL, including its /api suffix.
PostgreSQL/PostGIS stores school data. Meltano/Singer extracts source data;
dbt builds marts.*; FastAPI reads those tables. Typesense serves text search
and autocomplete. Payload runs inside Next.js and owns a separate payload
database schema and uploaded media.
There is no automatic CSV import or sample dataset on startup. A working school-data environment needs populated marts from the pipeline or an approved database snapshot. See development before choosing a setup.
Validation
cd nextjs-app
npm ci
npm run typecheck
npm test -- --runInBand
Backend checks, pipeline validation, runtime versions and E2E requirements are
listed in DEVELOPMENT.md. No npm run lint script is
currently defined.
Deployment
Work on a feature branch and open a PR. Merging to main builds images and
deploys staging. Production promotion is a separate, human-triggered Gitea
workflow. Use DEPLOY.md and the Portainer compose files as the
operational references. The generic compose examples still reference :latest,
which the current release workflow does not publish.