TudorandClaude Opus 5.5
65a2619e1d
fact_ks4_destinations and fact_ks5_destinations join dim_school, so the daily build's stg_gias_establishments+ and the monthly Ofsted build's dim_school+ both selected them. They also read stg_ees_ks4/ks5_destinations, which only the manually triggered EES DAG builds. Where that DAG hasn't run since the destinations models landed, dbt_build fails with "relation staging.stg_ees_ks4_destinations does not exist", and sync_typesense and invalidate_cache never run. Production's register data has been stuck at about 25 Aug 2026. Both builds now exclude the descendants of the two EES staging models, as the daily build already does for the KS2/KS4 lineage models. The EES DAG still rebuilds the marts when their data changes. test_dag_selectors reads the model graph from the SQL (CI has no dbt) and checks that every scheduled build only reads models it or the daily build builds. It failed for the daily and monthly Ofsted builds before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SchoolCompare
SchoolCompare compares schools across England: primary (KS2), secondary (KS4), all-through and post-16 provision, with coverage depending on the source dataset. It provides school search, postcode maps, comparisons, rankings, place pages, Ofsted information, admissions and destination measures. Editorial content lives in a Payload CMS blog.
Start here
- Architecture and data flow
- Development and validation
- Deployment and promotion
- Legacy and unused-code inventory
- Frontend conventions
- CMS publishing
Repository map
| Path | Responsibility |
|---|---|
backend/ |
FastAPI routes, cached school data, read-only SQLAlchemy mappings, feature flags |
nextjs-app/ |
Next.js App Router, React UI, Payload CMS, frontend tests |
pipeline/plugins/extractors/ |
Custom Singer taps for GIAS, EES, Ofsted and other datasets |
pipeline/transform/ |
dbt staging/intermediate models, marts, seeds and data tests |
pipeline/dags/ |
Airflow extraction, transformation and publication workflows |
pipeline/scripts/ |
Search indexing, code generation and operational diagnostics |
e2e/ |
Playwright journeys against a running environment |
.gitea/workflows/ |
PR checks, staging deployment and manual production promotion |
scripts/ |
CI review tooling and historical data utilities; see the legacy inventory |
docs/superpowers/, mockups/ |
Design history and prototypes, not application entry points |
Runtime
The public site is Next.js, not the FastAPI root page. Browser /api/*
requests pass through a Next.js route handler to FastAPI. Server-rendered pages
call FastAPI directly using FASTAPI_URL, including its /api suffix.
PostgreSQL/PostGIS stores school data. Meltano/Singer extracts source data;
dbt builds marts.*; FastAPI reads those tables. Typesense serves text search
and autocomplete. Payload runs inside Next.js and owns a separate payload
database schema and uploaded media.
There is no automatic CSV import or sample dataset on startup. A working school-data environment needs populated marts from the pipeline or an approved database snapshot. See development before choosing a setup.
Validation
cd nextjs-app
npm ci
npm run typecheck
npm test -- --runInBand
Backend checks, pipeline validation, runtime versions and E2E requirements are
listed in DEVELOPMENT.md. No npm run lint script is
currently defined.
Deployment
Work on a feature branch and open a PR. Merging to main builds images and
deploys staging. Production promotion is a separate, human-triggered Gitea
workflow. Use DEPLOY.md and the Portainer compose files as the
operational references. The generic compose examples still reference :latest,
which the current release workflow does not publish.