TudorandClaude Opus 5
eaf5e5d180
The README still opened on "Primary School Compass", a KS2 tool for Wandsworth and Merton served by FastAPI and vanilla JavaScript with Chart.js. Every layer of that sentence is now wrong: coverage is England-wide across KS2, KS4, all-through and post-16, Next.js owns the public UI, and school data comes from dbt-built `marts.*` rather than CSVs loaded at startup. The setup instructions walked a reader into a virtualenv and a CSV import that cannot build the current schema, so following the docs produced an empty database and a wrong mental model at the same time. Replace the narrative docs with two reference documents that were checked against the code: docs/ARCHITECTURE.md for request flow, data ownership, the backend/frontend module boundaries and the real publication sequence, and docs/DEVELOPMENT.md for the checks that actually run, including the container and CI version skew that makes "just run pytest" misleading. The env examples drifted the same way. ALLOWED_ORIGINS is a JSON array, not a comma-separated list; the frontend needs FASTAPI_URL, DATABASE_URL and PAYLOAD_SECRET, none of which were documented; and RATE_LIMIT_BURST, DEFAULT_PAGE_SIZE and MAX_PAGE_SIZE were presented as tuning controls the routes do not consult. Each is now stated as it behaves. MIGRATION_SUMMARY.md keeps its content but gains a banner, because it reads like setup instructions and is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016y2J6bs8gbuSJbH18w7Tan
108 lines
5.7 KiB
Markdown
108 lines
5.7 KiB
Markdown
# Architecture
|
|
|
|
This describes the implementation as reviewed on 2026-09-14. It distinguishes
|
|
current behaviour from improvements still to be implemented.
|
|
|
|
## Request flow
|
|
|
|
```text
|
|
Browser → Next.js public routes
|
|
├─ /api/* proxy → FastAPI → cached DataFrames / PostgreSQL marts
|
|
│ ├─ Typesense (search and suggestions)
|
|
│ └─ postcodes.io (postcode lookup)
|
|
└─ /admin, /cms-api, /blog → Payload → payload schema + media volume
|
|
|
|
Next.js server rendering → FastAPI directly through FASTAPI_URL
|
|
```
|
|
|
|
`nextjs-app/lib/api.ts` contains typed fetch wrappers and revalidation defaults.
|
|
The proxy is `nextjs-app/app/(frontend)/api/[...path]/route.ts`. Payload uses
|
|
`/cms-api` so its routes do not collide with the FastAPI proxy. The proxy denies
|
|
`/api/flags`; server-side rendering reads flags directly from FastAPI.
|
|
|
|
## Data ownership
|
|
|
|
| Layer | Owner and role |
|
|
|---|---|
|
|
| Source data | GIAS, DfE EES, Ofsted, finance, deprivation and council admission-distance sources |
|
|
| `raw` | Singer taps and the PostgreSQL target configured in `pipeline/meltano.yml` |
|
|
| Staging/intermediate/marts | dbt models in `pipeline/transform`; marts are materialized tables |
|
|
| `marts.dim_school`, `marts.dim_location` | School identity and location, filtered to supported England establishments |
|
|
| `marts.fact_*` | Performance and supplementary datasets; coverage and years vary |
|
|
| Typesense `schools` alias | Search documents built by `pipeline/scripts/sync_typesense.py` |
|
|
| `payload` | CMS collections and migrations in `nextjs-app/`; independent of dbt |
|
|
| Media volume | Uploaded blog media; requires backup and cannot be regenerated from school datasets |
|
|
|
|
`backend/models.py` maps existing marts for reading. It does not create the school
|
|
schema. There is no startup schema-version migration or CSV reimport. Payload's
|
|
`nextjs-app/migrations/` is active and must not be confused with the removed
|
|
legacy backend migration code.
|
|
|
|
Coordinates normally come from GIAS British National Grid coordinates transformed
|
|
by PostGIS in `dim_location.sql`. `pipeline/scripts/geocode_postcodes.py` is a
|
|
manual fallback utility, not a task wired into the current school-data DAG.
|
|
Backend postcode searches also use postcodes.io; that lookup does not populate
|
|
school coordinates in the database.
|
|
|
|
## Backend boundaries
|
|
|
|
- `app.py`: routes, middleware, search filtering, sitemap/place publication and response assembly.
|
|
- `data_loader.py`: SQL loading, process-local DataFrame caches, Typesense calls,
|
|
postcode lookups, supplementary queries and benchmark calculation.
|
|
- `database.py`: synchronous SQLAlchemy engine and sessions.
|
|
- `schemas.py`: metric definitions, column mappings and display metadata; despite
|
|
its name this is not a collection of Pydantic API response models.
|
|
- `places.py` and `localities.py`: place registry and curated locality information.
|
|
- `flags.py`: Unleash-backed feature flags, disabled when no server is configured.
|
|
- `gias_codes.py` / `ofsted_codes.py`: source-code translation and display rules.
|
|
|
|
Search starts from a cached latest-row-per-school snapshot. Detail pages read
|
|
history from the full DataFrame and supplementary data from marts. Comparisons
|
|
batch supplementary queries across selected URNs. Async routes still contain
|
|
synchronous dependency calls; a fully asynchronous database layer is not present.
|
|
|
|
## Frontend boundaries
|
|
|
|
`app/(frontend)` owns the public root layout and pages. `app/(payload)` owns the
|
|
CMS root layout. Do not add a shared `app/layout.tsx`: these groups deliberately
|
|
have separate root layouts. Root metadata files remain in `app/`.
|
|
|
|
Server pages fetch initial data and pass it to client views. Client state uses
|
|
React hooks, URL search parameters and the comparison context/localStorage.
|
|
There is no SWR dependency. Leaflet maps are loaded through dynamic wrappers;
|
|
Chart.js renders performance and comparison charts.
|
|
|
|
`components/school/` contains detail sections, with section decisions and data
|
|
preparation in `lib/schoolSections.ts`. `lib/types.ts` contains manually maintained
|
|
API types. `payload-types.ts` and the Payload import map are generated artifacts.
|
|
|
|
## Publication and caching today
|
|
|
|
1. Airflow DAGs extract and validate source data, then run selected dbt builds.
|
|
2. Relevant DAGs rebuild Typesense and swap the `schools` alias.
|
|
3. They call `POST /api/admin/reload` with `X-API-Key` to refresh school DataFrames.
|
|
4. A separate weekly sitemap DAG calls `POST /api/admin/regenerate-sitemap`,
|
|
rebuilding places and sitemaps.
|
|
|
|
GIAS is scheduled daily, Ofsted monthly, and annual datasets are manually
|
|
triggered. The DAG definitions are authoritative for selectors and dependencies.
|
|
|
|
Caches exist in several independent layers: backend DataFrames and registries,
|
|
backend HTTP Cache-Control/ETags, Next.js fetch/page revalidation, and browser or
|
|
shared HTTP caches where configured. Place fetches request a one-week revalidation
|
|
interval. HTTP ETags are computed after route execution, not before database work.
|
|
|
|
Known limitations: reload clears the old DataFrames before verifying replacement
|
|
data; places/sitemaps refresh separately; Next.js caches are not explicitly purged
|
|
by the pipeline; Typesense import results are not validated before alias publication.
|
|
Do not describe this sequence as an atomic dataset release. These are follow-up
|
|
reliability tasks, not changes implemented by the documentation cleanup.
|
|
|
|
## Deployment references
|
|
|
|
See [DEPLOY.md](DEPLOY.md). PR checks include frontend typechecking/tests, backend
|
|
unit tests, image builds and AI review. Staging journeys run after merging.
|
|
Production promotion retags a selected commit's images. Current health polling
|
|
checks HTTP success, not the deployed commit identity; overlapping staging runs
|
|
remain a release-verification concern.
|