0 Fork 0

Files

T

TudorandClaude Opus 5 eaf5e5d180 docs: describe the system that exists, not the one we started with

The README still opened on "Primary School Compass", a KS2 tool for Wandsworth
and Merton served by FastAPI and vanilla JavaScript with Chart.js. Every layer
of that sentence is now wrong: coverage is England-wide across KS2, KS4,
all-through and post-16, Next.js owns the public UI, and school data comes from
dbt-built `marts.*` rather than CSVs loaded at startup. The setup instructions
walked a reader into a virtualenv and a CSV import that cannot build the current
schema, so following the docs produced an empty database and a wrong mental
model at the same time.

Replace the narrative docs with two reference documents that were checked
against the code: docs/ARCHITECTURE.md for request flow, data ownership, the
backend/frontend module boundaries and the real publication sequence, and
docs/DEVELOPMENT.md for the checks that actually run, including the container
and CI version skew that makes "just run pytest" misleading.

The env examples drifted the same way. ALLOWED_ORIGINS is a JSON array, not a
comma-separated list; the frontend needs FASTAPI_URL, DATABASE_URL and
PAYLOAD_SECRET, none of which were documented; and RATE_LIMIT_BURST,
DEFAULT_PAGE_SIZE and MAX_PAGE_SIZE were presented as tuning controls the routes
do not consult. Each is now stated as it behaves.

MIGRATION_SUMMARY.md keeps its content but gains a banner, because it reads like
setup instructions and is not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016y2J6bs8gbuSJbH18w7Tan

2026-09-14 23:01:15 +01:00

5.7 KiB

Raw Blame History

Architecture

This describes the implementation as reviewed on 2026-09-14. It distinguishes current behaviour from improvements still to be implemented.

Request flow

Browser → Next.js public routes
          ├─ /api/* proxy → FastAPI → cached DataFrames / PostgreSQL marts
          │                         ├─ Typesense (search and suggestions)
          │                         └─ postcodes.io (postcode lookup)
          └─ /admin, /cms-api, /blog → Payload → payload schema + media volume

Next.js server rendering → FastAPI directly through FASTAPI_URL

nextjs-app/lib/api.ts contains typed fetch wrappers and revalidation defaults. The proxy is nextjs-app/app/(frontend)/api/[...path]/route.ts. Payload uses /cms-api so its routes do not collide with the FastAPI proxy. The proxy denies /api/flags; server-side rendering reads flags directly from FastAPI.

Data ownership

Layer	Owner and role
Source data	GIAS, DfE EES, Ofsted, finance, deprivation and council admission-distance sources
`raw`	Singer taps and the PostgreSQL target configured in `pipeline/meltano.yml`
Staging/intermediate/marts	dbt models in `pipeline/transform`; marts are materialized tables
`marts.dim_school`, `marts.dim_location`	School identity and location, filtered to supported England establishments
`marts.fact_*`	Performance and supplementary datasets; coverage and years vary
Typesense `schools` alias	Search documents built by `pipeline/scripts/sync_typesense.py`
`payload`	CMS collections and migrations in `nextjs-app/`; independent of dbt
Media volume	Uploaded blog media; requires backup and cannot be regenerated from school datasets

backend/models.py maps existing marts for reading. It does not create the school schema. There is no startup schema-version migration or CSV reimport. Payload's nextjs-app/migrations/ is active and must not be confused with the removed legacy backend migration code.

Coordinates normally come from GIAS British National Grid coordinates transformed by PostGIS in dim_location.sql. pipeline/scripts/geocode_postcodes.py is a manual fallback utility, not a task wired into the current school-data DAG. Backend postcode searches also use postcodes.io; that lookup does not populate school coordinates in the database.

Backend boundaries

app.py: routes, middleware, search filtering, sitemap/place publication and response assembly.
data_loader.py: SQL loading, process-local DataFrame caches, Typesense calls, postcode lookups, supplementary queries and benchmark calculation.
database.py: synchronous SQLAlchemy engine and sessions.
schemas.py: metric definitions, column mappings and display metadata; despite its name this is not a collection of Pydantic API response models.
places.py and localities.py: place registry and curated locality information.
flags.py: Unleash-backed feature flags, disabled when no server is configured.
gias_codes.py / ofsted_codes.py: source-code translation and display rules.

Search starts from a cached latest-row-per-school snapshot. Detail pages read history from the full DataFrame and supplementary data from marts. Comparisons batch supplementary queries across selected URNs. Async routes still contain synchronous dependency calls; a fully asynchronous database layer is not present.

Frontend boundaries

app/(frontend) owns the public root layout and pages. app/(payload) owns the CMS root layout. Do not add a shared app/layout.tsx: these groups deliberately have separate root layouts. Root metadata files remain in app/.

Server pages fetch initial data and pass it to client views. Client state uses React hooks, URL search parameters and the comparison context/localStorage. There is no SWR dependency. Leaflet maps are loaded through dynamic wrappers; Chart.js renders performance and comparison charts.

components/school/ contains detail sections, with section decisions and data preparation in lib/schoolSections.ts. lib/types.ts contains manually maintained API types. payload-types.ts and the Payload import map are generated artifacts.

Publication and caching today

Airflow DAGs extract and validate source data, then run selected dbt builds.
Relevant DAGs rebuild Typesense and swap the schools alias.
They call POST /api/admin/reload with X-API-Key to refresh school DataFrames.
A separate weekly sitemap DAG calls POST /api/admin/regenerate-sitemap, rebuilding places and sitemaps.

GIAS is scheduled daily, Ofsted monthly, and annual datasets are manually triggered. The DAG definitions are authoritative for selectors and dependencies.

Caches exist in several independent layers: backend DataFrames and registries, backend HTTP Cache-Control/ETags, Next.js fetch/page revalidation, and browser or shared HTTP caches where configured. Place fetches request a one-week revalidation interval. HTTP ETags are computed after route execution, not before database work.

Known limitations: reload clears the old DataFrames before verifying replacement data; places/sitemaps refresh separately; Next.js caches are not explicitly purged by the pipeline; Typesense import results are not validated before alias publication. Do not describe this sequence as an atomic dataset release. These are follow-up reliability tasks, not changes implemented by the documentation cleanup.

Deployment references

See DEPLOY.md. PR checks include frontend typechecking/tests, backend unit tests, image builds and AI review. Staging journeys run after merging. Production promotion retags a selected commit's images. Current health polling checks HTTP success, not the deployed commit identity; overlapping staging runs remain a release-verification concern.

5.7 KiB Raw Blame History