school_compare/claude.md

SchoolCompare.co.uk - Project Context

Overview

SchoolCompare is a web application for comparing UK primary school (KS2) performance data. It allows users to:

• Search and browse schools by name, location (postcode), or local authority
• Compare multiple schools side-by-side with charts and tables
• View school rankings by various KS2 metrics
• See historical performance trends across years

Architecture

Backend (Python/FastAPI)

• Framework: FastAPI with uvicorn
• Database: PostgreSQL with SQLAlchemy ORM
• Data Source: UK Government "Compare School Performance" CSV downloads

Key files:

• backend/app.py - Main FastAPI application, API routes
• backend/config.py - Configuration via pydantic-settings (env vars, .env file)
• backend/database.py - SQLAlchemy engine, session management
• backend/models.py - Database models (School, SchoolResult)
• backend/data_loader.py - Data queries, geocoding, legacy DataFrame compatibility
• backend/schemas.py - Column mappings, metric definitions, LA code mappings

Content / CMS (Payload)

Payload CMS runs inside the Next.js app — one image, one container, no separate service. It powers /blog; /about is a plain coded page.

• Admin panel: /admin. The only authenticated surface on the site. noindex via both robots.txt and X-Robots-Tag.
• CMS API: /cms-api, not /api. /api/* is a catch-all proxy to FastAPI (app/(frontend)/api/[...path]) which would silently swallow every admin call and forward it to the backend. Mount points are defined once in lib/payloadRoutes.ts.
• Database: the existing Postgres, in its own payload schema, so no pipeline operation on public — including scripts/migrate_csv_to_db.py --drop — can reach blog content.
• Uploads: the payload_media Docker volume at /app/media. Not reproducible from the pipeline; must be backed up.
• New env vars: DATABASE_URL and PAYLOAD_SECRET on the frontend service. Staging must use a different PAYLOAD_SECRET from production.
• Publishing workflow and house style: nextjs-app/docs/PUBLISHING.md.
• Admin field components resolve through a generated import map (app/(payload)/admin/importMap.js). Payload hands the client a path per field and looks it up there; a missing entry renders no field and reports no error, while required still blocks the save. After adding or changing any field, editor or lexical feature, run npm run generate:importmap in nextjs-app/ and commit the result.

Two route groups

nextjs-app/app/ has no root layout.tsx. It cannot: Payload's admin panel ships its own root layout rendering <html>/<body>, and Next permits multiple root layouts only when no app/layout.tsx exists.

• app/(frontend)/ — the site. Its layout.tsx is the site's root layout.
• app/(payload)/ — the admin panel and /cms-api.

Route groups are invisible to routing, so every public URL is unchanged.

The metadata file conventions stay at the app/ root — robots.ts, opengraph-image.tsx, icon.png, apple-icon.png. Inside a route group Next treats them as segment-scoped: it renames /icon.png to /icon-<hash>.png and drops /robots.txt entirely. Route handlers are unaffected.

The build must succeed with DATABASE_URL unset, because CI builds it that way. Never call getCachedPayload() at module scope, and never add generateStaticParams to a DB-backed route.

Frontend (Vanilla JS)

• Single-page application with hash-based routing
• Chart.js for data visualization
• No build step required

Key files:

• frontend/index.html - Main HTML structure
• frontend/app.js - All application logic, API calls, rendering
• frontend/styles.css - Styling (CSS variables, responsive design)

Database Schema

schools                          school_results
├── id (PK)                      ├── id (PK)
├── urn (unique, indexed)        ├── school_id (FK → schools.id)
├── school_name                  ├── year (indexed)
├── local_authority              ├── rwm_expected_pct
├── school_type                  ├── reading_expected_pct
├── postcode                     ├── ... (all KS2 metrics)
├── latitude, longitude          └── unique(school_id, year)
└── results → SchoolResult[]

Configuration

Environment variables (or .env file):

• DATABASE_URL - PostgreSQL connection string (default: postgresql://schoolcompare:schoolcompare@localhost:5432/schoolcompare)
• HOST, PORT - Server binding (default: 0.0.0.0:80)
• ALLOWED_ORIGINS - CORS origins

Running Locally

1. Start PostgreSQL:

   docker compose up -d db
   

2. Run migration to import CSV data:

   python scripts/migrate_csv_to_db.py --drop
   # Add --geocode to geocode postcodes (slower, adds lat/long)
   

3. Start the app:

   uvicorn backend.app:app --host 0.0.0.0 --port 8000
   

Docker Deployment

docker compose up -d

This starts:

• db - PostgreSQL 16 with persistent volume
• app - FastAPI application on port 80

Data

• Source: UK Government Compare School Performance downloads
• Location: data/ directory with year folders (e.g., 2023-2024/england_ks2final.csv)
• The scripts/download_data.py can fetch data from the government website

Key Features

• Location Search: Enter postcode to find nearby schools (uses postcodes.io API)
• Multi-school Comparison: Select multiple schools, view metrics across years
• Rankings: Top schools by any KS2 metric, filterable by local authority
• Variability Analysis: Shows standard deviation of scores across years

API Endpoints

• GET /api/schools - List/search schools (supports pagination, location search)
• GET /api/schools/{urn} - School details with all yearly data
• GET /api/compare?urns=123,456 - Compare multiple schools
• GET /api/rankings - School rankings by metric
• GET /api/filters - Available filter options (LAs, types, years)
• GET /api/metrics - Metric definitions (single source of truth)
• GET /api/data-info - Database stats

SDLC

Full details in docs/DEPLOY.md. The short version:

• Never push to main directly. Work on a feature branch and open a PR; branch protection requires the PR checks (typecheck, tests, builds, AI review) to pass before merge.
• Merging to main deploys automatically to staging only: images are built once, deployed to the staging Portainer stack, and verified by the Playwright journeys in e2e/. Production is a second, manual approval: the "Promote to Production (manual)" workflow in Gitea Actions, run after testing the feature on staging. It refuses commits whose staging E2E gate isn't green. Never trigger it yourself — promotion is the human's call.
• If you change user-facing behaviour, update or extend the e2e/ journey tests in the same PR — they gate whether staging is fit for human testing and whether a commit is promotable.

Recent Changes

• Added staging environment + automated staging→prod pipeline (Gitea Actions)
• Migrated from CSV file storage to PostgreSQL database
• Added location-based search using postcode geocoding
• Added local authority filter to rankings
• Improved frontend with featured schools, loading states, API caching

Important

• Do not attempt to start a local server to test the application, it does not work