school_compare/claude.md

# SchoolCompare.co.uk - Project Context

## Overview

SchoolCompare is a web application for comparing UK primary school (KS2) performance data. It allows users to:
- Search and browse schools by name, location (postcode), or local authority
- Compare multiple schools side-by-side with charts and tables
- View school rankings by various KS2 metrics
- See historical performance trends across years

## Architecture

### Backend (Python/FastAPI)
- **Framework**: FastAPI with uvicorn
- **Database**: PostgreSQL with SQLAlchemy ORM
- **Data Source**: UK Government "Compare School Performance" CSV downloads

Key files:
- `backend/app.py` - Main FastAPI application, API routes
- `backend/config.py` - Configuration via pydantic-settings (env vars, .env file)
- `backend/database.py` - SQLAlchemy engine, session management
- `backend/models.py` - Database models (School, SchoolResult)
- `backend/data_loader.py` - Data queries, geocoding, legacy DataFrame compatibility
- `backend/schemas.py` - Column mappings, metric definitions, LA code mappings

### Content / CMS (Payload)

Payload CMS runs **inside** the Next.js app — one image, one container, no
separate service. It powers `/blog`; `/about` is a plain coded page.

- **Admin panel:** `/admin`. The only authenticated surface on the site.
  `noindex` via both `robots.txt` and `X-Robots-Tag`.
- **CMS API:** `/cms-api`, **not** `/api`. `/api/*` is a catch-all proxy to
  FastAPI (`app/(frontend)/api/[...path]`) which would silently swallow every
  admin call and forward it to the backend. Mount points are defined once in
  `lib/payloadRoutes.ts`.
- **Database:** the existing Postgres, in its own `payload` schema, so no
  pipeline operation on `public` — including
  `scripts/migrate_csv_to_db.py --drop` — can reach blog content.
- **Uploads:** the `payload_media` Docker volume at `/app/media`. Not
  reproducible from the pipeline; must be backed up.
- **New env vars:** `DATABASE_URL` and `PAYLOAD_SECRET` on the frontend service.
  Staging must use a different `PAYLOAD_SECRET` from production.
- Publishing workflow and house style: `nextjs-app/docs/PUBLISHING.md`.
- **Admin field components resolve through a generated import map**
  (`app/(payload)/admin/importMap.js`). Payload hands the client a *path* per
  field and looks it up there; a missing entry renders no field and reports no
  error, while `required` still blocks the save. After adding or changing any
  field, editor or lexical feature, run `npm run generate:importmap` in
  `nextjs-app/` and commit the result.

### Two route groups

`nextjs-app/app/` has no root `layout.tsx`. It cannot: Payload's admin panel
ships its own root layout rendering `<html>`/`<body>`, and Next permits
multiple root layouts only when no `app/layout.tsx` exists.

- `app/(frontend)/` — the site. Its `layout.tsx` is the site's root layout.
- `app/(payload)/` — the admin panel and `/cms-api`.

Route groups are invisible to routing, so every public URL is unchanged.

**The metadata file conventions stay at the `app/` root** — `robots.ts`,
`opengraph-image.tsx`, `icon.png`, `apple-icon.png`. Inside a route group Next
treats them as segment-scoped: it renames `/icon.png` to `/icon-<hash>.png` and
drops `/robots.txt` entirely. Route handlers are unaffected.

The build must succeed with `DATABASE_URL` unset, because CI builds it that
way. Never call `getCachedPayload()` at module scope, and never add
`generateStaticParams` to a DB-backed route.

### Frontend (Vanilla JS)
- Single-page application with hash-based routing
- Chart.js for data visualization
- No build step required

Key files:
- `frontend/index.html` - Main HTML structure
- `frontend/app.js` - All application logic, API calls, rendering
- `frontend/styles.css` - Styling (CSS variables, responsive design)

### Database Schema

```
schools                          school_results
├── id (PK)                      ├── id (PK)
├── urn (unique, indexed)        ├── school_id (FK → schools.id)
├── school_name                  ├── year (indexed)
├── local_authority              ├── rwm_expected_pct
├── school_type                  ├── reading_expected_pct
├── postcode                     ├── ... (all KS2 metrics)
├── latitude, longitude          └── unique(school_id, year)
└── results → SchoolResult[]
```

## Configuration

Environment variables (or `.env` file):
- `DATABASE_URL` - PostgreSQL connection string (default: `postgresql://schoolcompare:schoolcompare@localhost:5432/schoolcompare`)
- `HOST`, `PORT` - Server binding (default: `0.0.0.0:80`)
- `ALLOWED_ORIGINS` - CORS origins

## Running Locally

1. Start PostgreSQL:
   ```bash
   docker compose up -d db
   ```

2. Run migration to import CSV data:
   ```bash
   python scripts/migrate_csv_to_db.py --drop
   # Add --geocode to geocode postcodes (slower, adds lat/long)
   ```

3. Start the app:
   ```bash
   uvicorn backend.app:app --host 0.0.0.0 --port 8000
   ```

## Docker Deployment

```bash
docker compose up -d
```

This starts:
- `db` - PostgreSQL 16 with persistent volume
- `app` - FastAPI application on port 80

## Data

- Source: UK Government Compare School Performance downloads
- Location: `data/` directory with year folders (e.g., `2023-2024/england_ks2final.csv`)
- The `scripts/download_data.py` can fetch data from the government website

## Key Features

- **Location Search**: Enter postcode to find nearby schools (uses postcodes.io API)
- **Multi-school Comparison**: Select multiple schools, view metrics across years
- **Rankings**: Top schools by any KS2 metric, filterable by local authority
- **Variability Analysis**: Shows standard deviation of scores across years

## API Endpoints

- `GET /api/schools` - List/search schools (supports pagination, location search)
- `GET /api/schools/{urn}` - School details with all yearly data
- `GET /api/compare?urns=123,456` - Compare multiple schools
- `GET /api/rankings` - School rankings by metric
- `GET /api/filters` - Available filter options (LAs, types, years)
- `GET /api/metrics` - Metric definitions (single source of truth)
- `GET /api/data-info` - Database stats

## SDLC

Full details in `docs/DEPLOY.md`. The short version:

- **Never push to `main` directly.** Work on a feature branch and open a PR;
  branch protection requires the PR checks (typecheck, tests, builds, AI review)
  to pass before merge.
- Merging to `main` deploys automatically **to staging only**: images are
  built once, deployed to the staging Portainer stack, and verified by the
  Playwright journeys in `e2e/`. Production is a second, manual approval:
  the "Promote to Production (manual)" workflow in Gitea Actions, run after
  testing the feature on staging. It refuses commits whose staging E2E gate
  isn't green. Never trigger it yourself — promotion is the human's call.
- If you change user-facing behaviour, update or extend the `e2e/` journey
  tests in the same PR — they gate whether staging is fit for human testing
  and whether a commit is promotable.

## Recent Changes

- Added staging environment + automated staging→prod pipeline (Gitea Actions)
- Migrated from CSV file storage to PostgreSQL database
- Added location-based search using postcode geocoding
- Added local authority filter to rankings
- Improved frontend with featured schools, loading states, API caching

# Important
- Do not attempt to start a local server to test the application, it does not work