school_compare/docs/DEPLOY.md

SDLC & Deployment Pipeline

SchoolCompare uses a two-stage deploy model on Gitea Actions with two human approvals. AI writes the code on feature branches; the first approval merges the PR, which deploys to staging and runs the E2E gate; the second approval — after manual testing on staging — promotes the exact same images to production via a manual workflow.

The flow

feature branch (AI-authored)
   │  PR to main                                    ← approval #1
   ▼
PR checks (.gitea/workflows/pr-checks.yml)
   typecheck + unit tests + backend smoke + image builds (no push)
   + Claude code review posted as a PR comment (severe findings fail the check)
   │  merge (branch protection requires green checks)
   ▼
Stage pipeline (.gitea/workflows/deploy.yml) — automatic
   1. build & push images        → tags sha-<sha>, staging
   2. staging Portainer webhook  → wait for staging health
   3. Playwright E2E journeys against staging      ← gate before human testing
   ▼
Manual testing on staging (stx.schoolcompare.co.uk)
   │  Actions → "Promote to Production (manual)"   ← approval #2
   ▼
Promote pipeline (.gitea/workflows/promote.yml) — manual dispatch
   1. resolve target sha (input, or latest main if empty)
   2. REFUSE unless that commit's "E2E Journeys against Staging" status is green
   3. retag sha-<sha> → :prod    (same bytes — build once, promote the image)
      previous :prod saved as :prod-previous
   4. prod Portainer webhook     → wait for prod health

Key principle: build once, promote the exact image. Production pins :prod, which only moves when a human runs the promote workflow — and the workflow only accepts commits that passed the staging E2E gate. Nothing tags :latest anymore.

Branch & PR workflow

• main is protected: no direct pushes, PRs require green status checks.
• All work (human or AI) happens on feature branches → PR to main.
• Merging to main releases to staging only. Production moves only on the second approval. If staging or the E2E gate fails, fix forward — production is untouched either way.

Promotion granularity

Staging always runs the latest main. Promoting approves a state of main, not a single PR — if two PRs merged since the last promotion, they ship together. Test staging accordingly. To promote an older state, pass its commit SHA to the promote workflow (its images must still exist in the registry).

Staging quirk for manual testing: external /api is broken at the staging proxy — exercise API endpoints from the host, not via the public staging URL.

Environments

	Production	Staging
Portainer stack file	docker-compose.portainer.yml	docker-compose.portainer.staging.yml
Image tag	:prod	:staging
Container prefix	sc_ / schoolcompare_	sc_staging_
Frontend macvlan IP	10.0.1.150	STAGING_FRONTEND_IP (default 10.0.1.151)
Postgres macvlan IP	10.0.1.189	STAGING_DB_IP (default 10.0.1.190)
Airflow UI port	8080	8081
Volumes	stack-prefixed	stack-prefixed (fully isolated)

Staging gets :staging images on every merge to main — even ones that later fail the E2E gate. That's the point: staging absorbs the risk.

Gitea repository secrets

Secret	Purpose
REGISTRY_TOKEN	push images to privaterepo.sitaru.org (already set)
CLAUDE_CODE_OAUTH_TOKEN	Claude Code subscription auth for the PR review — generate with claude setup-token on your machine
PORTAINER_STAGING_WEBHOOK	staging stack redeploy webhook URL
PORTAINER_PROD_WEBHOOK	production stack redeploy webhook URL
STAGING_BASE_URL	e.g. http://10.0.1.151:3000 — health poll + E2E target
PROD_BASE_URL	e.g. http://10.0.1.150:3000 — post-promotion health poll

One-time setup checklist

1. Create the staging stack in Portainer from docker-compose.portainer.staging.yml (stack name e.g. schoolcompare-staging). Set the same environment variables as prod plus STAGING_DB_IP / STAGING_FRONTEND_IP if the defaults clash.
2. Enable webhooks on both stacks (Portainer → Stack → Webhook) and store the URLs as PORTAINER_STAGING_WEBHOOK / PORTAINER_PROD_WEBHOOK. Remove the old hardcoded webhook usage (now gone from the workflows).
3. Add the remaining secrets listed above in Gitea → repo → Settings → Actions → Secrets.
4. Protect main in Gitea → Settings → Branches: require PRs, require the pr-checks status checks (frontend, backend, builds, ai-review) to pass.
5. Bootstrap staging data via Airflow (no prod dump — staging populates itself from source, exercising the pipeline image end-to-end):
   • Open the staging Airflow UI (http://<host>:8081) and trigger, in order: school_data_daily, school_data_monthly_ofsted, then the manual-schedule school_data_annual_ees and school_data_annual_idaci.
   • First runs download from government sources (GIAS, Ofsted, EES, IDACI), run dbt, and sync Typesense — expect the initial backfill to take a while.
   • The scheduled DAGs then keep staging fresh exactly like prod.
6. Switch the prod stack to :prod tags — the repo's docker-compose.portainer.yml is already updated; redeploy the prod stack from it. Until the first pipeline run promotes an image, tag the current images manually: docker buildx imagetools create -t <image>:prod <image>:latest for each of the three images.

Rollback

Re-run "Promote to Production (manual)" with the SHA of the last good commit (fastest, fully gated), or manually re-point the tags — every promotion first saves the outgoing :prod as :prod-previous:

for img in backend frontend pipeline; do
  docker buildx imagetools create \
    -t privaterepo.sitaru.org/tudor/school_compare-$img:prod \
    privaterepo.sitaru.org/tudor/school_compare-$img:prod-previous
done
curl -fsSk -X POST "$PORTAINER_PROD_WEBHOOK"

Or promote any older build directly: imagetools create -t <image>:prod <image>:sha-<shortsha>.

E2E suite

Lives in e2e/ (own package — CI installs it without the app's node_modules). Journeys: home + name search, postcode search, school detail, two-school comparison, rankings table. Run locally against any environment:

cd e2e && npm ci
BASE_URL=http://10.0.1.151:3000 npx playwright test

Tests assert data invariants (results exist, charts render), not exact numbers, so scheduled data refreshes don't break the gate.

AI code review

scripts/ci/ai_review.py pipes the PR diff through headless Claude Code (claude -p, authenticated with the subscription OAuth token — no API billing), posts the structured findings as a PR comment using the per-run token Gitea Actions provides automatically (secrets.GITEA_TOKEN — no setup needed), and fails the check only when a finding is rated severe (would break prod, leak data, or corrupt data). Minor findings are informational and never block a merge.