school_compare/docs/DEPLOY.md

SDLC & Deployment Pipeline

SchoolCompare uses a fully automated staging → production pipeline on Gitea Actions. AI writes the code on feature branches; the pipeline verifies every change on a staging environment before promoting the exact same images to production. Human input is directional only: feature requests, PR review if desired, and intervention when a gate fails.

The flow

feature branch (AI-authored)
   │  PR to main
   ▼
PR checks (.gitea/workflows/pr-checks.yml)
   typecheck + unit tests + backend smoke + image builds (no push)
   + Claude code review posted as a PR comment (severe findings fail the check)
   │  merge (branch protection requires green checks)
   ▼
Deploy pipeline (.gitea/workflows/deploy.yml)
   1. build & push images        → tags sha-<sha>, staging
   2. staging Portainer webhook  → wait for staging health
   3. Playwright E2E journeys against staging
   4. retag sha-<sha> → :prod    (same bytes — build once, promote the image)
      previous :prod saved as :prod-previous
   5. prod Portainer webhook     → wait for prod health

Key principle: build once, promote the exact image. Production pins :prod, which only moves after the E2E gate passes on staging. Nothing tags :latest anymore.

Branch & PR workflow

• main is protected: no direct pushes, PRs require green status checks.
• All work (human or AI) happens on feature branches → PR to main.
• Merging to main is the release action. If staging or the E2E gate fails, production is untouched.

Environments

	Production	Staging
Portainer stack file	docker-compose.portainer.yml	docker-compose.portainer.staging.yml
Image tag	:prod	:staging
Container prefix	sc_ / schoolcompare_	sc_staging_
Frontend macvlan IP	10.0.1.150	STAGING_FRONTEND_IP (default 10.0.1.151)
Postgres macvlan IP	10.0.1.189	STAGING_DB_IP (default 10.0.1.190)
Airflow UI port	8080	8081
Volumes	stack-prefixed	stack-prefixed (fully isolated)

Staging gets :staging images on every merge to main — even ones that later fail the E2E gate. That's the point: staging absorbs the risk.

Gitea repository secrets

Secret	Purpose
REGISTRY_TOKEN	push images to privaterepo.sitaru.org (already set)
ANTHROPIC_API_KEY	Claude PR review (scripts/ci/ai_review.py)
GITEA_TOKEN	post PR review comments (needs issue-comment scope)
PORTAINER_STAGING_WEBHOOK	staging stack redeploy webhook URL
PORTAINER_PROD_WEBHOOK	production stack redeploy webhook URL
STAGING_BASE_URL	e.g. http://10.0.1.151:3000 — health poll + E2E target
PROD_BASE_URL	e.g. http://10.0.1.150:3000 — post-promotion health poll

One-time setup checklist

1. Create the staging stack in Portainer from docker-compose.portainer.staging.yml (stack name e.g. schoolcompare-staging). Set the same environment variables as prod plus STAGING_DB_IP / STAGING_FRONTEND_IP if the defaults clash.
2. Enable webhooks on both stacks (Portainer → Stack → Webhook) and store the URLs as PORTAINER_STAGING_WEBHOOK / PORTAINER_PROD_WEBHOOK. Remove the old hardcoded webhook usage (now gone from the workflows).
3. Add the remaining secrets listed above in Gitea → repo → Settings → Actions → Secrets.
4. Protect main in Gitea → Settings → Branches: require PRs, require the pr-checks status checks (frontend, backend, builds, ai-review) to pass.
5. Bootstrap staging data via Airflow (no prod dump — staging populates itself from source, exercising the pipeline image end-to-end):
   • Open the staging Airflow UI (http://<host>:8081) and trigger, in order: school_data_daily, school_data_monthly_ofsted, school_data_monthly_parent_view, then the manual-schedule school_data_annual_ees and school_data_annual_idaci.
   • First runs download from government sources (GIAS, Ofsted, EES, IDACI), run dbt, and sync Typesense — expect the initial backfill to take a while.
   • The scheduled DAGs then keep staging fresh exactly like prod.
6. Switch the prod stack to :prod tags — the repo's docker-compose.portainer.yml is already updated; redeploy the prod stack from it. Until the first pipeline run promotes an image, tag the current images manually: docker buildx imagetools create -t <image>:prod <image>:latest for each of the three images.

Rollback

Every promotion first re-points :prod-previous at the outgoing :prod. To roll back:

for img in backend frontend pipeline; do
  docker buildx imagetools create \
    -t privaterepo.sitaru.org/tudor/school_compare-$img:prod \
    privaterepo.sitaru.org/tudor/school_compare-$img:prod-previous
done
curl -fsSk -X POST "$PORTAINER_PROD_WEBHOOK"

Or promote any older build directly: imagetools create -t <image>:prod <image>:sha-<shortsha>.

E2E suite

Lives in e2e/ (own package — CI installs it without the app's node_modules). Journeys: home + name search, postcode search, school detail, two-school comparison, rankings table. Run locally against any environment:

cd e2e && npm ci
BASE_URL=http://10.0.1.151:3000 npx playwright test

Tests assert data invariants (results exist, charts render), not exact numbers, so scheduled data refreshes don't break the gate.

AI code review

scripts/ci/ai_review.py sends the PR diff to Claude (claude-opus-4-8), posts the structured findings as a PR comment, and fails the check only when a finding is rated severe (would break prod, leak data, or corrupt data). Minor findings are informational and never block a merge.