school_compare/docs/DEPLOY.md

# SDLC & Deployment Pipeline

SchoolCompare uses a fully automated staging → production pipeline on Gitea
Actions. AI writes the code on feature branches; the pipeline verifies every
change on a staging environment before promoting the exact same images to
production. Human input is directional only: feature requests, PR review if
desired, and intervention when a gate fails.

## The flow

```
feature branch (AI-authored)
   │  PR to main
   ▼
PR checks (.gitea/workflows/pr-checks.yml)
   typecheck + unit tests + backend smoke + image builds (no push)
   + Claude code review posted as a PR comment (severe findings fail the check)
   │  merge (branch protection requires green checks)
   ▼
Deploy pipeline (.gitea/workflows/deploy.yml)
   1. build & push images        → tags sha-<sha>, staging
   2. staging Portainer webhook  → wait for staging health
   3. Playwright E2E journeys against staging
   4. retag sha-<sha> → :prod    (same bytes — build once, promote the image)
      previous :prod saved as :prod-previous
   5. prod Portainer webhook     → wait for prod health
```

Key principle: **build once, promote the exact image**. Production pins `:prod`,
which only moves after the E2E gate passes on staging. Nothing tags `:latest`
anymore.

## Branch & PR workflow

- `main` is protected: no direct pushes, PRs require green status checks.
- All work (human or AI) happens on feature branches → PR to `main`.
- Merging to `main` **is** the release action. If staging or the E2E gate
  fails, production is untouched.

## Environments

| | Production | Staging |
|---|---|---|
| Portainer stack file | `docker-compose.portainer.yml` | `docker-compose.portainer.staging.yml` |
| Image tag | `:prod` | `:staging` |
| Container prefix | `sc_` / `schoolcompare_` | `sc_staging_` |
| Frontend macvlan IP | 10.0.1.150 | `STAGING_FRONTEND_IP` (default 10.0.1.151) |
| Postgres macvlan IP | 10.0.1.189 | `STAGING_DB_IP` (default 10.0.1.190) |
| Airflow UI port | 8080 | 8081 |
| Volumes | stack-prefixed | stack-prefixed (fully isolated) |

Staging gets `:staging` images on every merge to main — even ones that later
fail the E2E gate. That's the point: staging absorbs the risk.

## Gitea repository secrets

| Secret | Purpose |
|---|---|
| `REGISTRY_TOKEN` | push images to privaterepo.sitaru.org (already set) |
| `ANTHROPIC_API_KEY` | Claude PR review (`scripts/ci/ai_review.py`) |
| `GITEA_TOKEN` | post PR review comments (needs issue-comment scope) |
| `PORTAINER_STAGING_WEBHOOK` | staging stack redeploy webhook URL |
| `PORTAINER_PROD_WEBHOOK` | production stack redeploy webhook URL |
| `STAGING_BASE_URL` | e.g. `http://10.0.1.151:3000` — health poll + E2E target |
| `PROD_BASE_URL` | e.g. `http://10.0.1.150:3000` — post-promotion health poll |

## One-time setup checklist

1. **Create the staging stack** in Portainer from
   `docker-compose.portainer.staging.yml` (stack name e.g.
   `schoolcompare-staging`). Set the same environment variables as prod plus
   `STAGING_DB_IP` / `STAGING_FRONTEND_IP` if the defaults clash.
2. **Enable webhooks** on both stacks (Portainer → Stack → Webhook) and store
   the URLs as `PORTAINER_STAGING_WEBHOOK` / `PORTAINER_PROD_WEBHOOK`. Remove
   the old hardcoded webhook usage (now gone from the workflows).
3. **Add the remaining secrets** listed above in Gitea → repo → Settings →
   Actions → Secrets.
4. **Protect `main`** in Gitea → Settings → Branches: require PRs, require the
   pr-checks status checks (frontend, backend, builds, ai-review) to pass.
5. **Bootstrap staging data via Airflow** (no prod dump — staging populates
   itself from source, exercising the pipeline image end-to-end):
   - Open the staging Airflow UI (`http://<host>:8081`) and trigger, in order:
     `school_data_daily`, `school_data_monthly_ofsted`,
     `school_data_monthly_parent_view`, then the manual-schedule
     `school_data_annual_ees` and `school_data_annual_idaci`.
   - First runs download from government sources (GIAS, Ofsted, EES, IDACI),
     run dbt, and sync Typesense — expect the initial backfill to take a while.
   - The scheduled DAGs then keep staging fresh exactly like prod.
6. **Switch the prod stack to `:prod` tags** — the repo's
   `docker-compose.portainer.yml` is already updated; redeploy the prod stack
   from it. Until the first pipeline run promotes an image, tag the current
   images manually: `docker buildx imagetools create -t <image>:prod <image>:latest`
   for each of the three images.

## Rollback

Every promotion first re-points `:prod-previous` at the outgoing `:prod`.
To roll back:

```bash
for img in backend frontend pipeline; do
  docker buildx imagetools create \
    -t privaterepo.sitaru.org/tudor/school_compare-$img:prod \
    privaterepo.sitaru.org/tudor/school_compare-$img:prod-previous
done
curl -fsSk -X POST "$PORTAINER_PROD_WEBHOOK"
```

Or promote any older build directly: `imagetools create -t <image>:prod <image>:sha-<shortsha>`.

## E2E suite

Lives in `e2e/` (own package — CI installs it without the app's node_modules).
Journeys: home + name search, postcode search, school detail, two-school
comparison, rankings table. Run locally against any environment:

```bash
cd e2e && npm ci
BASE_URL=http://10.0.1.151:3000 npx playwright test
```

Tests assert data invariants (results exist, charts render), not exact
numbers, so scheduled data refreshes don't break the gate.

## AI code review

`scripts/ci/ai_review.py` sends the PR diff to Claude (`claude-opus-4-8`),
posts the structured findings as a PR comment, and fails the check only when a
finding is rated **severe** (would break prod, leak data, or corrupt data).
Minor findings are informational and never block a merge.