feat(sdlc): staging environment + automated staging→prod pipeline

- pr-checks.yml: PR gate — frontend typecheck+jest, backend import smoke, image builds (no push), Claude AI review posted as PR comment (severe findings block merge) - deploy.yml (replaces build-and-push.yml): merge to main builds+pushes images tagged sha-<sha>/staging, deploys the staging Portainer stack via webhook, runs Playwright E2E journeys against staging, then retags the verified images :prod (previous kept as :prod-previous) and deploys prod - docker-compose.portainer.staging.yml: second Portainer stack — :staging images, sc_staging_* names, own macvlan IPs, Airflow on 8081; data bootstrapped from source via the staging Airflow DAGs - prod compose now pins :prod instead of :latest (only the promotion step moves it; :latest is no longer published) - e2e/: 6 Playwright journeys (search, postcode, detail, compare, rankings) driven by BASE_URL — the promotion gate - scripts/ci/ai_review.py: Claude review with structured JSON findings - docs/DEPLOY.md: full SDLC doc incl. one-time setup checklist and rollback - replaced removed 'next lint' with tsc typecheck; fixed stale jest tests (slug URLs, N/A formatting, stable trend, fake-timer setup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqGhF93UrpDNvXBLMjJENL
2026-07-03 06:50:40 +01:00
parent f2ed49c0a1
commit 4a52735356
15 changed files with 990 additions and 46 deletions
@@ -0,0 +1,130 @@
+# SDLC & Deployment Pipeline
+
+SchoolCompare uses a fully automated staging → production pipeline on Gitea
+Actions. AI writes the code on feature branches; the pipeline verifies every
+change on a staging environment before promoting the exact same images to
+production. Human input is directional only: feature requests, PR review if
+desired, and intervention when a gate fails.
+
+## The flow
+
+```
+feature branch (AI-authored)
+   │  PR to main
+   ▼
+PR checks (.gitea/workflows/pr-checks.yml)
+   typecheck + unit tests + backend smoke + image builds (no push)
+   + Claude code review posted as a PR comment (severe findings fail the check)
+   │  merge (branch protection requires green checks)
+   ▼
+Deploy pipeline (.gitea/workflows/deploy.yml)
+   1. build & push images        → tags sha-<sha>, staging
+   2. staging Portainer webhook  → wait for staging health
+   3. Playwright E2E journeys against staging
+   4. retag sha-<sha> → :prod    (same bytes — build once, promote the image)
+      previous :prod saved as :prod-previous
+   5. prod Portainer webhook     → wait for prod health
+```
+
+Key principle: **build once, promote the exact image**. Production pins `:prod`,
+which only moves after the E2E gate passes on staging. Nothing tags `:latest`
+anymore.
+
+## Branch & PR workflow
+
+- `main` is protected: no direct pushes, PRs require green status checks.
+- All work (human or AI) happens on feature branches → PR to `main`.
+- Merging to `main` **is** the release action. If staging or the E2E gate
+  fails, production is untouched.
+
+## Environments
+
+| | Production | Staging |
+|---|---|---|
+| Portainer stack file | `docker-compose.portainer.yml` | `docker-compose.portainer.staging.yml` |
+| Image tag | `:prod` | `:staging` |
+| Container prefix | `sc_` / `schoolcompare_` | `sc_staging_` |
+| Frontend macvlan IP | 10.0.1.150 | `STAGING_FRONTEND_IP` (default 10.0.1.151) |
+| Postgres macvlan IP | 10.0.1.189 | `STAGING_DB_IP` (default 10.0.1.190) |
+| Airflow UI port | 8080 | 8081 |
+| Volumes | stack-prefixed | stack-prefixed (fully isolated) |
+
+Staging gets `:staging` images on every merge to main — even ones that later
+fail the E2E gate. That's the point: staging absorbs the risk.
+
+## Gitea repository secrets
+
+| Secret | Purpose |
+|---|---|
+| `REGISTRY_TOKEN` | push images to privaterepo.sitaru.org (already set) |
+| `ANTHROPIC_API_KEY` | Claude PR review (`scripts/ci/ai_review.py`) |
+| `GITEA_TOKEN` | post PR review comments (needs issue-comment scope) |
+| `PORTAINER_STAGING_WEBHOOK` | staging stack redeploy webhook URL |
+| `PORTAINER_PROD_WEBHOOK` | production stack redeploy webhook URL |
+| `STAGING_BASE_URL` | e.g. `http://10.0.1.151:3000` — health poll + E2E target |
+| `PROD_BASE_URL` | e.g. `http://10.0.1.150:3000` — post-promotion health poll |
+
+## One-time setup checklist
+
+1. **Create the staging stack** in Portainer from
+   `docker-compose.portainer.staging.yml` (stack name e.g.
+   `schoolcompare-staging`). Set the same environment variables as prod plus
+   `STAGING_DB_IP` / `STAGING_FRONTEND_IP` if the defaults clash.
+2. **Enable webhooks** on both stacks (Portainer → Stack → Webhook) and store
+   the URLs as `PORTAINER_STAGING_WEBHOOK` / `PORTAINER_PROD_WEBHOOK`. Remove
+   the old hardcoded webhook usage (now gone from the workflows).
+3. **Add the remaining secrets** listed above in Gitea → repo → Settings →
+   Actions → Secrets.
+4. **Protect `main`** in Gitea → Settings → Branches: require PRs, require the
+   pr-checks status checks (frontend, backend, builds, ai-review) to pass.
+5. **Bootstrap staging data via Airflow** (no prod dump — staging populates
+   itself from source, exercising the pipeline image end-to-end):
+   - Open the staging Airflow UI (`http://<host>:8081`) and trigger, in order:
+     `school_data_daily`, `school_data_monthly_ofsted`,
+     `school_data_monthly_parent_view`, then the manual-schedule
+     `school_data_annual_ees` and `school_data_annual_idaci`.
+   - First runs download from government sources (GIAS, Ofsted, EES, IDACI),
+     run dbt, and sync Typesense — expect the initial backfill to take a while.
+   - The scheduled DAGs then keep staging fresh exactly like prod.
+6. **Switch the prod stack to `:prod` tags** — the repo's
+   `docker-compose.portainer.yml` is already updated; redeploy the prod stack
+   from it. Until the first pipeline run promotes an image, tag the current
+   images manually: `docker buildx imagetools create -t <image>:prod <image>:latest`
+   for each of the three images.
+
+## Rollback
+
+Every promotion first re-points `:prod-previous` at the outgoing `:prod`.
+To roll back:
+
+```bash
+for img in backend frontend pipeline; do
+  docker buildx imagetools create \
+    -t privaterepo.sitaru.org/tudor/school_compare-$img:prod \
+    privaterepo.sitaru.org/tudor/school_compare-$img:prod-previous
+done
+curl -fsSk -X POST "$PORTAINER_PROD_WEBHOOK"
+```
+
+Or promote any older build directly: `imagetools create -t <image>:prod <image>:sha-<shortsha>`.
+
+## E2E suite
+
+Lives in `e2e/` (own package — CI installs it without the app's node_modules).
+Journeys: home + name search, postcode search, school detail, two-school
+comparison, rankings table. Run locally against any environment:
+
+```bash
+cd e2e && npm ci
+BASE_URL=http://10.0.1.151:3000 npx playwright test
+```
+
+Tests assert data invariants (results exist, charts render), not exact
+numbers, so scheduled data refreshes don't break the gate.
+
+## AI code review
+
+`scripts/ci/ai_review.py` sends the PR diff to Claude (`claude-opus-4-8`),
+posts the structured findings as a PR comment, and fails the check only when a
+finding is rated **severe** (would break prod, leak data, or corrupt data).
+Minor findings are informational and never block a merge.