diff --git a/docs/superpowers/plans/2026-07-13-staged-prod-promotion.md b/docs/superpowers/plans/2026-07-13-staged-prod-promotion.md new file mode 100644 index 0000000..4179097 --- /dev/null +++ b/docs/superpowers/plans/2026-07-13-staged-prod-promotion.md @@ -0,0 +1,275 @@ +# Staged Production Promotion (Manual Gate) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Merging a PR deploys to staging only; production deployment requires a second, explicit human approval after manual testing on staging. + +**Architecture:** Split the existing single `deploy.yml` pipeline in two. The push-to-main workflow keeps build → staging deploy → e2e gate and **stops there**. A new `promote.yml` runs only on `workflow_dispatch` (the "Run workflow" button in Gitea's Actions UI, supported on this server — Gitea 1.26.4): it verifies the chosen commit passed the staging e2e gate, retags its `:sha-*` images to `:prod` (keeping `:prod-previous` for rollback), and triggers the Portainer prod webhook. Promotion granularity is a main-branch commit: staging always runs the latest main, so you approve a *state of main*, not an individual PR. + +**Tech Stack:** Gitea Actions (1.26.4), Docker buildx imagetools, Portainer webhooks, Gitea commit-status API. + +## Global Constraints + +- **Never push to `main` directly** — this change itself goes through a PR (`chore/staged-prod-promotion` branch). +- Existing image tagging scheme is unchanged: `type=sha` (e.g. `sha-6f925ab`) + `:staging`; promotion still retags `:sha-*` → `:prod` with `:prod-previous` kept as the rollback pointer. +- The e2e journeys remain a **hard gate before human testing** (a red staging never reaches the promote button) and the promote workflow must refuse to promote a commit whose staging e2e did not succeed. +- Secrets already exist and are reused: `REGISTRY_TOKEN` (also a Gitea API token), `PORTAINER_STAGING_WEBHOOK`, `PORTAINER_PROD_WEBHOOK`, `STAGING_BASE_URL`, `PROD_BASE_URL`. +- Staging quirk (memory): external `/api` is broken at the staging proxy — manual API testing happens from the host, not through stx.schoolcompare.co.uk; note it in the runbook, don't try to fix it in this plan. + +## Considered approaches (context for the reviewer) + +1. **Manual `workflow_dispatch` promote workflow (chosen).** Native on Gitea 1.26; the second approval is clicking "Run workflow" (or one API call) after testing staging. Least machinery, auditable via the Actions run history. +2. *Tag-driven promotion* (`push: tags: promote-*`): works on any Gitea version; approval = pushing a tag. Slightly more scriptable, less discoverable; kept as documented fallback only. +3. *GitOps `production` branch + promotion PR:* approval literally reuses the PR-review UI, but adds a second long-lived branch to keep in sync — too much ceremony for a solo project. Rejected. + +--- + +### Task 0: Branch + +- [ ] `git checkout main && git pull && git checkout -b chore/staged-prod-promotion` + +--- + +### Task 1: Stop the push-to-main workflow after the e2e gate + +**Files:** +- Modify: `.gitea/workflows/deploy.yml` + +**Interfaces:** +- Produces: images tagged `:sha-` + `:staging` (unchanged), a green `E2E Journeys against Staging` commit status that Task 2's promote workflow checks by name. **Do not rename the `e2e-staging` job's `name:` without updating Task 2's status check.** + +- [ ] **Step 1: Remove the auto-promotion** + +In `.gitea/workflows/deploy.yml`: +1. Change line 1 to: `name: Stage (build -> staging -> E2E gate)` +2. Delete the entire `promote-prod` job (lines 196–240 in the current file: from ` promote-prod:` to the end of the file). +3. Leave `build-*`, `deploy-staging`, and `e2e-staging` untouched. + +- [ ] **Step 2: Sanity-check the YAML** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/deploy.yml')); print('yaml ok')"` +Expected: `yaml ok` + +- [ ] **Step 3: Commit** + +```bash +git add .gitea/workflows/deploy.yml +git commit -m "ci: stop deploy pipeline at staging; production promotion becomes manual" +``` + +--- + +### Task 2: Manual promote workflow + +**Files:** +- Create: `.gitea/workflows/promote.yml` + +**Interfaces:** +- Consumes: `:sha-` images built by deploy.yml; the `E2E Journeys against Staging` commit status. +- Produces: `:prod` and `:prod-previous` tags; prod stack update. + +- [ ] **Step 1: Write the workflow** + +```yaml +name: Promote to Production (manual) + +on: + workflow_dispatch: + inputs: + sha: + description: >- + Commit SHA on main to promote (full or >=7 chars). + Leave empty to promote the latest main commit. + required: false + default: "" + +env: + REGISTRY: privaterepo.sitaru.org + BACKEND_IMAGE_NAME: ${{ gitea.repository }}-backend + FRONTEND_IMAGE_NAME: ${{ gitea.repository }}-frontend + PIPELINE_IMAGE_NAME: ${{ gitea.repository }}-pipeline + +jobs: + promote-prod: + name: Promote approved commit to Production + runs-on: ubuntu-latest + steps: + - name: Resolve target SHA + id: resolve + run: | + SHA_INPUT="${{ gitea.event.inputs.sha }}" + if [ -z "$SHA_INPUT" ]; then + SHA_INPUT="${{ gitea.sha }}" + fi + # Normalise to the full sha via the API so short inputs work + FULL_SHA=$(curl -fsS \ + -H "Authorization: token ${{ secrets.REGISTRY_TOKEN }}" \ + "https://${REGISTRY}/api/v1/repos/${{ gitea.repository }}/git/commits/${SHA_INPUT}" \ + | python3 -c "import json,sys; print(json.load(sys.stdin)['sha'])") + SHORT_SHA="sha-$(echo "$FULL_SHA" | cut -c1-7)" + echo "full=$FULL_SHA" >> "$GITHUB_OUTPUT" + echo "short=$SHORT_SHA" >> "$GITHUB_OUTPUT" + echo "Promoting $FULL_SHA (images tagged $SHORT_SHA)" + + - name: Verify the staging E2E gate passed for this commit + run: | + STATUS_JSON=$(curl -fsS \ + -H "Authorization: token ${{ secrets.REGISTRY_TOKEN }}" \ + "https://${REGISTRY}/api/v1/repos/${{ gitea.repository }}/commits/${{ steps.resolve.outputs.full }}/status") + echo "$STATUS_JSON" | python3 -c " + import json, sys + d = json.load(sys.stdin) + ok = [s for s in d.get('statuses', []) + if 'E2E Journeys against Staging' in s.get('context', '') + and s.get('status') == 'success'] + if not ok: + print('REFUSED: no successful \"E2E Journeys against Staging\" status on this commit.') + print('Contexts found:', [s.get('context') for s in d.get('statuses', [])]) + sys.exit(1) + print('E2E gate verified green for this commit.') + " + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to Gitea Container Registry + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ gitea.actor }} + password: ${{ secrets.REGISTRY_TOKEN }} + + - name: Retag approved images as prod (keeping rollback pointer) + run: | + SHORT_SHA="${{ steps.resolve.outputs.short }}" + for IMAGE in \ + "${REGISTRY}/${BACKEND_IMAGE_NAME}" \ + "${REGISTRY}/${FRONTEND_IMAGE_NAME}" \ + "${REGISTRY}/${PIPELINE_IMAGE_NAME}"; do + docker buildx imagetools create -t "${IMAGE}:prod-previous" "${IMAGE}:prod" || true + docker buildx imagetools create -t "${IMAGE}:prod" "${IMAGE}:${SHORT_SHA}" + echo "Promoted ${IMAGE}:${SHORT_SHA} -> :prod" + done + + - name: Trigger production stack update + run: curl -fsSk -X POST "${{ secrets.PORTAINER_PROD_WEBHOOK }}" + + - name: Wait for production to become healthy + run: | + echo "Polling ${PROD_BASE_URL} for up to 5 minutes..." + for i in $(seq 1 60); do + if curl -fsS -o /dev/null --max-time 10 "${PROD_BASE_URL}/"; then + echo "Production is up (attempt $i)" + exit 0 + fi + sleep 5 + done + echo "Production did not become healthy in time" >&2 + exit 1 + env: + PROD_BASE_URL: ${{ secrets.PROD_BASE_URL }} +``` + +Implementation notes for the engineer: +- Gitea Actions uses the GitHub-compatible `$GITHUB_OUTPUT` file for step outputs; if the runner image doesn't populate it, fall back to `$GITEA_OUTPUT` (check the runner's docs/output at first run). +- The retag step is copied verbatim from the old `promote-prod` job except the SHA comes from the resolved input instead of `gitea.sha` — behaviour for the default (empty input on latest main) is identical to before. +- If `docker buildx imagetools create` fails with "not found" for `${IMAGE}:${SHORT_SHA}`, the chosen commit predates the registry's retention or never built — the error message is the desired behaviour (refuse loudly). + +- [ ] **Step 2: YAML sanity check** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/promote.yml')); print('yaml ok')"` +Expected: `yaml ok` + +- [ ] **Step 3: Commit** + +```bash +git add .gitea/workflows/promote.yml +git commit -m "ci: manual production promotion workflow with e2e-gate verification" +``` + +--- + +### Task 3: Documentation — deploy model + runbook + +**Files:** +- Modify: `docs/DEPLOY.md` +- Modify: `claude.md` (the SDLC section) + +- [ ] **Step 1: Rewrite the flow description in `docs/DEPLOY.md`** + +Replace the staging→prod description with the new model (adapt to the file's existing structure; the substance to convey): + +```markdown +## Deploy model + +1. **PR → main (first approval).** Branch-protected merge; PR checks + (typecheck, tests, builds, AI review) must pass. +2. **Merge → staging (automatic).** Images are built once and tagged + `sha-` + `staging`; the staging stack updates; Playwright + journeys in `e2e/` run against staging. A red e2e run means staging + is not fit for testing — fix forward before considering promotion. +3. **Manual testing on staging.** stx.schoolcompare.co.uk. Note: + external `/api` is broken at the staging proxy — exercise API + endpoints from the host. +4. **Promote → production (second approval).** Actions → "Promote to + Production (manual)" → Run workflow. Leave the SHA empty to promote + the latest main, or paste a specific commit SHA. The workflow + refuses commits whose staging e2e gate is not green, retags the + images `:prod` (keeping `:prod-previous`), and updates the prod + stack. + +### Promotion granularity + +Staging always runs the latest `main`. Promoting approves a *state of +main*, not a single PR — if two PRs merged since the last promotion, +they ship together. Test staging accordingly. + +### Rollback + +Re-run "Promote to Production (manual)" with the SHA of the last good +commit (or retag manually: `docker buildx imagetools create -t +:prod :prod-previous` for each of the three images, then +POST the prod Portainer webhook). +``` + +- [ ] **Step 2: Update the SDLC bullet in `claude.md`** + +Replace the sentence "Merging to `main` deploys automatically: … retagged `:prod` and rolled out to production." with: + +```markdown +- Merging to `main` deploys automatically **to staging only**: images + are built once, deployed to the staging Portainer stack, and verified + by the Playwright journeys in `e2e/`. Production is a second, manual + approval: the "Promote to Production (manual)" workflow in Gitea + Actions, run after testing the feature on staging. It refuses commits + whose staging e2e gate isn't green. +``` + +- [ ] **Step 3: Commit** + +```bash +git add docs/DEPLOY.md claude.md +git commit -m "docs: two-stage deploy model (staging auto, production manual)" +``` + +--- + +### Task 4: PR + live validation + +- [ ] **Step 1: Push and open the PR** (Gitea API with credential-helper basic auth, as usual). PR body: the new model in three lines, the rollback recipe, and a warning that between merging this PR and its first promotion run, production receives no deployments (expected). + +- [ ] **Step 2: Validate after merge (human-in-the-loop):** +1. Merge this PR → confirm the `Stage (build -> staging -> E2E gate)` run goes green and **no** production deployment happens (prod image digest unchanged: `docker buildx imagetools inspect :prod` before/after, or check the Portainer prod stack's last-update time). +2. Test something trivial on staging. +3. Run "Promote to Production (manual)" with the SHA empty → confirm e2e verification passes, retag happens, prod becomes healthy. +4. Negative test: run the promote workflow with a garbage SHA (e.g. `deadbeef1`) → confirm it fails at resolve/verify without touching `:prod`. + +- [ ] **Step 3: Update the ledger/memory** with the new deploy model so future sessions stop assuming auto-promotion. + +--- + +## Out of scope / future options + +- Notifications when staging is ready for testing (Gitea can email on workflow completion; a webhook to ntfy/Matrix could be added later). +- Restricting who can run the promote workflow: Gitea 1.26 runs `workflow_dispatch` with the permissions of the dispatching user; for a solo repo this is already effectively restricted. +- The tag-driven fallback (`on: push: tags: promote-*`) if `workflow_dispatch` ever proves unreliable on the runner.