docs: plan for staged production promotion

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
2026-07-13 08:34:32 +01:00
parent 58e90fef61
commit dd0ff7d0c2
1 changed files with 275 additions and 0 deletions
@@ -0,0 +1,275 @@
+# Staged Production Promotion (Manual Gate) Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Merging a PR deploys to staging only; production deployment requires a second, explicit human approval after manual testing on staging.
+
+**Architecture:** Split the existing single `deploy.yml` pipeline in two. The push-to-main workflow keeps build → staging deploy → e2e gate and **stops there**. A new `promote.yml` runs only on `workflow_dispatch` (the "Run workflow" button in Gitea's Actions UI, supported on this server — Gitea 1.26.4): it verifies the chosen commit passed the staging e2e gate, retags its `:sha-*` images to `:prod` (keeping `:prod-previous` for rollback), and triggers the Portainer prod webhook. Promotion granularity is a main-branch commit: staging always runs the latest main, so you approve a *state of main*, not an individual PR.
+
+**Tech Stack:** Gitea Actions (1.26.4), Docker buildx imagetools, Portainer webhooks, Gitea commit-status API.
+
+## Global Constraints
+
+- **Never push to `main` directly** — this change itself goes through a PR (`chore/staged-prod-promotion` branch).
+- Existing image tagging scheme is unchanged: `type=sha` (e.g. `sha-6f925ab`) + `:staging`; promotion still retags `:sha-*` → `:prod` with `:prod-previous` kept as the rollback pointer.
+- The e2e journeys remain a **hard gate before human testing** (a red staging never reaches the promote button) and the promote workflow must refuse to promote a commit whose staging e2e did not succeed.
+- Secrets already exist and are reused: `REGISTRY_TOKEN` (also a Gitea API token), `PORTAINER_STAGING_WEBHOOK`, `PORTAINER_PROD_WEBHOOK`, `STAGING_BASE_URL`, `PROD_BASE_URL`.
+- Staging quirk (memory): external `/api` is broken at the staging proxy — manual API testing happens from the host, not through stx.schoolcompare.co.uk; note it in the runbook, don't try to fix it in this plan.
+
+## Considered approaches (context for the reviewer)
+
+1. **Manual `workflow_dispatch` promote workflow (chosen).** Native on Gitea 1.26; the second approval is clicking "Run workflow" (or one API call) after testing staging. Least machinery, auditable via the Actions run history.
+2. *Tag-driven promotion* (`push: tags: promote-*`): works on any Gitea version; approval = pushing a tag. Slightly more scriptable, less discoverable; kept as documented fallback only.
+3. *GitOps `production` branch + promotion PR:* approval literally reuses the PR-review UI, but adds a second long-lived branch to keep in sync — too much ceremony for a solo project. Rejected.
+
+---
+
+### Task 0: Branch
+
+- [ ] `git checkout main && git pull && git checkout -b chore/staged-prod-promotion`
+
+---
+
+### Task 1: Stop the push-to-main workflow after the e2e gate
+
+**Files:**
+- Modify: `.gitea/workflows/deploy.yml`
+
+**Interfaces:**
+- Produces: images tagged `:sha-<short>` + `:staging` (unchanged), a green `E2E Journeys against Staging` commit status that Task 2's promote workflow checks by name. **Do not rename the `e2e-staging` job's `name:` without updating Task 2's status check.**
+
+- [ ] **Step 1: Remove the auto-promotion**
+
+In `.gitea/workflows/deploy.yml`:
+1. Change line 1 to: `name: Stage (build -> staging -> E2E gate)`
+2. Delete the entire `promote-prod` job (lines 196–240 in the current file: from `  promote-prod:` to the end of the file).
+3. Leave `build-*`, `deploy-staging`, and `e2e-staging` untouched.
+
+- [ ] **Step 2: Sanity-check the YAML**
+
+Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/deploy.yml')); print('yaml ok')"`
+Expected: `yaml ok`
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add .gitea/workflows/deploy.yml
+git commit -m "ci: stop deploy pipeline at staging; production promotion becomes manual"
+```
+
+---
+
+### Task 2: Manual promote workflow
+
+**Files:**
+- Create: `.gitea/workflows/promote.yml`
+
+**Interfaces:**
+- Consumes: `:sha-<short>` images built by deploy.yml; the `E2E Journeys against Staging` commit status.
+- Produces: `:prod` and `:prod-previous` tags; prod stack update.
+
+- [ ] **Step 1: Write the workflow**
+
+```yaml
+name: Promote to Production (manual)
+
+on:
+  workflow_dispatch:
+    inputs:
+      sha:
+        description: >-
+          Commit SHA on main to promote (full or >=7 chars).
+          Leave empty to promote the latest main commit.
+        required: false
+        default: ""
+
+env:
+  REGISTRY: privaterepo.sitaru.org
+  BACKEND_IMAGE_NAME: ${{ gitea.repository }}-backend
+  FRONTEND_IMAGE_NAME: ${{ gitea.repository }}-frontend
+  PIPELINE_IMAGE_NAME: ${{ gitea.repository }}-pipeline
+
+jobs:
+  promote-prod:
+    name: Promote approved commit to Production
+    runs-on: ubuntu-latest
+    steps:
+      - name: Resolve target SHA
+        id: resolve
+        run: |
+          SHA_INPUT="${{ gitea.event.inputs.sha }}"
+          if [ -z "$SHA_INPUT" ]; then
+            SHA_INPUT="${{ gitea.sha }}"
+          fi
+          # Normalise to the full sha via the API so short inputs work
+          FULL_SHA=$(curl -fsS \
+            -H "Authorization: token ${{ secrets.REGISTRY_TOKEN }}" \
+            "https://${REGISTRY}/api/v1/repos/${{ gitea.repository }}/git/commits/${SHA_INPUT}" \
+            | python3 -c "import json,sys; print(json.load(sys.stdin)['sha'])")
+          SHORT_SHA="sha-$(echo "$FULL_SHA" | cut -c1-7)"
+          echo "full=$FULL_SHA" >> "$GITHUB_OUTPUT"
+          echo "short=$SHORT_SHA" >> "$GITHUB_OUTPUT"
+          echo "Promoting $FULL_SHA (images tagged $SHORT_SHA)"
+
+      - name: Verify the staging E2E gate passed for this commit
+        run: |
+          STATUS_JSON=$(curl -fsS \
+            -H "Authorization: token ${{ secrets.REGISTRY_TOKEN }}" \
+            "https://${REGISTRY}/api/v1/repos/${{ gitea.repository }}/commits/${{ steps.resolve.outputs.full }}/status")
+          echo "$STATUS_JSON" | python3 -c "
+          import json, sys
+          d = json.load(sys.stdin)
+          ok = [s for s in d.get('statuses', [])
+                if 'E2E Journeys against Staging' in s.get('context', '')
+                and s.get('status') == 'success']
+          if not ok:
+              print('REFUSED: no successful \"E2E Journeys against Staging\" status on this commit.')
+              print('Contexts found:', [s.get('context') for s in d.get('statuses', [])])
+              sys.exit(1)
+          print('E2E gate verified green for this commit.')
+          "
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to Gitea Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ gitea.actor }}
+          password: ${{ secrets.REGISTRY_TOKEN }}
+
+      - name: Retag approved images as prod (keeping rollback pointer)
+        run: |
+          SHORT_SHA="${{ steps.resolve.outputs.short }}"
+          for IMAGE in \
+            "${REGISTRY}/${BACKEND_IMAGE_NAME}" \
+            "${REGISTRY}/${FRONTEND_IMAGE_NAME}" \
+            "${REGISTRY}/${PIPELINE_IMAGE_NAME}"; do
+            docker buildx imagetools create -t "${IMAGE}:prod-previous" "${IMAGE}:prod" || true
+            docker buildx imagetools create -t "${IMAGE}:prod" "${IMAGE}:${SHORT_SHA}"
+            echo "Promoted ${IMAGE}:${SHORT_SHA} -> :prod"
+          done
+
+      - name: Trigger production stack update
+        run: curl -fsSk -X POST "${{ secrets.PORTAINER_PROD_WEBHOOK }}"
+
+      - name: Wait for production to become healthy
+        run: |
+          echo "Polling ${PROD_BASE_URL} for up to 5 minutes..."
+          for i in $(seq 1 60); do
+            if curl -fsS -o /dev/null --max-time 10 "${PROD_BASE_URL}/"; then
+              echo "Production is up (attempt $i)"
+              exit 0
+            fi
+            sleep 5
+          done
+          echo "Production did not become healthy in time" >&2
+          exit 1
+        env:
+          PROD_BASE_URL: ${{ secrets.PROD_BASE_URL }}
+```
+
+Implementation notes for the engineer:
+- Gitea Actions uses the GitHub-compatible `$GITHUB_OUTPUT` file for step outputs; if the runner image doesn't populate it, fall back to `$GITEA_OUTPUT` (check the runner's docs/output at first run).
+- The retag step is copied verbatim from the old `promote-prod` job except the SHA comes from the resolved input instead of `gitea.sha` — behaviour for the default (empty input on latest main) is identical to before.
+- If `docker buildx imagetools create` fails with "not found" for `${IMAGE}:${SHORT_SHA}`, the chosen commit predates the registry's retention or never built — the error message is the desired behaviour (refuse loudly).
+
+- [ ] **Step 2: YAML sanity check**
+
+Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/promote.yml')); print('yaml ok')"`
+Expected: `yaml ok`
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add .gitea/workflows/promote.yml
+git commit -m "ci: manual production promotion workflow with e2e-gate verification"
+```
+
+---
+
+### Task 3: Documentation — deploy model + runbook
+
+**Files:**
+- Modify: `docs/DEPLOY.md`
+- Modify: `claude.md` (the SDLC section)
+
+- [ ] **Step 1: Rewrite the flow description in `docs/DEPLOY.md`**
+
+Replace the staging→prod description with the new model (adapt to the file's existing structure; the substance to convey):
+
+```markdown
+## Deploy model
+
+1. **PR → main (first approval).** Branch-protected merge; PR checks
+   (typecheck, tests, builds, AI review) must pass.
+2. **Merge → staging (automatic).** Images are built once and tagged
+   `sha-<short>` + `staging`; the staging stack updates; Playwright
+   journeys in `e2e/` run against staging. A red e2e run means staging
+   is not fit for testing — fix forward before considering promotion.
+3. **Manual testing on staging.** stx.schoolcompare.co.uk. Note:
+   external `/api` is broken at the staging proxy — exercise API
+   endpoints from the host.
+4. **Promote → production (second approval).** Actions → "Promote to
+   Production (manual)" → Run workflow. Leave the SHA empty to promote
+   the latest main, or paste a specific commit SHA. The workflow
+   refuses commits whose staging e2e gate is not green, retags the
+   images `:prod` (keeping `:prod-previous`), and updates the prod
+   stack.
+
+### Promotion granularity
+
+Staging always runs the latest `main`. Promoting approves a *state of
+main*, not a single PR — if two PRs merged since the last promotion,
+they ship together. Test staging accordingly.
+
+### Rollback
+
+Re-run "Promote to Production (manual)" with the SHA of the last good
+commit (or retag manually: `docker buildx imagetools create -t
+<image>:prod <image>:prod-previous` for each of the three images, then
+POST the prod Portainer webhook).
+```
+
+- [ ] **Step 2: Update the SDLC bullet in `claude.md`**
+
+Replace the sentence "Merging to `main` deploys automatically: … retagged `:prod` and rolled out to production." with:
+
+```markdown
+- Merging to `main` deploys automatically **to staging only**: images
+  are built once, deployed to the staging Portainer stack, and verified
+  by the Playwright journeys in `e2e/`. Production is a second, manual
+  approval: the "Promote to Production (manual)" workflow in Gitea
+  Actions, run after testing the feature on staging. It refuses commits
+  whose staging e2e gate isn't green.
+```
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add docs/DEPLOY.md claude.md
+git commit -m "docs: two-stage deploy model (staging auto, production manual)"
+```
+
+---
+
+### Task 4: PR + live validation
+
+- [ ] **Step 1: Push and open the PR** (Gitea API with credential-helper basic auth, as usual). PR body: the new model in three lines, the rollback recipe, and a warning that between merging this PR and its first promotion run, production receives no deployments (expected).
+
+- [ ] **Step 2: Validate after merge (human-in-the-loop):**
+1. Merge this PR → confirm the `Stage (build -> staging -> E2E gate)` run goes green and **no** production deployment happens (prod image digest unchanged: `docker buildx imagetools inspect <image>:prod` before/after, or check the Portainer prod stack's last-update time).
+2. Test something trivial on staging.
+3. Run "Promote to Production (manual)" with the SHA empty → confirm e2e verification passes, retag happens, prod becomes healthy.
+4. Negative test: run the promote workflow with a garbage SHA (e.g. `deadbeef1`) → confirm it fails at resolve/verify without touching `:prod`.
+
+- [ ] **Step 3: Update the ledger/memory** with the new deploy model so future sessions stop assuming auto-promotion.
+
+---
+
+## Out of scope / future options
+
+- Notifications when staging is ready for testing (Gitea can email on workflow completion; a webhook to ntfy/Matrix could be added later).
+- Restricting who can run the promote workflow: Gitea 1.26 runs `workflow_dispatch` with the permissions of the dispatching user; for a solo repo this is already effectively restricted.
+- The tag-driven fallback (`on: push: tags: promote-*`) if `workflow_dispatch` ever proves unreliable on the runner.