# Staged Production Promotion (Manual Gate) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Merging a PR deploys to staging only; production deployment requires a second, explicit human approval after manual testing on staging.

**Architecture:** Split the existing single `deploy.yml` pipeline in two. The push-to-main workflow keeps build → staging deploy → e2e gate and **stops there**. A new `promote.yml` runs only on `workflow_dispatch` (the "Run workflow" button in Gitea's Actions UI, supported on this server — Gitea 1.26.4): it verifies the chosen commit passed the staging e2e gate, retags its `:sha-*` images to `:prod` (keeping `:prod-previous` for rollback), and triggers the Portainer prod webhook. Promotion granularity is a main-branch commit: staging always runs the latest main, so you approve a *state of main*, not an individual PR.

**Tech Stack:** Gitea Actions (1.26.4), Docker buildx imagetools, Portainer webhooks, Gitea commit-status API.

## Global Constraints

- **Never push to `main` directly** — this change itself goes through a PR (`chore/staged-prod-promotion` branch).
- Existing image tagging scheme is unchanged: `type=sha` (e.g. `sha-6f925ab`) + `:staging`; promotion still retags `:sha-*` → `:prod` with `:prod-previous` kept as the rollback pointer.
- The e2e journeys remain a **hard gate before human testing** (a red staging never reaches the promote button) and the promote workflow must refuse to promote a commit whose staging e2e did not succeed.
- Secrets already exist and are reused: `REGISTRY_TOKEN` (also a Gitea API token), `PORTAINER_STAGING_WEBHOOK`, `PORTAINER_PROD_WEBHOOK`, `STAGING_BASE_URL`, `PROD_BASE_URL`.
- Staging quirk (memory): external `/api` is broken at the staging proxy — manual API testing happens from the host, not through stx.schoolcompare.co.uk; note it in the runbook, don't try to fix it in this plan.

## Considered approaches (context for the reviewer)

1. **Manual `workflow_dispatch` promote workflow (chosen).** Native on Gitea 1.26; the second approval is clicking "Run workflow" (or one API call) after testing staging. Least machinery, auditable via the Actions run history.
2. *Tag-driven promotion* (`push: tags: promote-*`): works on any Gitea version; approval = pushing a tag. Slightly more scriptable, less discoverable; kept as documented fallback only.
3. *GitOps `production` branch + promotion PR:* approval literally reuses the PR-review UI, but adds a second long-lived branch to keep in sync — too much ceremony for a solo project. Rejected.

---

### Task 0: Branch

- [ ] `git checkout main && git pull && git checkout -b chore/staged-prod-promotion`

---

### Task 1: Stop the push-to-main workflow after the e2e gate

**Files:**
- Modify: `.gitea/workflows/deploy.yml`

**Interfaces:**
- Produces: images tagged `:sha-<short>` + `:staging` (unchanged), a green `E2E Journeys against Staging` commit status that Task 2's promote workflow checks by name. **Do not rename the `e2e-staging` job's `name:` without updating Task 2's status check.**

- [ ] **Step 1: Remove the auto-promotion**

In `.gitea/workflows/deploy.yml`:
1. Change line 1 to: `name: Stage (build -> staging -> E2E gate)`
2. Delete the entire `promote-prod` job (lines 196–240 in the current file: from `  promote-prod:` to the end of the file).
3. Leave `build-*`, `deploy-staging`, and `e2e-staging` untouched.

- [ ] **Step 2: Sanity-check the YAML**

Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/deploy.yml')); print('yaml ok')"`
Expected: `yaml ok`

- [ ] **Step 3: Commit**

```bash
git add .gitea/workflows/deploy.yml
git commit -m "ci: stop deploy pipeline at staging; production promotion becomes manual"
```

---

### Task 2: Manual promote workflow

**Files:**
- Create: `.gitea/workflows/promote.yml`

**Interfaces:**
- Consumes: `:sha-<short>` images built by deploy.yml; the `E2E Journeys against Staging` commit status.
- Produces: `:prod` and `:prod-previous` tags; prod stack update.

- [ ] **Step 1: Write the workflow**

```yaml
name: Promote to Production (manual)

on:
  workflow_dispatch:
    inputs:
      sha:
        description: >-
          Commit SHA on main to promote (full or >=7 chars).
          Leave empty to promote the latest main commit.
        required: false
        default: ""

env:
  REGISTRY: privaterepo.sitaru.org
  BACKEND_IMAGE_NAME: ${{ gitea.repository }}-backend
  FRONTEND_IMAGE_NAME: ${{ gitea.repository }}-frontend
  PIPELINE_IMAGE_NAME: ${{ gitea.repository }}-pipeline

jobs:
  promote-prod:
    name: Promote approved commit to Production
    runs-on: ubuntu-latest
    steps:
      - name: Resolve target SHA
        id: resolve
        run: |
          SHA_INPUT="${{ gitea.event.inputs.sha }}"
          if [ -z "$SHA_INPUT" ]; then
            SHA_INPUT="${{ gitea.sha }}"
          fi
          # Normalise to the full sha via the API so short inputs work
          FULL_SHA=$(curl -fsS \
            -H "Authorization: token ${{ secrets.REGISTRY_TOKEN }}" \
            "https://${REGISTRY}/api/v1/repos/${{ gitea.repository }}/git/commits/${SHA_INPUT}" \
            | python3 -c "import json,sys; print(json.load(sys.stdin)['sha'])")
          SHORT_SHA="sha-$(echo "$FULL_SHA" | cut -c1-7)"
          echo "full=$FULL_SHA" >> "$GITHUB_OUTPUT"
          echo "short=$SHORT_SHA" >> "$GITHUB_OUTPUT"
          echo "Promoting $FULL_SHA (images tagged $SHORT_SHA)"

      - name: Verify the staging E2E gate passed for this commit
        run: |
          STATUS_JSON=$(curl -fsS \
            -H "Authorization: token ${{ secrets.REGISTRY_TOKEN }}" \
            "https://${REGISTRY}/api/v1/repos/${{ gitea.repository }}/commits/${{ steps.resolve.outputs.full }}/status")
          echo "$STATUS_JSON" | python3 -c "
          import json, sys
          d = json.load(sys.stdin)
          ok = [s for s in d.get('statuses', [])
                if 'E2E Journeys against Staging' in s.get('context', '')
                and s.get('status') == 'success']
          if not ok:
              print('REFUSED: no successful \"E2E Journeys against Staging\" status on this commit.')
              print('Contexts found:', [s.get('context') for s in d.get('statuses', [])])
              sys.exit(1)
          print('E2E gate verified green for this commit.')
          "

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to Gitea Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ gitea.actor }}
          password: ${{ secrets.REGISTRY_TOKEN }}

      - name: Retag approved images as prod (keeping rollback pointer)
        run: |
          SHORT_SHA="${{ steps.resolve.outputs.short }}"
          for IMAGE in \
            "${REGISTRY}/${BACKEND_IMAGE_NAME}" \
            "${REGISTRY}/${FRONTEND_IMAGE_NAME}" \
            "${REGISTRY}/${PIPELINE_IMAGE_NAME}"; do
            docker buildx imagetools create -t "${IMAGE}:prod-previous" "${IMAGE}:prod" || true
            docker buildx imagetools create -t "${IMAGE}:prod" "${IMAGE}:${SHORT_SHA}"
            echo "Promoted ${IMAGE}:${SHORT_SHA} -> :prod"
          done

      - name: Trigger production stack update
        run: curl -fsSk -X POST "${{ secrets.PORTAINER_PROD_WEBHOOK }}"

      - name: Wait for production to become healthy
        run: |
          echo "Polling ${PROD_BASE_URL} for up to 5 minutes..."
          for i in $(seq 1 60); do
            if curl -fsS -o /dev/null --max-time 10 "${PROD_BASE_URL}/"; then
              echo "Production is up (attempt $i)"
              exit 0
            fi
            sleep 5
          done
          echo "Production did not become healthy in time" >&2
          exit 1
        env:
          PROD_BASE_URL: ${{ secrets.PROD_BASE_URL }}
```

Implementation notes for the engineer:
- Gitea Actions uses the GitHub-compatible `$GITHUB_OUTPUT` file for step outputs; if the runner image doesn't populate it, fall back to `$GITEA_OUTPUT` (check the runner's docs/output at first run).
- The retag step is copied verbatim from the old `promote-prod` job except the SHA comes from the resolved input instead of `gitea.sha` — behaviour for the default (empty input on latest main) is identical to before.
- If `docker buildx imagetools create` fails with "not found" for `${IMAGE}:${SHORT_SHA}`, the chosen commit predates the registry's retention or never built — the error message is the desired behaviour (refuse loudly).

- [ ] **Step 2: YAML sanity check**

Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/promote.yml')); print('yaml ok')"`
Expected: `yaml ok`

- [ ] **Step 3: Commit**

```bash
git add .gitea/workflows/promote.yml
git commit -m "ci: manual production promotion workflow with e2e-gate verification"
```

---

### Task 3: Documentation — deploy model + runbook

**Files:**
- Modify: `docs/DEPLOY.md`
- Modify: `claude.md` (the SDLC section)

- [ ] **Step 1: Rewrite the flow description in `docs/DEPLOY.md`**

Replace the staging→prod description with the new model (adapt to the file's existing structure; the substance to convey):

```markdown
## Deploy model

1. **PR → main (first approval).** Branch-protected merge; PR checks
   (typecheck, tests, builds, AI review) must pass.
2. **Merge → staging (automatic).** Images are built once and tagged
   `sha-<short>` + `staging`; the staging stack updates; Playwright
   journeys in `e2e/` run against staging. A red e2e run means staging
   is not fit for testing — fix forward before considering promotion.
3. **Manual testing on staging.** stx.schoolcompare.co.uk. Note:
   external `/api` is broken at the staging proxy — exercise API
   endpoints from the host.
4. **Promote → production (second approval).** Actions → "Promote to
   Production (manual)" → Run workflow. Leave the SHA empty to promote
   the latest main, or paste a specific commit SHA. The workflow
   refuses commits whose staging e2e gate is not green, retags the
   images `:prod` (keeping `:prod-previous`), and updates the prod
   stack.

### Promotion granularity

Staging always runs the latest `main`. Promoting approves a *state of
main*, not a single PR — if two PRs merged since the last promotion,
they ship together. Test staging accordingly.

### Rollback

Re-run "Promote to Production (manual)" with the SHA of the last good
commit (or retag manually: `docker buildx imagetools create -t
<image>:prod <image>:prod-previous` for each of the three images, then
POST the prod Portainer webhook).
```

- [ ] **Step 2: Update the SDLC bullet in `claude.md`**

Replace the sentence "Merging to `main` deploys automatically: … retagged `:prod` and rolled out to production." with:

```markdown
- Merging to `main` deploys automatically **to staging only**: images
  are built once, deployed to the staging Portainer stack, and verified
  by the Playwright journeys in `e2e/`. Production is a second, manual
  approval: the "Promote to Production (manual)" workflow in Gitea
  Actions, run after testing the feature on staging. It refuses commits
  whose staging e2e gate isn't green.
```

- [ ] **Step 3: Commit**

```bash
git add docs/DEPLOY.md claude.md
git commit -m "docs: two-stage deploy model (staging auto, production manual)"
```

---

### Task 4: PR + live validation

- [ ] **Step 1: Push and open the PR** (Gitea API with credential-helper basic auth, as usual). PR body: the new model in three lines, the rollback recipe, and a warning that between merging this PR and its first promotion run, production receives no deployments (expected).

- [ ] **Step 2: Validate after merge (human-in-the-loop):**
1. Merge this PR → confirm the `Stage (build -> staging -> E2E gate)` run goes green and **no** production deployment happens (prod image digest unchanged: `docker buildx imagetools inspect <image>:prod` before/after, or check the Portainer prod stack's last-update time).
2. Test something trivial on staging.
3. Run "Promote to Production (manual)" with the SHA empty → confirm e2e verification passes, retag happens, prod becomes healthy.
4. Negative test: run the promote workflow with a garbage SHA (e.g. `deadbeef1`) → confirm it fails at resolve/verify without touching `:prod`.

- [ ] **Step 3: Update the ledger/memory** with the new deploy model so future sessions stop assuming auto-promotion.

---

## Out of scope / future options

- Notifications when staging is ready for testing (Gitea can email on workflow completion; a webhook to ntfy/Matrix could be added later).
- Restricting who can run the promote workflow: Gitea 1.26 runs `workflow_dispatch` with the permissions of the dispatching user; for a solo repo this is already effectively restricted.
- The tag-driven fallback (`on: push: tags: promote-*`) if `workflow_dispatch` ever proves unreliable on the runner.
