feat(ci): gate promotion on the image set that actually passed E2E
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m11s
PR Checks / Backend Smoke (pull_request) Successful in 9s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m19s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 36s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 6m3s
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m11s
PR Checks / Backend Smoke (pull_request) Successful in 9s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 1m19s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 36s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 6m3s
Staging health polling asked only whether something answered HTTP 200 at the base URL. It could not tell the new deployment from the old one, so journeys could pass against the previous release, and concurrent merges could move the staging tags underneath a run in flight. Each staging run now mints a build ID and stamps all three images with the commit and that ID, as labels and — for frontend and backend — as a build-time JSON file that environment overrides cannot rewrite. /release.json reports both identities uncached, and scripts/ci/release.py polls for the expected pair before and after the journeys. Only then are the captured build digests tagged verified-<sha>. Promotion resolves those verified tags to immutable digests, revalidates their labels, and refuses a mixed or incomplete set before any :prod tag moves. The whole staging workflow shares one concurrency group with cancellation disabled, so releases serialise. The scripts are stdlib-only and unit-tested against mocked registry and HTTP behaviour; PR checks now run the pipeline and CI suites too. The runbook records what this cannot prove locally, and that the first rollout needs a commit built by this workflow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7b41218e6e
commit
0c901cd0d1
15 files changed
+494
-61
No files matched your search
+44
-4
@@ -19,8 +19,9 @@ PR checks (.gitea/workflows/pr-checks.yml)
|
||||
▼
|
||||
Stage pipeline (.gitea/workflows/deploy.yml) — automatic
|
||||
1. build & push images → tags sha-<sha>, staging
|
||||
2. staging Portainer webhook → wait for staging health
|
||||
2. staging Portainer webhook → verify frontend/backend SHA + build ID
|
||||
3. Playwright E2E journeys against staging ← gate before human testing
|
||||
4. verify identity again; tag tested digests verified-<full-sha>
|
||||
▼
|
||||
Manual testing on staging (stx.schoolcompare.co.uk)
|
||||
│ Actions → "Promote to Production (manual)" ← approval #2
|
||||
@@ -28,14 +29,15 @@ Manual testing on staging (stx.schoolcompare.co.uk)
|
||||
Promote pipeline (.gitea/workflows/promote.yml) — manual dispatch
|
||||
1. resolve target sha (input, or latest main if empty)
|
||||
2. REFUSE unless that commit's "E2E Journeys against Staging" status is green
|
||||
3. retag sha-<sha> → :prod (same bytes — build once, promote the image)
|
||||
3. resolve verified-<full-sha> digests, validate labels, retag digests → :prod
|
||||
previous :prod saved as :prod-previous
|
||||
4. prod Portainer webhook → wait for prod health
|
||||
4. prod Portainer webhook → verify expected SHA + build ID
|
||||
```
|
||||
|
||||
Key principle: **build once, promote the exact image**. Production pins `:prod`,
|
||||
which only moves when a human runs the promote workflow — and the workflow
|
||||
only accepts commits that passed the staging E2E gate. Nothing tags `:latest`
|
||||
only accepts commits that passed the staging E2E gate and have a complete verified
|
||||
image set. Nothing tags `:latest`
|
||||
anymore.
|
||||
|
||||
## Branch & PR workflow
|
||||
@@ -256,3 +258,41 @@ how long any feature is exposed to this.
|
||||
If `UNLEASH_URL` is unset, every flag is `False` and no connection is
|
||||
attempted. That is the correct behaviour for local development and CI, and it
|
||||
means the test suites need no flag server.
|
||||
|
||||
## Release identity and the P1 reliability gate
|
||||
|
||||
Every staging run creates a random build ID before building its three images.
|
||||
Each image carries the commit and build ID as labels. Frontend/backend images
|
||||
also contain a build-time JSON file; environment overrides cannot rewrite it.
|
||||
`/release.json` returns both identities with `Cache-Control: no-store`. It fails
|
||||
with 503 when either identity cannot be read. FastAPI's internal endpoint is
|
||||
`/api/release`.
|
||||
|
||||
The entire staging workflow shares one concurrency group, with cancellation
|
||||
disabled. This needs Gitea 1.26 or newer, where workflow concurrency is supported
|
||||
([release notes](https://blog.gitea.com/release-of-1.26.0/)); the configured server
|
||||
reported 1.27.3 during this change. Do not run the workflow on an older server
|
||||
that ignores the concurrency key. Manual deployments outside this workflow must
|
||||
also avoid changing staging during journeys.
|
||||
|
||||
The gate checks both identities before and after Playwright. It then validates
|
||||
labels on the captured build output digests and tags them `verified-<full-sha>`.
|
||||
The manual promotion script resolves all three verified tags to immutable digests
|
||||
and confirms one matching commit/build ID before moving any `:prod` tag. It polls
|
||||
production for that same identity using a locally saved release manifest.
|
||||
A registry error can still interrupt the three tag writes; the Portainer webhook
|
||||
only runs after successful promotion, and rerunning promotion resolves the full
|
||||
verified set again. There is no cross-registry atomic tag transaction.
|
||||
|
||||
**First rollout:** old green commits without verified tags/build identities are
|
||||
not promotable through this gate. Build and test a commit containing the new
|
||||
workflow first. The release route must be reachable through the configured
|
||||
`STAGING_BASE_URL`/`PROD_BASE_URL`; it deliberately avoids the public staging
|
||||
`/api` proxy limitation. No new deployment secret is required.
|
||||
|
||||
`scripts/ci/release.py` implements identity polling and digest verification.
|
||||
Its mocked tests run in PR checks alongside backend and index-publication tests.
|
||||
The new Playwright journeys also check deployed identity and stale pagination.
|
||||
Local unit checks do not validate registry credentials, Portainer behaviour,
|
||||
proxy routing or a deployed image; those require the staging run. Production
|
||||
promotion remains a separate human action.
|
||||
Reference in new issue
Block a user