Two small pipeline changes. The first is a straggler from #137.
build(pipeline) — install the destinations tap in the image
pipeline/Dockerfile installs each custom Singer tap by name, and tap-uk-ees-destinations was not in the list. I pushed this to #137 a few
minutes after it was merged, so it never landed — main currently builds a
pipeline image that does not explicitly install the new tap.
meltano install would most likely resolve it from its pip_url (that is how tap-uk-school-distance works today), so this may not be strictly load-bearing.
But a brand-new plugin failing to appear is not something worth debugging from a
deploy log when one line prevents it.
fix(airflow) — a login that survives a container restart
Airflow's simple auth manager generates a random password on first start and
writes it to $AIRFLOW_HOME/simple_auth_manager_passwords.json.generated, so
every restart of the api-server invalidated the last one and the password had to
be dug back out of the container logs.
Airflow generates nothing when that file already exists, so the stack now writes
it before exec'ing the api-server:
Applied to the prod stack, the staging stack and local docker-compose.yml.
Why python rather than echo/printf:json.dumps escapes a password
containing quotes, backslashes or non-ASCII correctly. Verified with p@ss "wo\rd' £5, which round-trips intact — a shell heredoc would have
mangled it.
Why it fails closed: an unset AIRFLOW_ADMIN_PASSWORD raises KeyError and
the container exits, and the compose :? default gives the same refusal a
readable reason. Falling back to a generated password would silently undo the
point of the change.
Neither Docker gotcha from the Airflow docs applies here: this image has no USER directive so it runs as root, and the file is rewritten from the
environment on every start rather than persisted on a volume — so there is
nothing to own wrongly and nothing to lose on restart.
⚠️ Before deploying
AIRFLOW_ADMIN_PASSWORDmust be set in the Portainer stack environment for
both staging and prod before this deploys, or the api-server will not start. AIRFLOW_ADMIN_USER still defaults to admin.
Two small pipeline changes. The first is a straggler from #137.
## `build(pipeline)` — install the destinations tap in the image
`pipeline/Dockerfile` installs each custom Singer tap by name, and
`tap-uk-ees-destinations` was not in the list. I pushed this to #137 a few
minutes after it was merged, so it never landed — `main` currently builds a
pipeline image that does not explicitly install the new tap.
`meltano install` would most likely resolve it from its `pip_url` (that is how
`tap-uk-school-distance` works today), so this may not be strictly load-bearing.
But a brand-new plugin failing to appear is not something worth debugging from a
deploy log when one line prevents it.
## `fix(airflow)` — a login that survives a container restart
Airflow's simple auth manager generates a random password on first start and
writes it to `$AIRFLOW_HOME/simple_auth_manager_passwords.json.generated`, so
every restart of the api-server invalidated the last one and the password had to
be dug back out of the container logs.
Airflow generates nothing when that file already exists, so the stack now writes
it before exec'ing the api-server:
```yaml
command:
- bash
- -c
- |
set -euo pipefail
mkdir -p /opt/airflow
python -c "import json, os, pathlib; pathlib.Path(...).write_text(json.dumps({...}))"
exec airflow api-server --port 8080
```
Applied to the prod stack, the staging stack and local `docker-compose.yml`.
**Why python rather than `echo`/`printf`:** `json.dumps` escapes a password
containing quotes, backslashes or non-ASCII correctly. Verified with
`p@ss "wo\rd' £5`, which round-trips intact — a shell heredoc would have
mangled it.
**Why it fails closed:** an unset `AIRFLOW_ADMIN_PASSWORD` raises `KeyError` and
the container exits, and the compose `:?` default gives the same refusal a
readable reason. Falling back to a generated password would silently undo the
point of the change.
Neither Docker gotcha from the Airflow docs applies here: this image has no
`USER` directive so it runs as root, and the file is rewritten from the
environment on every start rather than persisted on a volume — so there is
nothing to own wrongly and nothing to lose on restart.
## ⚠️ Before deploying
`AIRFLOW_ADMIN_PASSWORD` **must be set in the Portainer stack environment for
both staging and prod** before this deploys, or the api-server will not start.
`AIRFLOW_ADMIN_USER` still defaults to `admin`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob
meltano install would resolve it from pip_url, but five of the six custom
taps are also installed explicitly and a new plugin failing to appear is
not something you want to debug from a deploy log.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob
The simple auth manager generates a random password on first start and
writes it to a file, so every restart of the api-server invalidated the
last one and the password had to be dug out of the container logs again.
The stack now writes that file itself from AIRFLOW_ADMIN_PASSWORD before
exec'ing the api-server. Airflow generates nothing when the file already
exists, so the login is whatever the stack environment says it is.
Written with python rather than echo, so json.dumps escapes a password
containing quotes, backslashes or non-ASCII correctly — verified against
`p@ss "wo\rd' £5`, which round-trips intact.
An unset AIRFLOW_ADMIN_PASSWORD raises KeyError and the container exits.
Falling back to a generated password would silently undo the point of the
change, and a compose-level `:?` gives the same refusal a readable reason.
This does mean the variable MUST be set in Portainer before the next
deploy of either stack.
Not affected by the two Docker gotchas in the upstream docs: this image
has no USER directive so it runs as root, and the file is rewritten from
the environment on every start rather than persisted on a volume.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob
This PR makes the Airflow admin login deterministic across container restarts by writing the simple-auth-manager passwords file from an AIRFLOW_ADMIN_PASSWORD env var before airflow api-server starts, replacing the previous behavior of a random password regenerated on every restart. The compose/Dockerfile/doc changes are internally consistent (correct env var name, exec'd process, idempotent mkdir, python-based JSON escaping) and add the new destinations tap to the pipeline image build.
🔴 Severe (blocks merge)
docker-compose.portainer.staging.yml: AIRFLOW_ADMIN_PASSWORD is now required (${AIRFLOW_ADMIN_PASSWORD:?...}) for airflow-api-server to start. Since deploy.yml auto-triggers a Portainer webhook redeploy of the staging stack on every push to main, if this required variable isn't added to the already-running staging stack's Portainer environment before this PR merges, airflow-api-server will fail to start (crash-loop) after deploy. The deploy-staging job's health check only polls STAGING_BASE_URL (frontend/backend), so this failure would go undetected and the E2E/deploy gate would report staging as healthy while Airflow (and therefore the data pipeline) is down. The same applies to docker-compose.portainer.yml on the next production promote.
🟡 Minor
docs/DEPLOY.md: The AIRFLOW_ADMIN_PASSWORD requirement is documented only under the 'one-time setup checklist' for bootstrapping a brand-new stack. There's no explicit migration note telling operators of an already-provisioned staging/prod stack to add this newly-required variable before the next deploy, which is the scenario most likely to actually occur here.
## 🤖 AI Code Review (Claude Code)
This PR makes the Airflow admin login deterministic across container restarts by writing the simple-auth-manager passwords file from an AIRFLOW_ADMIN_PASSWORD env var before airflow api-server starts, replacing the previous behavior of a random password regenerated on every restart. The compose/Dockerfile/doc changes are internally consistent (correct env var name, exec'd process, idempotent mkdir, python-based JSON escaping) and add the new destinations tap to the pipeline image build.
### 🔴 Severe (blocks merge)
- **docker-compose.portainer.staging.yml**: AIRFLOW_ADMIN_PASSWORD is now required (${AIRFLOW_ADMIN_PASSWORD:?...}) for airflow-api-server to start. Since deploy.yml auto-triggers a Portainer webhook redeploy of the staging stack on every push to main, if this required variable isn't added to the already-running staging stack's Portainer environment before this PR merges, airflow-api-server will fail to start (crash-loop) after deploy. The deploy-staging job's health check only polls STAGING_BASE_URL (frontend/backend), so this failure would go undetected and the E2E/deploy gate would report staging as healthy while Airflow (and therefore the data pipeline) is down. The same applies to docker-compose.portainer.yml on the next production promote.
### 🟡 Minor
- **docs/DEPLOY.md**: The AIRFLOW_ADMIN_PASSWORD requirement is documented only under the 'one-time setup checklist' for bootstrapping a brand-new stack. There's no explicit migration note telling operators of an already-provisioned staging/prod stack to add this newly-required variable before the next deploy, which is the scenario most likely to actually occur here.
tudor
merged commit fb5a0928bd into main2026-08-31 19:50:00 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Two small pipeline changes. The first is a straggler from #137.
build(pipeline)— install the destinations tap in the imagepipeline/Dockerfileinstalls each custom Singer tap by name, andtap-uk-ees-destinationswas not in the list. I pushed this to #137 a fewminutes after it was merged, so it never landed —
maincurrently builds apipeline image that does not explicitly install the new tap.
meltano installwould most likely resolve it from itspip_url(that is howtap-uk-school-distanceworks today), so this may not be strictly load-bearing.But a brand-new plugin failing to appear is not something worth debugging from a
deploy log when one line prevents it.
fix(airflow)— a login that survives a container restartAirflow's simple auth manager generates a random password on first start and
writes it to
$AIRFLOW_HOME/simple_auth_manager_passwords.json.generated, soevery restart of the api-server invalidated the last one and the password had to
be dug back out of the container logs.
Airflow generates nothing when that file already exists, so the stack now writes
it before exec'ing the api-server:
Applied to the prod stack, the staging stack and local
docker-compose.yml.Why python rather than
echo/printf:json.dumpsescapes a passwordcontaining quotes, backslashes or non-ASCII correctly. Verified with
p@ss "wo\rd' £5, which round-trips intact — a shell heredoc would havemangled it.
Why it fails closed: an unset
AIRFLOW_ADMIN_PASSWORDraisesKeyErrorandthe container exits, and the compose
:?default gives the same refusal areadable reason. Falling back to a generated password would silently undo the
point of the change.
Neither Docker gotcha from the Airflow docs applies here: this image has no
USERdirective so it runs as root, and the file is rewritten from theenvironment on every start rather than persisted on a volume — so there is
nothing to own wrongly and nothing to lose on restart.
⚠️ Before deploying
AIRFLOW_ADMIN_PASSWORDmust be set in the Portainer stack environment forboth staging and prod before this deploys, or the api-server will not start.
AIRFLOW_ADMIN_USERstill defaults toadmin.🤖 Generated with Claude Code
https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob
🤖 AI Code Review (Claude Code)
This PR makes the Airflow admin login deterministic across container restarts by writing the simple-auth-manager passwords file from an AIRFLOW_ADMIN_PASSWORD env var before airflow api-server starts, replacing the previous behavior of a random password regenerated on every restart. The compose/Dockerfile/doc changes are internally consistent (correct env var name, exec'd process, idempotent mkdir, python-based JSON escaping) and add the new destinations tap to the pipeline image build.
🔴 Severe (blocks merge)
🟡 Minor