Root cause of the staging e2e failures persisting after the marts rebuild: the DAG docstring promises Extract → … → Sync Typesense → Invalidate Cache, but no such task ever existed — the backend keeps serving its startup-cached (empty, in this incident) DataFrame until a container restart.
Adds a POST /api/admin/reload task (X-API-Key auth, same pattern as the sitemap DAG) to the end of the daily, monthly-Ofsted, and annual-EES DAGs. The task fails loudly on non-200 so a broken reload is visible in Airflow instead of silently leaving the API stale.
Requires BACKEND_URL and ADMIN_API_KEY env vars on the Airflow containers (the sitemap DAG already uses the same ones).
Immediate incident remedy is independent of this PR: restart the staging backend container or call /api/admin/reload manually, then re-run the e2e gate.
Root cause of the staging e2e failures persisting after the marts rebuild: the DAG docstring promises Extract → … → Sync Typesense → **Invalidate Cache**, but no such task ever existed — the backend keeps serving its startup-cached (empty, in this incident) DataFrame until a container restart.
Adds a `POST /api/admin/reload` task (X-API-Key auth, same pattern as the sitemap DAG) to the end of the daily, monthly-Ofsted, and annual-EES DAGs. The task fails loudly on non-200 so a broken reload is visible in Airflow instead of silently leaving the API stale.
Requires `BACKEND_URL` and `ADMIN_API_KEY` env vars on the Airflow containers (the sitemap DAG already uses the same ones).
Immediate incident remedy is independent of this PR: restart the staging backend container or call `/api/admin/reload` manually, then re-run the e2e gate.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The daily/monthly/annual DAG docstring promised an Invalidate Cache step
that never existed — after a marts rebuild the backend kept serving its
startup-cached (possibly empty) DataFrame until a container restart.
Add a POST /api/admin/reload task at the end of each pipeline DAG,
mirroring the sitemap DAG's admin-call pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR adds a cache-invalidation step (POST /api/admin/reload) to the end of the daily, monthly, and annual EES pipeline DAGs so the backend picks up freshly rebuilt marts data instead of serving stale/cached data until a restart. The mechanism itself (curl with HTTP-status check, non-zero exit on failure) is sound and matches the backend's existing verify_admin_api_key auth, but the change is applied inconsistently across DAGs.
🔴 Severe (blocks merge)
pipeline/dags/school_data_pipeline.py: The school_data_annual_idaci DAG (extract_idaci >> dbt_build_idaci) rebuilds the fact_deprivation mart but no invalidate_cache task was added to it, unlike the other three DAGs. Per the PR's own stated rationale, the backend will keep serving stale (pre-IDACI-update) data indefinitely after every IDACI rebuild until the container is manually restarted, since nothing calls /api/admin/reload for this pipeline.
🟡 Minor
pipeline/dags/school_data_pipeline.py: The curl call in INVALIDATE_CACHE_CMD has no --max-time/--connect-timeout, so if the backend is unreachable or hangs, the Airflow task (and downstream retries) can block indefinitely instead of failing fast.
## 🤖 AI Code Review (Claude Code)
The PR adds a cache-invalidation step (POST /api/admin/reload) to the end of the daily, monthly, and annual EES pipeline DAGs so the backend picks up freshly rebuilt marts data instead of serving stale/cached data until a restart. The mechanism itself (curl with HTTP-status check, non-zero exit on failure) is sound and matches the backend's existing verify_admin_api_key auth, but the change is applied inconsistently across DAGs.
### 🔴 Severe (blocks merge)
- **pipeline/dags/school_data_pipeline.py**: The school_data_annual_idaci DAG (extract_idaci >> dbt_build_idaci) rebuilds the fact_deprivation mart but no invalidate_cache task was added to it, unlike the other three DAGs. Per the PR's own stated rationale, the backend will keep serving stale (pre-IDACI-update) data indefinitely after every IDACI rebuild until the container is manually restarted, since nothing calls /api/admin/reload for this pipeline.
### 🟡 Minor
- **pipeline/dags/school_data_pipeline.py**: The curl call in INVALIDATE_CACHE_CMD has no --max-time/--connect-timeout, so if the backend is unreachable or hangs, the Airflow task (and downstream retries) can block indefinitely instead of failing fast.
Addresses AI-review findings: the annual IDACI DAG also rebuilds a mart
(fact_deprivation) and needs the reload; curl gets connect/max timeouts
so an unreachable backend fails fast instead of hanging the task.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The diff wires up cache invalidation (a POST to the backend's existing, authenticated /api/admin/reload endpoint) as the final task in all four pipeline DAGs (daily, monthly, annual EES, annual IDACI), fixing the previously reported issue where the backend served stale data after rebuilds. The pattern mirrors an existing DAG (sitemap_dag.py), uses proper curl timeouts, checks the HTTP status code, and fails the task loudly on error, which is consistent with the codebase's existing 'changeme' env-var-fallback convention used throughout docker-compose configs.
✅ No issues found.
## 🤖 AI Code Review (Claude Code)
The diff wires up cache invalidation (a POST to the backend's existing, authenticated /api/admin/reload endpoint) as the final task in all four pipeline DAGs (daily, monthly, annual EES, annual IDACI), fixing the previously reported issue where the backend served stale data after rebuilds. The pattern mirrors an existing DAG (sitemap_dag.py), uses proper curl timeouts, checks the HTTP status code, and fails the task loudly on error, which is consistent with the codebase's existing 'changeme' env-var-fallback convention used throughout docker-compose configs.
✅ No issues found.
tudor
merged commit 159207c6f5 into main2026-07-09 20:41:42 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Root cause of the staging e2e failures persisting after the marts rebuild: the DAG docstring promises Extract → … → Sync Typesense → Invalidate Cache, but no such task ever existed — the backend keeps serving its startup-cached (empty, in this incident) DataFrame until a container restart.
Adds a
POST /api/admin/reloadtask (X-API-Key auth, same pattern as the sitemap DAG) to the end of the daily, monthly-Ofsted, and annual-EES DAGs. The task fails loudly on non-200 so a broken reload is visible in Airflow instead of silently leaving the API stale.Requires
BACKEND_URLandADMIN_API_KEYenv vars on the Airflow containers (the sitemap DAG already uses the same ones).Immediate incident remedy is independent of this PR: restart the staging backend container or call
/api/admin/reloadmanually, then re-run the e2e gate.🤖 Generated with Claude Code
🤖 AI Code Review (Claude Code)
The PR adds a cache-invalidation step (POST /api/admin/reload) to the end of the daily, monthly, and annual EES pipeline DAGs so the backend picks up freshly rebuilt marts data instead of serving stale/cached data until a restart. The mechanism itself (curl with HTTP-status check, non-zero exit on failure) is sound and matches the backend's existing verify_admin_api_key auth, but the change is applied inconsistently across DAGs.
🔴 Severe (blocks merge)
🟡 Minor
🤖 AI Code Review (Claude Code)
The diff wires up cache invalidation (a POST to the backend's existing, authenticated /api/admin/reload endpoint) as the final task in all four pipeline DAGs (daily, monthly, annual EES, annual IDACI), fixing the previously reported issue where the backend served stale data after rebuilds. The pattern mirrors an existing DAG (sitemap_dag.py), uses proper curl timeouts, checks the HTTP status code, and fails the task loudly on error, which is consistent with the codebase's existing 'changeme' env-var-fallback convention used throughout docker-compose configs.
✅ No issues found.