feat(integrator): add KS2 re-import via Kestra and backend admin endpoint
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 47s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m13s
Build and Push Docker Images / Build Integrator (push) Successful in 40s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s

- backend: POST /api/admin/reimport-ks2 runs full CSV migration in a thread
- backend/docker-compose: ADMIN_API_KEY env var (default: changeme) so the
  key is stable across restarts and the integrator can call the endpoint
- integrator: sources/ks2.py triggers the backend endpoint (900s timeout)
- integrator: flows/ks2.yml Kestra flow (manual trigger, no schedule)

To re-ingest after a DB wipe: trigger the ks2-reimport flow from the
Kestra UI at http://localhost:8080.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-24 12:25:29 +00:00
parent 822ec936bf
commit f1fb847164
5 changed files with 98 additions and 0 deletions

View File

@@ -19,6 +19,7 @@ from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded
from starlette.middleware.base import BaseHTTPMiddleware
import asyncio
from .config import settings
from .data_loader import (
clear_cache,
@@ -28,6 +29,7 @@ from .data_loader import (
)
from .data_loader import get_data_info as get_db_info
from .database import check_and_migrate_if_needed
from .migration import run_full_migration
from .schemas import METRIC_DEFINITIONS, RANKING_COLUMNS, SCHOOL_COLUMNS
from .utils import clean_for_json
@@ -633,6 +635,27 @@ async def reload_data(
return {"status": "reloaded"}
@app.post("/api/admin/reimport-ks2")
@limiter.limit("2/minute")
async def reimport_ks2(
request: Request,
_: bool = Depends(verify_admin_api_key)
):
"""
Re-run the full KS2 CSV migration (drop + reimport schools and results).
Use when the database has been wiped and needs repopulating from the CSV files.
Runs synchronously — expect this to take several minutes.
Requires X-API-Key header with valid admin API key.
"""
loop = asyncio.get_event_loop()
success = await loop.run_in_executor(None, run_full_migration)
if not success:
raise HTTPException(status_code=500, detail="Migration failed: no CSV data found in data directory")
clear_cache()
load_school_data()
return {"status": "reimported"}
# =============================================================================
# SEO FILES
# =============================================================================