feat(integrator): add KS2 re-import via Kestra and backend admin endpoint
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 47s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m13s
Build and Push Docker Images / Build Integrator (push) Successful in 40s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 47s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m13s
Build and Push Docker Images / Build Integrator (push) Successful in 40s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
- backend: POST /api/admin/reimport-ks2 runs full CSV migration in a thread - backend/docker-compose: ADMIN_API_KEY env var (default: changeme) so the key is stable across restarts and the integrator can call the endpoint - integrator: sources/ks2.py triggers the backend endpoint (900s timeout) - integrator: flows/ks2.yml Kestra flow (manual trigger, no schedule) To re-ingest after a DB wipe: trigger the ks2-reimport flow from the Kestra UI at http://localhost:8080. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -19,6 +19,7 @@ from slowapi.util import get_remote_address
|
||||
from slowapi.errors import RateLimitExceeded
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
|
||||
import asyncio
|
||||
from .config import settings
|
||||
from .data_loader import (
|
||||
clear_cache,
|
||||
@@ -28,6 +29,7 @@ from .data_loader import (
|
||||
)
|
||||
from .data_loader import get_data_info as get_db_info
|
||||
from .database import check_and_migrate_if_needed
|
||||
from .migration import run_full_migration
|
||||
from .schemas import METRIC_DEFINITIONS, RANKING_COLUMNS, SCHOOL_COLUMNS
|
||||
from .utils import clean_for_json
|
||||
|
||||
@@ -633,6 +635,27 @@ async def reload_data(
|
||||
return {"status": "reloaded"}
|
||||
|
||||
|
||||
@app.post("/api/admin/reimport-ks2")
|
||||
@limiter.limit("2/minute")
|
||||
async def reimport_ks2(
|
||||
request: Request,
|
||||
_: bool = Depends(verify_admin_api_key)
|
||||
):
|
||||
"""
|
||||
Re-run the full KS2 CSV migration (drop + reimport schools and results).
|
||||
Use when the database has been wiped and needs repopulating from the CSV files.
|
||||
Runs synchronously — expect this to take several minutes.
|
||||
Requires X-API-Key header with valid admin API key.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
success = await loop.run_in_executor(None, run_full_migration)
|
||||
if not success:
|
||||
raise HTTPException(status_code=500, detail="Migration failed: no CSV data found in data directory")
|
||||
clear_cache()
|
||||
load_school_data()
|
||||
return {"status": "reimported"}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# SEO FILES
|
||||
# =============================================================================
|
||||
|
||||
Reference in New Issue
Block a user