refactor(legacy-ks2): use explicit year→URL mapping instead of base URL pattern
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s

The file hosting uses non-deterministic URLs, so replace legacy_ks2_base_url
+ legacy_ks2_years with a single legacy_ks2_urls object mapping year codes
to download URLs. Configure the 4 pre-COVID years in meltano.yml.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor Sitaru
2026-03-31 22:44:11 +01:00
parent 6d4962639c
commit fba8e74b72
2 changed files with 21 additions and 27 deletions

View File

@@ -23,6 +23,15 @@ plugins:
- name: datasets
kind: array
description: List of EES dataset configs to extract
- name: legacy_ks2_urls
kind: object
description: "Year code → URL mapping for legacy KS2 CSVs"
config:
legacy_ks2_urls:
"201516": "http://10.0.1.224:8081/filebrowser/api/public/dl/R9jjXFWa?inline=true"
"201617": "http://10.0.1.224:8081/filebrowser/api/public/dl/tIwJPVQS?inline=true"
"201718": "http://10.0.1.224:8081/filebrowser/api/public/dl/GO7SKE0p?inline=true"
"201819": "http://10.0.1.224:8081/filebrowser/api/public/dl/jchDEHsv?inline=true"
- name: tap-uk-ofsted
namespace: uk_ofsted