refactor(legacy-ks2): use explicit year→URL mapping instead of base URL pattern
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
The file hosting uses non-deterministic URLs, so replace legacy_ks2_base_url + legacy_ks2_years with a single legacy_ks2_urls object mapping year codes to download URLs. Configure the 4 pre-COVID years in meltano.yml. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -23,6 +23,15 @@ plugins:
|
||||
- name: datasets
|
||||
kind: array
|
||||
description: List of EES dataset configs to extract
|
||||
- name: legacy_ks2_urls
|
||||
kind: object
|
||||
description: "Year code → URL mapping for legacy KS2 CSVs"
|
||||
config:
|
||||
legacy_ks2_urls:
|
||||
"201516": "http://10.0.1.224:8081/filebrowser/api/public/dl/R9jjXFWa?inline=true"
|
||||
"201617": "http://10.0.1.224:8081/filebrowser/api/public/dl/tIwJPVQS?inline=true"
|
||||
"201718": "http://10.0.1.224:8081/filebrowser/api/public/dl/GO7SKE0p?inline=true"
|
||||
"201819": "http://10.0.1.224:8081/filebrowser/api/public/dl/jchDEHsv?inline=true"
|
||||
|
||||
- name: tap-uk-ofsted
|
||||
namespace: uk_ofsted
|
||||
|
||||
Reference in New Issue
Block a user