GIAS CSV dates are DD-MM-YYYY format — use to_date() instead of cast().
Exclude int_ks2_with_lineage+ and int_ks4_with_lineage+ from daily DAG
selector since they depend on EES data not yet loaded.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Declare all 34 columns needed by dbt in GIAS tap schema (target-postgres
only persists columns present in the Singer schema message)
- Use nullif() for empty-string-to-integer/date casts in staging models
- Scope daily DAG dbt build to GIAS models only (stg_gias_establishments+
stg_gias_links+) to avoid errors on unloaded sources
- Scope annual EES DAG similarly; remove redundant dbt test steps
- Make dim_school gracefully handle missing int_ofsted_latest table
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GIAS tap emits uppercase URN column — add quote: true so dbt source tests
reference "URN" instead of urn. Remove source-level tests from tables not yet
loaded (ofsted, ees, parent_view, fbit, idaci) to prevent relation-not-found
errors during dbt build.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace Airflow 2.x env vars (CORE__SECRET_KEY, CORE__INTERNAL_API_URL) with
correct Airflow 3.x equivalents (API_AUTH__JWT_SECRET, API_AUTH__JWT_ISSUER,
CORE__EXECUTION_API_SERVER_URL) on all three Airflow services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With separate containers, task workers in the scheduler need the
api-server's address for the Execution API. Defaults to localhost:8080
which fails across containers. Set INTERNAL_API_URL to the api-server's
Docker service name.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Running both in one container caused JWT secret key race conditions.
Separate containers with the same AIRFLOW__CORE__SECRET_KEY env var
ensures both processes use identical JWT signing keys. Shared
airflow_logs volume allows the api-server to read task logs written
by the scheduler.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The init container and airflow container have separate filesystems, so
airflow.cfg generated by db migrate is not available to the scheduler/
api-server. Without a config file, both processes race to generate
their own with different random JWT secret keys.
Fix by:
1. Running `airflow config list` first to generate airflow.cfg once
2. Setting a fixed SECRET_KEY via env var (>= 64 bytes for SHA512)
3. Adding sleep 3 so scheduler writes config before api-server starts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deleting airflow.cfg at container start caused the scheduler and
api-server to each generate their own random JWT secret key, leading
to 'Signature verification failed' when task workers communicated
with the api-server. Let both processes share the config file
generated by db migrate (env vars still override where needed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When scheduler and api-server run in the same container, both generate
independent JWT signing keys on startup. The scheduler's task workers
then fail with 'Invalid auth token: Signature verification failed'
when communicating with the api-server. Fix by setting a shared
INTERNAL_API_SECRET_KEY via env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docker/setup-buildx-action creates a BuildKit builder that ignores
the host daemon's registry-mirrors setting. Configure buildkitd inline
to route docker.io pulls through the local pull-through cache at
172.17.0.1:6000 (Docker bridge gateway → host port 6000).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CSV is read with dtype=str so all values arrive as strings. Declaring
LA (code) and EstablishmentNumber as IntegerType caused schema
validation failures in target-postgres. Use StringType for all columns
except URN (which is explicitly cast to int for the primary key).
Type casting happens in dbt staging models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Meltano 4.x requires an environment to be specified. Set production as
the default. Also remove the deprecated 'version: 2' field.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The meltanolabs target-postgres variant expects 'database' as the
config key, not 'dbname' (which was the pipelinewise variant's key).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The `catalog` capability forced Meltano to run --discover and generate
a catalog file (tap.properties.json) before each extraction. This fails
because our Singer SDK taps emit schemas inline and don't need external
catalog files. Removing the capability makes Meltano invoke taps
directly without catalog generation.
Also switch from deprecated `meltano elt` to `meltano run` for
Meltano 4.x compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Meltano elt requires catalog files (tap.properties.json) to exist.
These are generated by `meltano install` which discovers tap schemas
and installs the target-postgres loader. Without this step, `meltano
elt` fails with "catalog file is missing".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With LocalExecutor, tasks run in the scheduler process and logs are
written locally. Running api-server and scheduler in separate containers
meant the api-server couldn't read task logs (empty hostname in log
fetch URL). Combining them into one container eliminates the issue —
logs are always on the local filesystem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
airflow db migrate generates airflow.cfg with default values that
shadow our env vars (DAGS_FOLDER, WORKER_LOG_SERVER_HOST, etc).
Delete the generated config file before starting each service so
Airflow falls through to env var configuration exclusively.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The scheduler's log server binds to [::]:8793 but doesn't advertise a
hostname, so the api-server gets 'http://:8793/...' (no host) when
fetching task logs. Fix by setting the scheduler's hostname and
configuring WORKER_LOG_SERVER_HOST so the api-server can reach it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The api-server couldn't fetch task logs because LocalExecutor runs tasks
in the scheduler process, writing logs to its local filesystem. The
api-server tried to fetch via HTTP but the scheduler's log server had
no hostname set. Fix by sharing a named volume for logs between both
containers so the api-server reads logs directly from the filesystem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port critical patterns from the working integrator into Singer taps:
- GIAS: add 404 fallback to yesterday's date, increase timeout to 300s,
use latin-1 encoding, use dated URL for links (static URL returns 500)
- FBIT: add GIAS date fallback, increase timeout, fix encoding to latin-1
- IDACI: use dated GIAS URL with fallback instead of undated static URL,
fix encoding to latin-1, increase timeout to 300s
- Ofsted: try utf-8-sig then fall back to latin-1 encoding
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add AIRFLOW__CORE__DAGS_FOLDER env var in Dockerfile so it's always set
- Run `airflow dags reserialize` after `db migrate` in init container so
DAGs appear immediately without waiting for scheduler scan interval
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- MELTANO_BIN/DBT_BIN pointed to .venv/bin/ but Dockerfile installs globally
- Add try/except for BashOperator import to handle both Airflow 3 provider
path and legacy path, preventing silent DAG import failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port extraction logic from integrator scripts into Singer SDK taps:
- tap-uk-parent-view: scrapes Ofsted open data portal, parses survey responses (14 questions)
- tap-uk-fbit: queries FBIT API per-URN with rate limiting, computes per-pupil spend
- tap-uk-idaci: downloads IoD2019 XLSX, batch-resolves postcodes→LSOAs via postcodes.io
Update dbt models to match actual tap output schemas:
- stg_idaci now includes URN (tap does the postcode→LSOA→school join)
- stg_parent_view expanded from 8 to 13 question columns
- fact_deprivation simplified (no longer needs postcode→LSOA join in dbt)
- fact_parent_view expanded to include all 13 question metrics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The named volume was shadowing the DAGs built into the pipeline image
with an empty directory. DAGs now served directly from the image and
update on each CI build.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Typesense image has neither curl nor wget. Use bash /dev/tcp for a
simple port connectivity check instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Airflow 3 replaced `airflow webserver` with `airflow api-server` and
removed the `airflow users` CLI. Auth is now via SimpleAuthManager
configured through AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The MI CSV contains both OEIF and RC column sets simultaneously — OEIF columns
are populated for older inspections, RC columns for post-Nov-2025 inspections.
File-level detection wrongly classified all schools based on column presence alone.
Replace _detect_framework(df) with _framework_for_row(row):
- ReportCard: any rc_* column has a value
- OEIF: overall_effectiveness or quality_of_education has a value
- None: neither has data (no graded inspection on record)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The old OEIF CSV contains columns whose names include substrings like
'inclusion' and 'achievement', causing _detect_framework() to wrongly return
'ReportCard' for pre-Nov-2025 inspections.
Fix: check for OEIF-specific phrases first ('overall effectiveness', 'quality
of education', 'behaviour and attitudes'). Only if none are found, look for
multi-word RC-specific phrases. Default to OEIF as a safe fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The v4 migration already ran before _apply_schema_alterations() was added,
so the new ofsted_inspections columns were never created. Bump to v5 so the
next backend restart re-runs the migration and applies the ALTER TABLE statements.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
create_all() only creates missing tables; it won't modify tables that already
exist from an older schema version. Add _apply_schema_alterations() which runs
idempotent ADD COLUMN IF NOT EXISTS statements after every migration so
supplementary tables (like ofsted_inspections) gain new columns without
dropping their existing data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ofsted replaced single overall grades with Report Cards from Nov 2025.
Both systems are retained during the transition period.
- DB: new framework + 9 RC columns on ofsted_inspections (schema v4)
- Integrator: auto-detect OEIF vs Report Card from CSV column headers;
parse 5-level RC grades and safeguarding met/not-met
- API: expose all new fields in the ofsted response dict
- Frontend: branch on framework='ReportCard' to show safeguarding badge
+ 8-category grid; fall back to legacy OEIF layout otherwise;
always show inspection date in both layouts
- CSS: rcGrade1–5 and safeguardingMet/NotMet classes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove standalone back button div (looked out of place)
- Back button now lives in the sticky section nav bar, styled as a
bordered pill with coral accent — consistent with page design
- Fix sticky nav top offset from 0 to 3rem so it sticks below the
site-wide header instead of sliding behind it
- Increase scroll-margin-top on cards to 6rem to account for both
site header and section nav height
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two issues caused the backend to drop and reimport school data on restart:
1. schema_version table was in the drop list inside run_full_migration(),
so after any migration the breadcrumb was destroyed and the next
restart would see no version → re-trigger migration
2. Schema version was set after migration, so a crash mid-migration
left no version → infinite re-migration loop
Fix: remove schema_version from the drop list, and set the version
before running migration so crashes don't cause loops.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UX audit round 2:
- Remove Summary Strip (duplicated Ofsted grade + parent happy/safe/recommend)
- Fold "% would recommend" into Ofsted section header
- Merge SATs Results + Subject Breakdown into one section
- Merge Results Over Time chart + Year-by-Year table into one section
- Add sticky section nav with dynamic pills based on available data
- Unify colour system: replace ad-hoc pill colours with semantic status classes
- Guard Pupils & Inclusion so it only renders with actual data
- Add year to Admissions section title
- Fix progress score 0.0 colour (was neutral gap at ±0.1, now at 0)
- Remove unused .metricTrend CSS class
Page reduced from 16 to 13 sections.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The EES statistics API only exposes ~13 publications; admissions data is not
among them. Switch to the EES content API (content.explore-education-statistics.
service.gov.uk) which covers all publications.
- ees.py: add get_content_release_id() and download_release_zip_csv() that
fetch the release ZIP and extract a named CSV member from it
- admissions.py: use corrected slug (primary-and-secondary-school-applications-
and-offers), correct column names from actual CSV (school_urn,
total_number_places_offered, times_put_as_1st_preference, etc.), derive
first_preference_offers_pct from offer/application ratio, filter to primary
schools only, keep most recent year per URN
Also includes SchoolDetailView UX redesign: parent-first section ordering,
plain-English labels, national average benchmarks, progress score colour
coding, expanded header, quick summary strip, and CSS consolidation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before dropping tables, save all existing lat/lon coordinates keyed by URN.
After reimport, merge cached coordinates with any newly geocoded ones so
schools that already have coordinates skip the postcodes.io API call.
This makes repeated reimports fast and avoids re-geocoding ~15k schools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ofsted renamed all columns in the OEIF framework:
- grades are now 'Latest OEIF overall effectiveness' etc.
- dates are 'Inspection start date of latest OEIF graded inspection'
Replace flat COLUMN_MAP with a priority list per field so both current
OEIF and legacy column names work without duplicate-column conflicts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
