perf(dbt): materialize stg_ees_ks2 and stg_ees_ks4 as tables
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s

KS2 attainment has 1.2M rows in long format. As a view, the pivot was
re-executed inline for every downstream model (intermediate → fact),
causing fact_ks2_performance CREATE TABLE to run for 18+ minutes.

Materializing as tables means the pivot runs once during staging, and
downstream models read from a pre-computed ~16k-row result.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-27 11:20:20 +00:00
parent 33b395d2bd
commit ae9fd26eba
2 changed files with 4 additions and 0 deletions

View File

@@ -1,3 +1,5 @@
{{ config(materialized='table') }}
-- Staging model: KS2 attainment + information
-- Pivots long-format attainment data (one row per subject × breakdown) into
-- wide format (one row per school per year) and joins context from info table.

View File

@@ -1,3 +1,5 @@
{{ config(materialized='table') }}
-- Staging model: KS4 attainment data from EES
-- KS4 performance data is long-format with breakdown dimensions (breakdown_topic,
-- breakdown, sex). Unlike KS2 which has a subject dimension, KS4 metrics are