perf(dbt): materialize stg_ees_ks2 and stg_ees_ks4 as tables
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
KS2 attainment has 1.2M rows in long format. As a view, the pivot was re-executed inline for every downstream model (intermediate → fact), causing fact_ks2_performance CREATE TABLE to run for 18+ minutes. Materializing as tables means the pivot runs once during staging, and downstream models read from a pre-computed ~16k-row result. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,3 +1,5 @@
|
||||
{{ config(materialized='table') }}
|
||||
|
||||
-- Staging model: KS2 attainment + information
|
||||
-- Pivots long-format attainment data (one row per subject × breakdown) into
|
||||
-- wide format (one row per school per year) and joins context from info table.
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
{{ config(materialized='table') }}
|
||||
|
||||
-- Staging model: KS4 attainment data from EES
|
||||
-- KS4 performance data is long-format with breakdown dimensions (breakdown_topic,
|
||||
-- breakdown, sex). Unlike KS2 which has a subject dimension, KS4 metrics are
|
||||
|
||||
Reference in New Issue
Block a user