perf(dbt): materialize stg_ees_ks2 and stg_ees_ks4 as tables
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m30s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
KS2 attainment has 1.2M rows in long format. As a view, the pivot was re-executed inline for every downstream model (intermediate → fact), causing fact_ks2_performance CREATE TABLE to run for 18+ minutes. Materializing as tables means the pivot runs once during staging, and downstream models read from a pre-computed ~16k-row result. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,3 +1,5 @@
|
|||||||
|
{{ config(materialized='table') }}
|
||||||
|
|
||||||
-- Staging model: KS2 attainment + information
|
-- Staging model: KS2 attainment + information
|
||||||
-- Pivots long-format attainment data (one row per subject × breakdown) into
|
-- Pivots long-format attainment data (one row per subject × breakdown) into
|
||||||
-- wide format (one row per school per year) and joins context from info table.
|
-- wide format (one row per school per year) and joins context from info table.
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
|
{{ config(materialized='table') }}
|
||||||
|
|
||||||
-- Staging model: KS4 attainment data from EES
|
-- Staging model: KS4 attainment data from EES
|
||||||
-- KS4 performance data is long-format with breakdown dimensions (breakdown_topic,
|
-- KS4 performance data is long-format with breakdown dimensions (breakdown_topic,
|
||||||
-- breakdown, sex). Unlike KS2 which has a subject dimension, KS4 metrics are
|
-- breakdown, sex). Unlike KS2 which has a subject dimension, KS4 metrics are
|
||||||
|
|||||||
Reference in New Issue
Block a user