perf(compare): import-time KS4 national averages mart; no refetch on metric change #36

Merged
tudor merged 3 commits from perf/compare-loading into main 2026-07-14 20:04:46 +00:00
Owner

Fixes the noticeable compare-page load time. Profiled on staging: /api/compare was ~400ms (vs ~95ms for a detail call), and the UI refetched it on every metric change.

Root causes and fixes

  1. KS4 national averages were aggregated from the full performance dataframe on every request (~250ms — a per-year loop of full-frame scans, inherited from the old heavily-CDN-cached endpoint). Now computed once at import time as a new mart, fact_ks4_national_averages (mirroring fact_ks2_national_averages), built by the EES DAG via its existing stg_ees_ks4+ selector. _national_averages_payload just reads the two mart tables; if the KS4 mart hasn't been built yet (deploy-before-DAG window) it falls back to computing the latest year only — never the historical loop.
  2. Metric changes refired the whole comparison fetch with cache: 'no-store' — but since the redesign the metric is purely presentational. The fetch effect now depends only on the school set; URL sync is its own effect. Metric changes are now zero-network.
  3. Mount double-fetch removed: the client skips its initial fetch when the SSR payload already covers the selected schools; national averages + benchmarks travel via SSR props so nothing is lost.
  4. SSR fetches parallelised (Promise.all for comparison + metrics).

Expected effect

Server time for /api/compare drops to roughly detail-endpoint territory after the next EES DAG run (~250ms saved even before it, via the latest-year-only fallback); metric interaction goes from ~400ms round-trips to instant; initial paint stops paying for a duplicate fetch.

Tests

46 backend (2 new: mart-driven payload asserts the KS4 value comes from the mart NOT the dataframe mean; missing-mart fallback computes latest year only) · 77 frontend · tsc clean.

Deploy note

marts.fact_ks4_national_averages appears after the next school_data_weekly_ees (or full) DAG run; until then the API serves the bounded fallback. No action needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB

Fixes the noticeable compare-page load time. Profiled on staging: `/api/compare` was ~400ms (vs ~95ms for a detail call), and the UI refetched it on every metric change. ## Root causes and fixes 1. **KS4 national averages were aggregated from the full performance dataframe on every request** (~250ms — a per-year loop of full-frame scans, inherited from the old heavily-CDN-cached endpoint). Now **computed once at import time** as a new mart, `fact_ks4_national_averages` (mirroring `fact_ks2_national_averages`), built by the EES DAG via its existing `stg_ees_ks4+` selector. `_national_averages_payload` just reads the two mart tables; if the KS4 mart hasn't been built yet (deploy-before-DAG window) it falls back to computing the **latest year only** — never the historical loop. 2. **Metric changes refired the whole comparison fetch** with `cache: 'no-store'` — but since the redesign the metric is purely presentational. The fetch effect now depends only on the school set; URL sync is its own effect. Metric changes are now zero-network. 3. **Mount double-fetch removed**: the client skips its initial fetch when the SSR payload already covers the selected schools; national averages + benchmarks travel via SSR props so nothing is lost. 4. **SSR fetches parallelised** (`Promise.all` for comparison + metrics). ## Expected effect Server time for `/api/compare` drops to roughly detail-endpoint territory after the next EES DAG run (~250ms saved even before it, via the latest-year-only fallback); metric interaction goes from ~400ms round-trips to instant; initial paint stops paying for a duplicate fetch. ## Tests 46 backend (2 new: mart-driven payload asserts the KS4 value comes from the mart NOT the dataframe mean; missing-mart fallback computes latest year only) · 77 frontend · tsc clean. ## Deploy note `marts.fact_ks4_national_averages` appears after the next `school_data_weekly_ees` (or full) DAG run; until then the API serves the bounded fallback. No action needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
tudor added 2 commits 2026-07-14 12:07:25 +00:00
fact_ks4_national_averages is computed once at dbt build time (covered by
the EES DAG's stg_ees_ks4+ selector). _national_averages_payload now reads
both national-averages marts instead of scanning the performance dataframe
per year on every /api/compare request (~250ms saved per call). Fallback
for the deploy-before-DAG window computes the latest year only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
perf(compare): fetch only on school-set changes; use SSR payload; parallel page fetches
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m46s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 18s
PR Checks / Build Frontend (no push) (pull_request) Successful in 47s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 36s
PR Checks / AI Code Review (Claude) (pull_request) Failing after 5m4s
619e3a1189
- Metric changes no longer refire /api/compare (the data is already
  client-side; the picker is presentational) — the fetch effect depends
  only on the URN set, with URL sync split into its own effect.
- The initial client fetch is skipped when the SSR payload already covers
  the selected schools; national averages + benchmarks now arrive via SSR
  props so nothing is lost by skipping.
- page.tsx fetches comparison and metrics in parallel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
tudor added 1 commit 2026-07-14 12:25:26 +00:00
ci: re-run PR checks (AI review job errored without posting findings)
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 9m39s
PR Checks / Backend Smoke (pull_request) Successful in 7s
PR Checks / Build Backend (no push) (pull_request) Successful in 17s
PR Checks / Build Frontend (no push) (pull_request) Successful in 48s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 36s
PR Checks / AI Code Review (Claude) (pull_request) Failing after 2m5s
d2dc78aeb5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146VHeLAWjDVE2B5uU67jCB
tudor merged commit 8e4ee64140 into main 2026-07-14 20:04:46 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: tudor/school_compare#36