refactor(phase): merge KS2+KS4 into fact_performance, fix all phase inconsistencies
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 50s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m24s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s

Root cause: the UNION ALL query in data_loader.py produced two rows per
all-through school per year (one KS2, one KS4), with drop_duplicates()
silently discarding the KS4 row. Fixes:

- New dbt mart `fact_performance`: FULL OUTER JOIN of fact_ks2_performance
  and fact_ks4_performance on (urn, year). One row per school per year.
  All-through schools have both KS2 and KS4 columns populated.
- data_loader.py: replace 175-line UNION ALL with a simple JOIN to
  fact_performance. No more duplicate rows or drop_duplicates needed.
- sync_typesense.py: single LATERAL JOIN to fact_performance instead of
  two separate KS2/KS4 joins.
- app.py: remove drop_duplicates (no longer needed); add PHASE_GROUPS
  constant so all-through/middle schools appear in primary and secondary
  filter results (were previously invisible to both); scope result_filters
  gender/admissions_policies to secondary schools only.
- HomeView.tsx: isSecondaryView is now majority-based (not "any secondary")
  and isMixedView shows both sort option sets for mixed result sets.
- school/[slug]/page.tsx: all-through schools route to SchoolDetailView
  (renders both SATs + GCSE sections) instead of SecondarySchoolDetailView
  (KS4-only). Dedicated SEO metadata for all-through schools.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-30 14:07:30 +01:00
parent 695a571c1f
commit 6e5249aa1e
7 changed files with 227 additions and 216 deletions

View File

@@ -109,11 +109,12 @@ def haversine_distance(lat1: float, lon1: float, lat2: float, lon2: float) -> fl
# =============================================================================
# MAIN DATA LOAD — joins dim_school + dim_location + fact_ks2_performance
# MAIN DATA LOAD — joins dim_school + dim_location + fact_performance
# fact_performance is a merged KS2+KS4 table (one row per URN per year).
# All-through schools have both KS2 and KS4 columns populated in the same row.
# =============================================================================
_MAIN_QUERY = text("""
-- Branch 1: Primary schools (KS2 data; KS4 columns NULL)
SELECT
s.urn,
s.school_name,
@@ -139,155 +140,67 @@ _MAIN_QUERY = text("""
l.postcode,
l.latitude,
l.longitude,
k.year,
k.source_urn,
k.total_pupils,
k.eligible_pupils,
-- KS2 columns
k.rwm_expected_pct,
k.rwm_high_pct,
k.reading_expected_pct,
k.reading_high_pct,
k.reading_avg_score,
k.reading_progress,
k.writing_expected_pct,
k.writing_high_pct,
k.writing_progress,
k.maths_expected_pct,
k.maths_high_pct,
k.maths_avg_score,
k.maths_progress,
k.gps_expected_pct,
k.gps_high_pct,
k.gps_avg_score,
k.science_expected_pct,
k.reading_absence_pct,
k.writing_absence_pct,
k.maths_absence_pct,
k.gps_absence_pct,
k.science_absence_pct,
k.rwm_expected_boys_pct,
k.rwm_high_boys_pct,
k.rwm_expected_girls_pct,
k.rwm_high_girls_pct,
k.rwm_expected_disadvantaged_pct,
k.rwm_expected_non_disadvantaged_pct,
k.disadvantaged_gap,
k.disadvantaged_pct,
k.eal_pct,
k.sen_support_pct,
k.sen_ehcp_pct,
k.stability_pct,
-- KS4 columns (NULL for primary)
NULL::numeric AS attainment_8_score,
NULL::numeric AS progress_8_score,
NULL::numeric AS progress_8_lower_ci,
NULL::numeric AS progress_8_upper_ci,
NULL::numeric AS progress_8_english,
NULL::numeric AS progress_8_maths,
NULL::numeric AS progress_8_ebacc,
NULL::numeric AS progress_8_open,
NULL::numeric AS english_maths_strong_pass_pct,
NULL::numeric AS english_maths_standard_pass_pct,
NULL::numeric AS ebacc_entry_pct,
NULL::numeric AS ebacc_strong_pass_pct,
NULL::numeric AS ebacc_standard_pass_pct,
NULL::numeric AS ebacc_avg_score,
NULL::numeric AS gcse_grade_91_pct,
NULL::numeric AS prior_attainment_avg
p.year,
p.source_urn,
p.total_pupils,
p.eligible_pupils,
-- KS2 columns (NULL for pure secondary schools)
p.rwm_expected_pct,
p.rwm_high_pct,
p.reading_expected_pct,
p.reading_high_pct,
p.reading_avg_score,
p.reading_progress,
p.writing_expected_pct,
p.writing_high_pct,
p.writing_progress,
p.maths_expected_pct,
p.maths_high_pct,
p.maths_avg_score,
p.maths_progress,
p.gps_expected_pct,
p.gps_high_pct,
p.gps_avg_score,
p.science_expected_pct,
p.reading_absence_pct,
p.writing_absence_pct,
p.maths_absence_pct,
p.gps_absence_pct,
p.science_absence_pct,
p.rwm_expected_boys_pct,
p.rwm_high_boys_pct,
p.rwm_expected_girls_pct,
p.rwm_high_girls_pct,
p.rwm_expected_disadvantaged_pct,
p.rwm_expected_non_disadvantaged_pct,
p.disadvantaged_gap,
p.disadvantaged_pct,
p.eal_pct,
p.stability_pct,
-- KS4 columns (NULL for pure primary schools)
p.attainment_8_score,
p.progress_8_score,
p.progress_8_lower_ci,
p.progress_8_upper_ci,
p.progress_8_english,
p.progress_8_maths,
p.progress_8_ebacc,
p.progress_8_open,
p.english_maths_strong_pass_pct,
p.english_maths_standard_pass_pct,
p.ebacc_entry_pct,
p.ebacc_strong_pass_pct,
p.ebacc_standard_pass_pct,
p.ebacc_avg_score,
p.gcse_grade_91_pct,
p.prior_attainment_avg,
-- SEN (coalesced KS2+KS4 in fact_performance)
p.sen_support_pct,
p.sen_ehcp_pct
FROM marts.dim_school s
JOIN marts.dim_location l ON s.urn = l.urn
JOIN marts.fact_ks2_performance k ON s.urn = k.urn
UNION ALL
-- Branch 2: Secondary schools (KS4 data; KS2 columns NULL)
SELECT
s.urn,
s.school_name,
s.phase,
s.school_type,
s.academy_trust_name AS trust_name,
s.academy_trust_uid AS trust_uid,
s.religious_character AS religious_denomination,
s.gender,
s.age_range,
s.admissions_policy,
s.capacity,
s.headteacher_name,
s.website,
s.ofsted_grade,
s.ofsted_date,
s.ofsted_framework,
l.local_authority_name AS local_authority,
l.local_authority_code,
l.address_line1 AS address1,
l.address_line2 AS address2,
l.town,
l.postcode,
l.latitude,
l.longitude,
k4.year,
k4.source_urn,
k4.total_pupils,
k4.eligible_pupils,
-- KS2 columns (NULL for secondary)
NULL::numeric AS rwm_expected_pct,
NULL::numeric AS rwm_high_pct,
NULL::numeric AS reading_expected_pct,
NULL::numeric AS reading_high_pct,
NULL::numeric AS reading_avg_score,
NULL::numeric AS reading_progress,
NULL::numeric AS writing_expected_pct,
NULL::numeric AS writing_high_pct,
NULL::numeric AS writing_progress,
NULL::numeric AS maths_expected_pct,
NULL::numeric AS maths_high_pct,
NULL::numeric AS maths_avg_score,
NULL::numeric AS maths_progress,
NULL::numeric AS gps_expected_pct,
NULL::numeric AS gps_high_pct,
NULL::numeric AS gps_avg_score,
NULL::numeric AS science_expected_pct,
NULL::numeric AS reading_absence_pct,
NULL::numeric AS writing_absence_pct,
NULL::numeric AS maths_absence_pct,
NULL::numeric AS gps_absence_pct,
NULL::numeric AS science_absence_pct,
NULL::numeric AS rwm_expected_boys_pct,
NULL::numeric AS rwm_high_boys_pct,
NULL::numeric AS rwm_expected_girls_pct,
NULL::numeric AS rwm_high_girls_pct,
NULL::numeric AS rwm_expected_disadvantaged_pct,
NULL::numeric AS rwm_expected_non_disadvantaged_pct,
NULL::numeric AS disadvantaged_gap,
NULL::numeric AS disadvantaged_pct,
NULL::numeric AS eal_pct,
k4.sen_support_pct,
k4.sen_ehcp_pct,
NULL::numeric AS stability_pct,
-- KS4 columns
k4.attainment_8_score,
k4.progress_8_score,
k4.progress_8_lower_ci,
k4.progress_8_upper_ci,
k4.progress_8_english,
k4.progress_8_maths,
k4.progress_8_ebacc,
k4.progress_8_open,
k4.english_maths_strong_pass_pct,
k4.english_maths_standard_pass_pct,
k4.ebacc_entry_pct,
k4.ebacc_strong_pass_pct,
k4.ebacc_standard_pass_pct,
k4.ebacc_avg_score,
k4.gcse_grade_91_pct,
k4.prior_attainment_avg
FROM marts.dim_school s
JOIN marts.dim_location l ON s.urn = l.urn
JOIN marts.fact_ks4_performance k4 ON s.urn = k4.urn
ORDER BY school_name, year
JOIN marts.fact_performance p ON s.urn = p.urn
ORDER BY s.school_name, p.year
""")