refactor(phase): merge KS2+KS4 into fact_performance, fix all phase inconsistencies
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 50s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m24s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 50s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m24s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Root cause: the UNION ALL query in data_loader.py produced two rows per all-through school per year (one KS2, one KS4), with drop_duplicates() silently discarding the KS4 row. Fixes: - New dbt mart `fact_performance`: FULL OUTER JOIN of fact_ks2_performance and fact_ks4_performance on (urn, year). One row per school per year. All-through schools have both KS2 and KS4 columns populated. - data_loader.py: replace 175-line UNION ALL with a simple JOIN to fact_performance. No more duplicate rows or drop_duplicates needed. - sync_typesense.py: single LATERAL JOIN to fact_performance instead of two separate KS2/KS4 joins. - app.py: remove drop_duplicates (no longer needed); add PHASE_GROUPS constant so all-through/middle schools appear in primary and secondary filter results (were previously invisible to both); scope result_filters gender/admissions_policies to secondary schools only. - HomeView.tsx: isSecondaryView is now majority-based (not "any secondary") and isMixedView shows both sort option sets for mixed result sets. - school/[slug]/page.tsx: all-through schools route to SchoolDetailView (renders both SATs + GCSE sections) instead of SecondarySchoolDetailView (KS4-only). Dedicated SEO metadata for all-through schools. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -109,11 +109,12 @@ def haversine_distance(lat1: float, lon1: float, lat2: float, lon2: float) -> fl
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# MAIN DATA LOAD — joins dim_school + dim_location + fact_ks2_performance
|
||||
# MAIN DATA LOAD — joins dim_school + dim_location + fact_performance
|
||||
# fact_performance is a merged KS2+KS4 table (one row per URN per year).
|
||||
# All-through schools have both KS2 and KS4 columns populated in the same row.
|
||||
# =============================================================================
|
||||
|
||||
_MAIN_QUERY = text("""
|
||||
-- Branch 1: Primary schools (KS2 data; KS4 columns NULL)
|
||||
SELECT
|
||||
s.urn,
|
||||
s.school_name,
|
||||
@@ -139,155 +140,67 @@ _MAIN_QUERY = text("""
|
||||
l.postcode,
|
||||
l.latitude,
|
||||
l.longitude,
|
||||
k.year,
|
||||
k.source_urn,
|
||||
k.total_pupils,
|
||||
k.eligible_pupils,
|
||||
-- KS2 columns
|
||||
k.rwm_expected_pct,
|
||||
k.rwm_high_pct,
|
||||
k.reading_expected_pct,
|
||||
k.reading_high_pct,
|
||||
k.reading_avg_score,
|
||||
k.reading_progress,
|
||||
k.writing_expected_pct,
|
||||
k.writing_high_pct,
|
||||
k.writing_progress,
|
||||
k.maths_expected_pct,
|
||||
k.maths_high_pct,
|
||||
k.maths_avg_score,
|
||||
k.maths_progress,
|
||||
k.gps_expected_pct,
|
||||
k.gps_high_pct,
|
||||
k.gps_avg_score,
|
||||
k.science_expected_pct,
|
||||
k.reading_absence_pct,
|
||||
k.writing_absence_pct,
|
||||
k.maths_absence_pct,
|
||||
k.gps_absence_pct,
|
||||
k.science_absence_pct,
|
||||
k.rwm_expected_boys_pct,
|
||||
k.rwm_high_boys_pct,
|
||||
k.rwm_expected_girls_pct,
|
||||
k.rwm_high_girls_pct,
|
||||
k.rwm_expected_disadvantaged_pct,
|
||||
k.rwm_expected_non_disadvantaged_pct,
|
||||
k.disadvantaged_gap,
|
||||
k.disadvantaged_pct,
|
||||
k.eal_pct,
|
||||
k.sen_support_pct,
|
||||
k.sen_ehcp_pct,
|
||||
k.stability_pct,
|
||||
-- KS4 columns (NULL for primary)
|
||||
NULL::numeric AS attainment_8_score,
|
||||
NULL::numeric AS progress_8_score,
|
||||
NULL::numeric AS progress_8_lower_ci,
|
||||
NULL::numeric AS progress_8_upper_ci,
|
||||
NULL::numeric AS progress_8_english,
|
||||
NULL::numeric AS progress_8_maths,
|
||||
NULL::numeric AS progress_8_ebacc,
|
||||
NULL::numeric AS progress_8_open,
|
||||
NULL::numeric AS english_maths_strong_pass_pct,
|
||||
NULL::numeric AS english_maths_standard_pass_pct,
|
||||
NULL::numeric AS ebacc_entry_pct,
|
||||
NULL::numeric AS ebacc_strong_pass_pct,
|
||||
NULL::numeric AS ebacc_standard_pass_pct,
|
||||
NULL::numeric AS ebacc_avg_score,
|
||||
NULL::numeric AS gcse_grade_91_pct,
|
||||
NULL::numeric AS prior_attainment_avg
|
||||
p.year,
|
||||
p.source_urn,
|
||||
p.total_pupils,
|
||||
p.eligible_pupils,
|
||||
-- KS2 columns (NULL for pure secondary schools)
|
||||
p.rwm_expected_pct,
|
||||
p.rwm_high_pct,
|
||||
p.reading_expected_pct,
|
||||
p.reading_high_pct,
|
||||
p.reading_avg_score,
|
||||
p.reading_progress,
|
||||
p.writing_expected_pct,
|
||||
p.writing_high_pct,
|
||||
p.writing_progress,
|
||||
p.maths_expected_pct,
|
||||
p.maths_high_pct,
|
||||
p.maths_avg_score,
|
||||
p.maths_progress,
|
||||
p.gps_expected_pct,
|
||||
p.gps_high_pct,
|
||||
p.gps_avg_score,
|
||||
p.science_expected_pct,
|
||||
p.reading_absence_pct,
|
||||
p.writing_absence_pct,
|
||||
p.maths_absence_pct,
|
||||
p.gps_absence_pct,
|
||||
p.science_absence_pct,
|
||||
p.rwm_expected_boys_pct,
|
||||
p.rwm_high_boys_pct,
|
||||
p.rwm_expected_girls_pct,
|
||||
p.rwm_high_girls_pct,
|
||||
p.rwm_expected_disadvantaged_pct,
|
||||
p.rwm_expected_non_disadvantaged_pct,
|
||||
p.disadvantaged_gap,
|
||||
p.disadvantaged_pct,
|
||||
p.eal_pct,
|
||||
p.stability_pct,
|
||||
-- KS4 columns (NULL for pure primary schools)
|
||||
p.attainment_8_score,
|
||||
p.progress_8_score,
|
||||
p.progress_8_lower_ci,
|
||||
p.progress_8_upper_ci,
|
||||
p.progress_8_english,
|
||||
p.progress_8_maths,
|
||||
p.progress_8_ebacc,
|
||||
p.progress_8_open,
|
||||
p.english_maths_strong_pass_pct,
|
||||
p.english_maths_standard_pass_pct,
|
||||
p.ebacc_entry_pct,
|
||||
p.ebacc_strong_pass_pct,
|
||||
p.ebacc_standard_pass_pct,
|
||||
p.ebacc_avg_score,
|
||||
p.gcse_grade_91_pct,
|
||||
p.prior_attainment_avg,
|
||||
-- SEN (coalesced KS2+KS4 in fact_performance)
|
||||
p.sen_support_pct,
|
||||
p.sen_ehcp_pct
|
||||
FROM marts.dim_school s
|
||||
JOIN marts.dim_location l ON s.urn = l.urn
|
||||
JOIN marts.fact_ks2_performance k ON s.urn = k.urn
|
||||
|
||||
UNION ALL
|
||||
|
||||
-- Branch 2: Secondary schools (KS4 data; KS2 columns NULL)
|
||||
SELECT
|
||||
s.urn,
|
||||
s.school_name,
|
||||
s.phase,
|
||||
s.school_type,
|
||||
s.academy_trust_name AS trust_name,
|
||||
s.academy_trust_uid AS trust_uid,
|
||||
s.religious_character AS religious_denomination,
|
||||
s.gender,
|
||||
s.age_range,
|
||||
s.admissions_policy,
|
||||
s.capacity,
|
||||
s.headteacher_name,
|
||||
s.website,
|
||||
s.ofsted_grade,
|
||||
s.ofsted_date,
|
||||
s.ofsted_framework,
|
||||
l.local_authority_name AS local_authority,
|
||||
l.local_authority_code,
|
||||
l.address_line1 AS address1,
|
||||
l.address_line2 AS address2,
|
||||
l.town,
|
||||
l.postcode,
|
||||
l.latitude,
|
||||
l.longitude,
|
||||
k4.year,
|
||||
k4.source_urn,
|
||||
k4.total_pupils,
|
||||
k4.eligible_pupils,
|
||||
-- KS2 columns (NULL for secondary)
|
||||
NULL::numeric AS rwm_expected_pct,
|
||||
NULL::numeric AS rwm_high_pct,
|
||||
NULL::numeric AS reading_expected_pct,
|
||||
NULL::numeric AS reading_high_pct,
|
||||
NULL::numeric AS reading_avg_score,
|
||||
NULL::numeric AS reading_progress,
|
||||
NULL::numeric AS writing_expected_pct,
|
||||
NULL::numeric AS writing_high_pct,
|
||||
NULL::numeric AS writing_progress,
|
||||
NULL::numeric AS maths_expected_pct,
|
||||
NULL::numeric AS maths_high_pct,
|
||||
NULL::numeric AS maths_avg_score,
|
||||
NULL::numeric AS maths_progress,
|
||||
NULL::numeric AS gps_expected_pct,
|
||||
NULL::numeric AS gps_high_pct,
|
||||
NULL::numeric AS gps_avg_score,
|
||||
NULL::numeric AS science_expected_pct,
|
||||
NULL::numeric AS reading_absence_pct,
|
||||
NULL::numeric AS writing_absence_pct,
|
||||
NULL::numeric AS maths_absence_pct,
|
||||
NULL::numeric AS gps_absence_pct,
|
||||
NULL::numeric AS science_absence_pct,
|
||||
NULL::numeric AS rwm_expected_boys_pct,
|
||||
NULL::numeric AS rwm_high_boys_pct,
|
||||
NULL::numeric AS rwm_expected_girls_pct,
|
||||
NULL::numeric AS rwm_high_girls_pct,
|
||||
NULL::numeric AS rwm_expected_disadvantaged_pct,
|
||||
NULL::numeric AS rwm_expected_non_disadvantaged_pct,
|
||||
NULL::numeric AS disadvantaged_gap,
|
||||
NULL::numeric AS disadvantaged_pct,
|
||||
NULL::numeric AS eal_pct,
|
||||
k4.sen_support_pct,
|
||||
k4.sen_ehcp_pct,
|
||||
NULL::numeric AS stability_pct,
|
||||
-- KS4 columns
|
||||
k4.attainment_8_score,
|
||||
k4.progress_8_score,
|
||||
k4.progress_8_lower_ci,
|
||||
k4.progress_8_upper_ci,
|
||||
k4.progress_8_english,
|
||||
k4.progress_8_maths,
|
||||
k4.progress_8_ebacc,
|
||||
k4.progress_8_open,
|
||||
k4.english_maths_strong_pass_pct,
|
||||
k4.english_maths_standard_pass_pct,
|
||||
k4.ebacc_entry_pct,
|
||||
k4.ebacc_strong_pass_pct,
|
||||
k4.ebacc_standard_pass_pct,
|
||||
k4.ebacc_avg_score,
|
||||
k4.gcse_grade_91_pct,
|
||||
k4.prior_attainment_avg
|
||||
FROM marts.dim_school s
|
||||
JOIN marts.dim_location l ON s.urn = l.urn
|
||||
JOIN marts.fact_ks4_performance k4 ON s.urn = k4.urn
|
||||
|
||||
ORDER BY school_name, year
|
||||
JOIN marts.fact_performance p ON s.urn = p.urn
|
||||
ORDER BY s.school_name, p.year
|
||||
""")
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user