TudorandClaude Opus 5
7650b16f62
GIAS ships the whole UK plus overseas and offshore establishments. None of
them carry comparable DfE performance data — Wales does not publish on the
English measures at all — so every one of these pages rendered with null
results, null Ofsted and null phase. There were 2,036 of them: 1,569 Welsh,
123 offshore (Jersey, Guernsey, Isle of Man, Gibraltar), 316 British schools
overseas and 28 service children's schools. All 2,036 were being submitted to
search engines, alongside 29 local authorities that existed in the filters
purely to list them.
Filter at the mart boundary rather than the view layer. dim_school and
dim_location both exclude TypeOfEstablishment in {25, 26, 30, 37}, listed once
as vars.non_england_school_type_codes. Everything downstream reads those two
marts — search, the school page, /api/filters, rankings, Typesense and
build_sitemap() — so one filter removes them from the site and the sitemap
together, and Typesense drops them on its next rebuild since it recreates the
collection and swaps the alias rather than upserting in place.
coalesce rather than a bare NOT IN: a null type code would make the predicate
null and drop the row silently, and an unknown type is not grounds for
exclusion. No establishment has a null type today, but a future GIAS refresh
could ship one and the loss would be invisible.
assert_england_only_schools guards both directions: no excluded type survives
in dim_school, and dim_location holds no URN dim_school lacks — the API
inner-joins them, so the two filters drifting apart would silently shrink the
corpus.
Corpus goes from 27,229 schools to 25,193, and the authority list from 182 to
153. The 1,569 Welsh URLs now 404.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
99 lines
4.1 KiB
SQL
99 lines
4.1 KiB
SQL
-- Mart: Canonical school dimension — one row per active URN
|
|
|
|
-- int_ofsted_latest is only ref()'d inside a conditional block below (it may not
|
|
-- exist before the Ofsted pipeline first runs), so declare the dependency
|
|
-- explicitly for dbt's DAG.
|
|
-- depends_on: {{ ref('int_ofsted_latest') }}
|
|
|
|
with schools as (
|
|
select * from {{ ref('stg_gias_establishments') }}
|
|
)
|
|
|
|
{% set ofsted_relation = adapter.get_relation(
|
|
database=target.database,
|
|
schema='intermediate',
|
|
identifier='int_ofsted_latest'
|
|
) %}
|
|
|
|
select
|
|
s.urn,
|
|
s.local_authority_code * 1000 + s.establishment_number as laestab,
|
|
s.school_name,
|
|
-- Phase in GIAS code space (see seeds/gias_code_names.csv):
|
|
-- 2 = Primary, 4 = Secondary, 7 = All-through, 0 = Not applicable.
|
|
case
|
|
-- 1. Trust GIAS phase when it's a real value (0 = the catch-all "Not Applicable")
|
|
when s.phase_code is not null and s.phase_code != 0
|
|
then s.phase_code
|
|
-- 2. Infer from statutory age range (independent schools still publish these)
|
|
when s.statutory_high_age is not null and s.statutory_high_age <= 11 then 2
|
|
when s.statutory_low_age is not null and s.statutory_low_age >= 11 then 4
|
|
when s.statutory_low_age is not null and s.statutory_high_age is not null
|
|
and s.statutory_low_age < 11 and s.statutory_high_age > 11 then 7
|
|
-- 3. Fallback: infer from school name (covers independents with missing ages)
|
|
when s.school_name ilike '%primary%'
|
|
or s.school_name ilike '%infant%'
|
|
or s.school_name ilike '%junior%'
|
|
or s.school_name ilike '%preparatory%'
|
|
or s.school_name ilike '% prep school%'
|
|
or s.school_name ilike '% prep %'
|
|
then 2
|
|
when s.school_name ilike '%secondary%'
|
|
or s.school_name ilike '%high school%'
|
|
or s.school_name ilike '%grammar%'
|
|
or s.school_name ilike '%senior school%'
|
|
or s.school_name ilike '%upper school%'
|
|
then 4
|
|
-- 4. Give up — null renders no phase pill
|
|
else null
|
|
end as phase_code,
|
|
s.school_type_code,
|
|
s.academy_trust_name,
|
|
s.academy_trust_uid,
|
|
s.religious_character_code,
|
|
s.gender,
|
|
s.statutory_low_age || '-' || s.statutory_high_age as age_range,
|
|
-- GIAS OfficialSixthForm in code space: 1 = has, 2 = does not, 0 = N/A.
|
|
-- Null (rare, new establishments) falls back to the statutory age range.
|
|
case
|
|
when s.official_sixth_form_code = 1 then true
|
|
when s.official_sixth_form_code in (0, 2) then false
|
|
else coalesce(s.statutory_high_age >= 18, false)
|
|
end as has_sixth_form,
|
|
s.capacity,
|
|
s.total_pupils,
|
|
concat_ws(' ', s.head_title, s.head_first_name, s.head_last_name) as headteacher_name,
|
|
s.website,
|
|
s.telephone,
|
|
s.open_date,
|
|
s.close_date,
|
|
s.status_code,
|
|
s.nursery_provision,
|
|
s.admissions_policy_code,
|
|
|
|
-- Latest Ofsted (populated after monthly Ofsted pipeline runs)
|
|
{% if ofsted_relation is not none %}
|
|
-- Prefer the graded overall effectiveness; fall back to the grade parsed
|
|
-- from the latest ungraded (Section 8) outcome when no graded grade exists.
|
|
coalesce(o.overall_effectiveness, o.ungraded_grade) as ofsted_grade,
|
|
o.inspection_date as ofsted_date,
|
|
o.framework as ofsted_framework
|
|
{% else %}
|
|
null::text as ofsted_grade,
|
|
null::date as ofsted_date,
|
|
null::text as ofsted_framework
|
|
{% endif %}
|
|
|
|
from schools s
|
|
{% if ofsted_relation is not none %}
|
|
left join {{ ref('int_ofsted_latest') }} o on s.urn = o.urn
|
|
{% endif %}
|
|
-- 1 = Open; 3 = Open, but proposed to close (still operating; drops out when
|
|
-- GIAS flips to Closed — marts fully rebuild each run).
|
|
where s.status_code in (1, 3)
|
|
-- England only. dim_location must apply this filter identically (the API
|
|
-- inner-joins the two). See vars in dbt_project.yml for what the codes are.
|
|
-- coalesce, not a bare NOT IN: a null type code would make the predicate
|
|
-- null and drop the row silently. Unknown type is not grounds for exclusion.
|
|
and coalesce(s.school_type_code, -1) not in ({{ var('non_england_school_type_codes') | join(', ') }})
|