feat(admissions): show the last distance offered where councils publish it

Adds the cut-off distance a parent actually asks about — "how close do we
need to live?" — end to end: a Singer tap, dbt staging and mart models, an
Airflow DAG, and a tile on both detail templates. 3,597 schools across 57
local authorities carry a figure; the rest are unchanged.

There is no national source for this. Each LA publishes its own cut-offs in
its own format, and the collected CSV is transcribed from PDFs, spreadsheets
and web pages — so most of the work here is deciding what is safe to show.

Data
  * tap-uk-school-distance loads the CSV verbatim into raw. Keyed on
    (urn, year, school_name), because school_name carries the admission
    route: (urn, year) alone collides on 118 keys and a reload would have
    silently dropped every band but one.
  * stg_school_distance applies a 25 m – 25 km plausibility band. The source
    contains 0.0-mile rows (published where a school filled on a higher
    criterion), 1-metre cut-offs, and one reading 533 miles — ~4% of rows,
    all of which would put a visibly wrong number on a live page.
  * fact_admission_distance collapses routes to one row per school per year
    using the furthest, and keeps route_count so the page can say the figure
    is the widest of several bands rather than the one for a given child.

Serving
  * Kept out of fact_admissions: that mart is EES-derived and near-complete
    for England, this one covers 57 LAs, and the two refresh independently.
  * Latest year only. Coverage is ragged — a school may have 2021 and 2026
    and nothing between — so a history array would invite a trend line drawn
    through gaps that are absences of publication, not of a cut-off.
  * The Admissions section now renders on either source. 3% of the schools
    that render have a cut-off and no EES admissions row, and gating on
    admissions alone would have hidden the figure on those pages.

Interface
  * The year travels with the figure everywhere it appears; a cut-off
    detached from its admissions round is not a fact about anything.
  * "Not a fixed catchment — it moves every year" sits under every instance,
    because that is the inference a parent will otherwise draw.
  * Replaces a hardcoded "Historical distance cut-off data is not available
    for this school" that appeared on every secondary page, including the
    ones whose council does publish it. The absence is now stated only when
    it is real, and names the authority that would hold it.

The tint costs the muted tokens their AA margin: measured on the composited
backdrop (not the computed one, which reports the untinted card), --text-muted
falls to 4.09:1 in dark theme. The tile uses --text-secondary instead — 6.50:1
dark, 6.60:1 light.

The DAG is manual, like the other annual ones: councils publish on allocation
day, each on its own timetable, so there is no date worth scheduling against.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WDvkyqqHABm4bmth2kjAxE

pipeline/dags/school_data_pipeline.py

@@ -224,3 +224,68 @@ with DAG(
     )
 
     extract_idaci >> dbt_build_idaci >> invalidate_cache_idaci
+
+
+# ── Annual DAG (Last distance offered) ────────────────────────────────
+
+with DAG(
+    dag_id="school_data_annual_distance",
+    default_args=default_args,
+    description="Last distance offered (LA admission cut-offs) extraction and transform",
+    # Councils publish on allocation day (March for secondary, April for
+    # primary) and each one on its own timetable, so there is no date worth
+    # scheduling against. Triggered manually after a collection run refreshes
+    # the CSV.
+    schedule=None,
+    start_date=datetime(2025, 1, 1),
+    catchup=False,
+    tags=["school-compare", "annual"],
+) as annual_distance_dag:
+
+    extract_distance = BashOperator(
+        task_id="extract_distance",
+        bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-school-distance target-postgres",
+    )
+
+    # Coverage is the thing that silently rots here: the CSV is assembled by
+    # hand from council publications, so a collection run that half-failed
+    # produces a valid file with a fraction of the schools in it. A row count
+    # alone would not catch that — losing an entire local authority leaves the
+    # total looking healthy — so the floor is checked on distinct LAs too.
+    validate_distance = BashOperator(
+        task_id="validate_distance",
+        bash_command=f"""
+            cd {PIPELINE_DIR} && python -c "
+import psycopg2, os, sys
+conn = psycopg2.connect(
+    host=os.environ.get('PG_HOST', 'localhost'),
+    port=os.environ.get('PG_PORT', '5432'),
+    user=os.environ.get('PG_USER', 'postgres'),
+    password=os.environ.get('PG_PASSWORD', 'postgres'),
+    dbname=os.environ.get('PG_DATABASE', 'school_compare'),
+)
+cur = conn.cursor()
+cur.execute('SELECT count(*), count(distinct urn), count(distinct la_code) FROM raw.school_distance_offered')
+rows, urns, las = cur.fetchone()
+conn.close()
+print(f'Loaded {{rows}} rows, {{urns}} schools, {{las}} local authorities')
+if rows < 7000 or urns < 3000 or las < 45:
+    print('ERROR: distance extract is short of expected coverage '
+          '(baseline 2026-08: 9128 rows / 3726 schools / 57 LAs)', file=sys.stderr)
+    sys.exit(1)
+print('Validation passed')
+"
+        """,
+    )
+
+    dbt_build_distance = BashOperator(
+        task_id="dbt_build",
+        bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_school_distance+ fact_admission_distance+",
+    )
+
+    invalidate_cache_distance = BashOperator(
+        task_id="invalidate_cache",
+        bash_command=INVALIDATE_CACHE_CMD,
+    )
+
+    extract_distance >> validate_distance >> dbt_build_distance >> invalidate_cache_distance

pipeline/meltano.yml

@@ -67,6 +67,21 @@ plugins:
       pip_url: ./plugins/extractors/tap-uk-idaci
       executable: tap-uk-idaci
 
+    # Last distance offered — admission cut-offs collected from LA
+    # publications. There is no national source for this, so the URL points at
+    # the collected CSV rather than at a government endpoint (same pattern as
+    # the legacy KS2/KS4 URLs above).
+    - name: tap-uk-school-distance
+      namespace: uk_school_distance
+      pip_url: ./plugins/extractors/tap-uk-school-distance
+      executable: tap-uk-school-distance
+      settings:
+        - name: download_url
+          kind: string
+          description: URL of the collected last-distance-offered CSV
+      config:
+        download_url: http://10.0.1.224:8081/filebrowser/api/public/dl/UU_oEqVr?inline=true
+
   loaders:
     - name: target-postgres
       variant: meltanolabs

pipeline/plugins/extractors/tap-uk-school-distance/pyproject.toml

@@ -0,0 +1,16 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "tap-uk-school-distance"
+version = "0.1.0"
+description = "Singer tap for last-distance-offered admission cut-offs collected from LA admissions publications"
+requires-python = ">=3.10"
+dependencies = [
+    "singer-sdk~=0.53",
+    "requests>=2.31",
+]
+
+[project.scripts]
+tap-uk-school-distance = "tap_uk_school_distance.tap:TapUKSchoolDistance.cli"

pipeline/plugins/extractors/tap-uk-school-distance/tap_uk_school_distance/__init__.py

@@ -0,0 +1 @@
+"""tap-uk-school-distance: Singer tap for LA "last distance offered" admission cut-offs."""

pipeline/plugins/extractors/tap-uk-school-distance/tap_uk_school_distance/tap.py

@@ -0,0 +1,135 @@
+"""Singer tap for "last distance offered" school admission cut-offs.
+
+Unlike every other tap in this project, the upstream here is not a single
+government endpoint. There is no national dataset of admission cut-off
+distances: each local authority publishes its own, in its own format
+(PDF booklets, XLSX allocation tables, HTML pages), and only some publish
+at all. The CSV this tap reads is the output of that collection work —
+one row per school per year per admission route, already normalised to
+metres in `distance_m`.
+
+The tap deliberately does no cleaning. Implausible values (0.0 miles,
+1-metre cut-offs, a 533-mile outlier) are loaded verbatim into `raw` and
+filtered in `stg_school_distance`, so the raw table stays a faithful
+record of what the councils published and the plausibility rules live in
+one reviewable place.
+"""
+
+from __future__ import annotations
+
+import csv
+import io
+
+import requests
+from singer_sdk import Stream, Tap
+from singer_sdk import typing as th
+
+REQUEST_TIMEOUT = 300
+
+
+class SchoolDistanceStream(Stream):
+    """Stream: last distance offered, one row per school × year × admission route."""
+
+    name = "school_distance_offered"
+    # school_name is part of the key, not decoration: where a school admits
+    # through several routes the route is encoded in the name ("… Band 1",
+    # "… Band 2"), and (urn, year) alone is not unique. Verified against the
+    # source: (urn, year) collides on 118 keys, (urn, year, school_name) on
+    # none. Without school_name in the key a reload would silently drop the
+    # other bands.
+    primary_keys = ["urn", "year", "school_name"]
+    replication_key = None
+
+    schema = th.PropertiesList(
+        th.Property("urn", th.IntegerType, required=True),
+        th.Property("year", th.IntegerType, required=True),
+        th.Property("school_name", th.StringType, required=True),
+        th.Property("la_code", th.IntegerType),
+        th.Property("la_name", th.StringType),
+        # The figure as the council published it, before unit conversion —
+        # kept so a surprising metre value can be traced back to "0.31 miles"
+        # without reopening the source PDF.
+        th.Property("distance_value_raw", th.NumberType),
+        th.Property("distance_unit_raw", th.StringType),
+        th.Property("distance_m", th.NumberType),
+        # Path of the council publication the row was read from. This is the
+        # provenance trail for a figure we cannot re-derive from an API.
+        th.Property("source_file", th.StringType),
+    ).to_dict()
+
+    @staticmethod
+    def _num(value: str) -> float | None:
+        try:
+            return float(value)
+        except (TypeError, ValueError):
+            return None
+
+    @staticmethod
+    def _int(value: str) -> int | None:
+        try:
+            return int(float(value))
+        except (TypeError, ValueError):
+            return None
+
+    def get_records(self, context):
+        url = self.config["download_url"]
+        self.logger.info("Downloading school distance data from %s", url)
+
+        resp = requests.get(url, timeout=REQUEST_TIMEOUT)
+        resp.raise_for_status()
+
+        reader = csv.DictReader(io.StringIO(resp.text))
+
+        yielded = 0
+        skipped = 0
+        for row in reader:
+            urn = self._int(row.get("urn"))
+            year = self._int(row.get("year"))
+            name = (row.get("school_name") or "").strip()
+
+            # Every part of the primary key must be present, or the load is
+            # not idempotent. One row in the source has a blank year.
+            if urn is None or year is None or not name:
+                skipped += 1
+                continue
+
+            yield {
+                "urn": urn,
+                "year": year,
+                "school_name": name,
+                "la_code": self._int(row.get("la_code")),
+                "la_name": (row.get("la_name") or "").strip() or None,
+                "distance_value_raw": self._num(row.get("distance_value_raw")),
+                "distance_unit_raw": (row.get("distance_unit_raw") or "").strip() or None,
+                "distance_m": self._num(row.get("distance_m")),
+                "source_file": (row.get("source_file") or "").strip() or None,
+            }
+            yielded += 1
+
+        self.logger.info(
+            "Yielded %d distance records (%d skipped: missing urn, year or school name)",
+            yielded,
+            skipped,
+        )
+
+
+class TapUKSchoolDistance(Tap):
+    """Singer tap for LA-published admission cut-off distances."""
+
+    name = "tap-uk-school-distance"
+
+    config_jsonschema = th.PropertiesList(
+        th.Property(
+            "download_url",
+            th.StringType,
+            required=True,
+            description="URL of the collected last-distance-offered CSV",
+        ),
+    ).to_dict()
+
+    def discover_streams(self):
+        return [SchoolDistanceStream(self)]
+
+
+if __name__ == "__main__":
+    TapUKSchoolDistance.cli()

pipeline/transform/models/marts/_marts_schema.yml

@@ -156,6 +156,22 @@ models:
       - name: cross_la_offers
       - name: total_offers
 
+  - name: fact_admission_distance
+    description: >
+      Last distance offered — one row per URN per year. Where a school admits
+      through several routes (ability bands, separate reception/junior
+      intakes) the figure is the furthest of them and route_count says how
+      many were collapsed.
+    columns:
+      - name: urn
+        tests: [not_null]
+      - name: year
+        tests: [not_null]
+      - name: distance_m
+        tests: [not_null]
+      - name: route_count
+        tests: [not_null]
+
   - name: fact_finance
     description: School financial data — one row per URN per year
     columns:

pipeline/transform/models/marts/fact_admission_distance.sql

@@ -0,0 +1,50 @@
+-- Mart: last distance offered — one row per URN per year.
+--
+-- Collapses the per-route grain of stg_school_distance to the grain the site
+-- serves. Where a school admits through several routes the figure shown is the
+-- FURTHEST of them, which is what "last distance offered" literally means: the
+-- distance of the last place the school gave out that year.
+--
+-- route_count travels with it so the page can say so. A parent looking at a
+-- banded secondary needs to know that 2.4 miles is the widest of four bands
+-- and not the number that applies to their own child — the count is what makes
+-- that sentence possible, and dropping it would make the figure quietly
+-- optimistic.
+--
+-- Kept separate from fact_admissions rather than joined into it. That mart is
+-- EES-derived and near-complete for England; this one comes from 57 local
+-- authorities that publish cut-offs at all, so folding it in would add mostly
+-- null columns to every school in the country and tie two very different
+-- refresh cadences together.
+
+with routes as (
+    select * from {{ ref('stg_school_distance') }}
+),
+
+per_year as (
+    select
+        urn,
+        year,
+        max(distance_m)             as distance_m,
+        count(*)                    as route_count,
+        min(la_code)                as la_code,
+        min(la_name)                as la_name,
+        -- The unit the council actually published in. Where routes disagree
+        -- (they do not in the current data) the alphabetically first is taken
+        -- rather than an arbitrary row, so the mart is deterministic.
+        min(distance_unit_raw)      as distance_unit_raw,
+        min(source_file)            as source_file
+    from routes
+    group by urn, year
+)
+
+select
+    urn,
+    year,
+    distance_m,
+    route_count,
+    la_code,
+    la_name,
+    distance_unit_raw,
+    source_file
+from per_year

pipeline/transform/models/staging/_stg_sources.yml

@@ -61,3 +61,11 @@ sources:
 
       - name: idaci
         description: Income Deprivation Affecting Children Index lookups
+
+      - name: school_distance_offered
+        description: >
+          Last distance offered (admission cut-offs) transcribed from local
+          authority admissions publications. One row per school × year ×
+          admission route; no national source exists, so coverage is limited to
+          the LAs that publish. Loaded verbatim — implausible values are
+          filtered in stg_school_distance, not here.

pipeline/transform/models/staging/stg_school_distance.sql

@@ -0,0 +1,55 @@
+-- Staging model: "last distance offered" admission cut-offs.
+--
+-- One row per school × year × admission route. A school admits through
+-- several routes when it bands by ability (Islington publishes four bands per
+-- secondary) or takes more than one intake (Gloucestershire publishes
+-- reception, junior and secondary allocations separately). The route is
+-- carried in school_name, which is why it forms part of the grain here and is
+-- collapsed only in fact_admission_distance.
+--
+-- PLAUSIBILITY BAND (25 m – 25 km)
+--
+-- The source is transcribed from council PDFs, spreadsheets and web pages, and
+-- a minority of rows are transcription or unit errors rather than real
+-- cut-offs. Three failure modes, all present:
+--
+--   * 0.0 miles (34 rows, mostly Staffordshire). Published where a school
+--     filled on a higher criterion so distance was never reached — a real
+--     fact, but "0 metres" read as a cut-off would tell a parent the opposite
+--     of the truth.
+--   * 1–10 metre values (72 rows, e.g. Hillingdon). A cut-off inside the
+--     school gate is not physically possible; these are unit slips.
+--   * Values above 25 km (~240 rows), topping out at 533 miles. Genuine rural
+--     secondary cut-offs reach roughly 12–15 miles, so the ceiling keeps those
+--     while cutting figures that are visibly wrong.
+--
+-- Together these are ~4% of rows. They are loaded verbatim into raw and
+-- excluded here, so the rule is reviewable in one place and the raw table
+-- stays a faithful record of what each council published.
+
+{% set min_plausible_m = 25 %}
+{% set max_plausible_m = 25000 %}
+
+with source as (
+    select * from {{ source('raw', 'school_distance_offered') }}
+),
+
+renamed as (
+    select
+        cast(urn as integer)                as urn,
+        cast(year as integer)               as year,
+        trim(school_name)                   as admission_route,
+        cast(la_code as integer)            as la_code,
+        la_name,
+        cast(distance_value_raw as numeric) as distance_value_raw,
+        distance_unit_raw,
+        cast(distance_m as numeric)         as distance_m,
+        source_file
+    from source
+    where urn is not null
+      and year is not null
+      and distance_m is not null
+      and cast(distance_m as numeric) between {{ min_plausible_m }} and {{ max_plausible_m }}
+)
+
+select * from renamed

pipeline/transform/tests/assert_admission_distance_plausible.sql

@@ -0,0 +1,27 @@
+-- Custom test: every published cut-off distance is inside the plausibility
+-- band, and there is exactly one row per school per year.
+--
+-- The band is enforced in stg_school_distance; this asserts it survived the
+-- max() collapse in the mart. A failure means either the staging filter was
+-- edited without updating this test, or the grain broke — both of which put a
+-- visibly wrong number ("533 miles") on a live school page, which is the one
+-- outcome this feature cannot afford.
+
+with out_of_band as (
+    select urn, year, distance_m, 'out of plausibility band' as problem
+    from {{ ref('fact_admission_distance') }}
+    where distance_m is null
+       or distance_m < 25
+       or distance_m > 25000
+),
+
+duplicated as (
+    select urn, year, max(distance_m) as distance_m, 'duplicate urn/year' as problem
+    from {{ ref('fact_admission_distance') }}
+    group by urn, year
+    having count(*) > 1
+)
+
+select * from out_of_band
+union all
+select * from duplicated