fix(destinations): school rows and the England reference are different grains
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m4s
PR Checks / Backend Smoke (pull_request) Successful in 9s
PR Checks / Build Backend (no push) (pull_request) Successful in 11s
PR Checks / Build Frontend (no push) (pull_request) Successful in 45s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 52s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m27s
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m4s
PR Checks / Backend Smoke (pull_request) Successful in 9s
PR Checks / Build Backend (no push) (pull_request) Successful in 11s
PR Checks / Build Frontend (no push) (pull_request) Successful in 45s
PR Checks / Build Pipeline (no push) (pull_request) Successful in 52s
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m27s
The annual DAG died with a BrokenPipeError from Meltano's log writer, which is several frames from the cause: target-postgres exited first and the tap saw its stdout close. The tap declared primary_keys = [urn, ...] while emitting urn=None for the national rows, and target-postgres turns primary_keys into a NOT NULL constraint. The first national row of the run failed the insert and took the loader with it. Every other tap in this repo keys on non-null columns. Carrying two grains in one stream was the actual mistake, so the fix is to separate them rather than paper over the null: four streams now, with ees_ks4/ks5_destinations_national carrying no urn column at all — a school identifier that is null in every row is a grain mismatch, not a column. The staging models split the same way and the national mart reads the new pair instead of filtering `where urn is null`. Verified against the live API: the school stream yields 135,240 rows over 4,508 schools with no duplicate keys, no null key columns and all 31,382 suppression sentinels intact; the national streams yield 30 and 33 rows with no urn column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob
This commit is contained in:
1 parent
fb5a0928bd
commit
e236669fde
9 files changed
+242
-34
No files matched your search
@@ -132,3 +132,68 @@ def test_establishment_dimensions_are_not_pinned():
|
||||
establishment_totals = {"4369U", "rgHcN", "EfHQq", "S4ROV"}
|
||||
assert not set(KS4_PINNED) & establishment_totals
|
||||
assert not set(KS5_PINNED) & establishment_totals
|
||||
|
||||
|
||||
# ── Grain separation ────────────────────────────────────────────────────────
|
||||
#
|
||||
# School rows and the England reference were originally one stream with a
|
||||
# nullable `urn` in the primary key. target-postgres turns primary_keys into a
|
||||
# NOT NULL constraint, so the first national row killed the loader mid-run and
|
||||
# the tap saw only a BrokenPipeError on its stdout — a symptom several frames
|
||||
# away from the cause.
|
||||
|
||||
def _streams():
|
||||
from tap_uk_ees_destinations.tap import TapUKEESDestinations
|
||||
return TapUKEESDestinations(config={}, validate_config=False).discover_streams()
|
||||
|
||||
|
||||
def test_no_stream_has_a_nullable_primary_key_column():
|
||||
for stream in _streams():
|
||||
props = stream.schema["properties"]
|
||||
for key in stream.primary_keys:
|
||||
assert key in props, f"{stream.name}: key {key} is not in the schema"
|
||||
if "urn" in stream.primary_keys:
|
||||
assert stream._level == "SCH", (
|
||||
f"{stream.name} keys on urn but does not emit school rows"
|
||||
)
|
||||
|
||||
|
||||
def test_national_streams_carry_no_urn_column_at_all():
|
||||
for stream in _streams():
|
||||
if not stream.name.endswith("_national"):
|
||||
continue
|
||||
assert "urn" not in stream.schema["properties"], (
|
||||
"a school identifier that is null in every row is a grain "
|
||||
"mismatch, not a column"
|
||||
)
|
||||
assert "urn" not in stream.primary_keys
|
||||
|
||||
|
||||
def test_national_post_process_drops_the_null_urn():
|
||||
from tap_uk_ees_destinations.tap import KS4NationalDestinationsStream, TapUKEESDestinations
|
||||
tap = TapUKEESDestinations(config={}, validate_config=False)
|
||||
stream = KS4NationalDestinationsStream(tap)
|
||||
row = {"urn": None, "time_period": "202223", "pupil_group": "all",
|
||||
"destination_measure": "school_sixth_form", "cohort_pupils": "1",
|
||||
"pupils_raw": "1", "percentage_raw": "1"}
|
||||
assert "urn" not in stream.post_process(dict(row))
|
||||
|
||||
|
||||
def test_school_and_national_streams_exist_for_both_phases():
|
||||
names = {s.name for s in _streams()}
|
||||
assert names == {
|
||||
"ees_ks4_destinations", "ees_ks5_destinations",
|
||||
"ees_ks4_destinations_national", "ees_ks5_destinations_national",
|
||||
}
|
||||
|
||||
|
||||
def test_each_stream_queries_its_own_geographic_level_with_its_own_pins():
|
||||
"""The national pass needs establishment pinned to Total; the school pass
|
||||
must not pin it at all, or every school returns zero rows."""
|
||||
for stream in _streams():
|
||||
if stream.name.endswith("_national"):
|
||||
assert stream._level == "NAT"
|
||||
assert stream._pinned_for_level is stream._national_pinned
|
||||
else:
|
||||
assert stream._level == "SCH"
|
||||
assert stream._pinned_for_level is stream._pinned
|
||||
Reference in new issue
Block a user