feat(destinations): a tap that preserves the suppression sentinel
EES writes 'c' where a figure is withheld and the categories sum to the cohort, so counts and percentages are emitted as text with the sentinel intact. safe_numeric must never be pointed at them. School rows and the England reference need different establishment pins: at national level selective schools, studios and UTCs are separate populations rather than labels, so leaving establishment open multiplies 30 rows into 190. Two queries per period, each keeping its own level. Verified against the live API for 2022/23: 135,240 school records over 4,508 schools, exactly 30 each, no duplicate keys, 31,382 sentinels kept. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvdDKvFFSZuMVDH5fEyTob
This commit is contained in:
1 parent
7ae9ecdc36
commit
9188626051
13 files changed
+1467
-1
No files matched your search
@@ -178,9 +178,19 @@ with DAG(
|
||||
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-ees target-postgres",
|
||||
)
|
||||
|
||||
# Destinations come from the EES query API rather than a release ZIP,
|
||||
# so they are a separate tap. Run after extract_ees rather than beside
|
||||
# it: both write to the raw schema and the loader is happier serial.
|
||||
extract_ees_destinations = BashOperator(
|
||||
task_id="extract_ees_destinations",
|
||||
bash_command=f"cd {PIPELINE_DIR} && {MELTANO_BIN} run tap-uk-ees-destinations target-postgres",
|
||||
)
|
||||
|
||||
extract_ees >> extract_ees_destinations
|
||||
|
||||
dbt_build_ees = BashOperator(
|
||||
task_id="dbt_build",
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ees_ks2+ stg_legacy_ks2+ stg_ees_ks4+ stg_legacy_ks4+ stg_ees_census+ stg_ees_admissions+ stg_ees_ks2_national+ stg_ees_ks4_national+",
|
||||
bash_command=f"cd {PIPELINE_DIR}/transform && {DBT_BIN} build --profiles-dir . --target production --select stg_ees_ks2+ stg_legacy_ks2+ stg_ees_ks4+ stg_legacy_ks4+ stg_ees_census+ stg_ees_admissions+ stg_ees_ks2_national+ stg_ees_ks4_national+ stg_ees_ks4_destinations+ stg_ees_ks5_destinations+",
|
||||
)
|
||||
|
||||
sync_typesense_ees = BashOperator(
|
||||
|
||||
Reference in new issue
Block a user