tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ees-tap): filter out rows with null URN before emitting
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m47s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details

Browse Source 
The admissions school-level file contains some rows with null school_urn
(LA/category aggregates that survive the geographic_level filter). These
cause a not-null constraint violation at target-postgres. Drop any row
where the URN column is null or empty before yielding records.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor
2026-03-27 10:13:17 +00:00
parent c7357336e3
commit 8e8d1bd8c5
1 changed files with 5 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
pipeline/plugins/extractors/tap-uk-ees/tap_uk_ees/tap.py
View File

@@ -95,6 +95,11 @@ class EESDatasetStream(Stream):
if "geographic_level" in df.columns: if "geographic_level" in df.columns:
df = df[df["geographic_level"] == "School"] df = df[df["geographic_level"] == "School"]
# Drop rows with no URN (LA/category aggregates that slip through the level filter)
urn_col = self._urn_column
if urn_col in df.columns:
df = df[df[urn_col].notna() & (df[urn_col] != "")]
self.logger.info("Emitting %d school-level rows", len(df)) self.logger.info("Emitting %d school-level rows", len(df))
for _, row in df.iterrows(): for _, row in df.iterrows():