tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
188 Commits 2 Branches 0 Tags
2ba5e572863741100cc7d0d845d7fd6d9c6fcb59
Commit Graph

10 Commits

Author SHA1 Message Date
Tudor
2ba5e57286 fix(airflow): set scheduler hostname for log server resolution
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m11s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
The scheduler's log server binds to [::]:8793 but doesn't advertise a
hostname, so the api-server gets 'http://:8793/...' (no host) when
fetching task logs. Fix by setting the scheduler's hostname and
configuring WORKER_LOG_SERVER_HOST so the api-server can reach it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 12:06:22 +00:00
Tudor
6b4eb08a5e fix(airflow): share logs volume between scheduler and api-server
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
The api-server couldn't fetch task logs because LocalExecutor runs tasks
in the scheduler process, writing logs to its local filesystem. The
api-server tried to fetch via HTTP but the scheduler's log server had
no hostname set. Fix by sharing a named volume for logs between both
containers so the api-server reads logs directly from the filesystem.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 11:55:43 +00:00
Tudor
b6a487776b fix(airflow): set DAGS_FOLDER in image env and reserialize on init
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details 
- Add AIRFLOW__CORE__DAGS_FOLDER env var in Dockerfile so it's always set
- Run `airflow dags reserialize` after `db migrate` in init container so
  DAGs appear immediately without waiting for scheduler scan interval

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 11:05:41 +00:00
Tudor
904093ea8a fix(airflow): remove DAG volume mounts, use image-baked DAGs
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 32s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
The named volume was shadowing the DAGs built into the pipeline image
with an empty directory. DAGs now served directly from the image and
update on each CI build.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 10:27:39 +00:00
Tudor
c4e3b6a7e4 fix(typesense): use TCP check for healthcheck, no curl/wget available
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
Typesense image has neither curl nor wget. Use bash /dev/tcp for a
simple port connectivity check instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 10:14:59 +00:00
Tudor
09d704c325 fix(typesense): use wget instead of curl for healthcheck
Some checks failed
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 33s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Details
Build and Push Docker Images / Build Kestra Init (push) Has been cancelled
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Has been cancelled
Details
Build and Push Docker Images / Trigger Portainer Update (push) Has been cancelled
Details
Build and Push Docker Images / Build Integrator (push) Has been cancelled
Details 
Typesense Docker image ships with wget but not curl.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 10:12:54 +00:00
Tudor
1574089b95 fix(pipeline): update Airflow healthcheck to /api/v2/monitor/health
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
Airflow 3 moved the health endpoint from /health to /api/v2/monitor/health.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 10:01:09 +00:00
Tudor
a7904b627d fix(pipeline): migrate to Airflow 3 API server and SimpleAuthManager
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m12s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 58s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
Airflow 3 replaced `airflow webserver` with `airflow api-server` and
removed the `airflow users` CLI. Auth is now via SimpleAuthManager
configured through AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS env var.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 09:32:08 +00:00
Tudor
deb4024731 chore(pipeline): bump all dependencies to latest stable versions
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m45s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details 
- Airflow 2.11 → 3.1 (BashOperator moved to providers-standard)
- Meltano 3.5 → 4.1 (meltano.yml version 2, meltanolabs target-postgres)
- dbt-postgres 1.9 → 1.10
- singer-sdk 0.39 → 0.53 (all 6 taps)
- Typesense Docker 27.1 → 30.1
- Typesense Python client >=2.0
- Python base image 3.12 → 3.13

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 09:18:11 +00:00
Tudor
8f02b5125e feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.

- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 08:37:53 +00:00