tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
192 Commits 2 Branches 0 Tags
c576bba06a3c5b1e15408d274bac9b46a780b46f
Commit Graph

4 Commits

Author SHA1 Message Date
Tudor
c576bba06a fix(meltano): remove catalog capability and switch elt to run
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 34s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m26s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details 
The `catalog` capability forced Meltano to run --discover and generate
a catalog file (tap.properties.json) before each extraction. This fails
because our Singer SDK taps emit schemas inline and don't need external
catalog files. Removing the capability makes Meltano invoke taps
directly without catalog generation.

Also switch from deprecated `meltano elt` to `meltano run` for
Meltano 4.x compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 13:45:23 +00:00
Tudor
deb4024731 chore(pipeline): bump all dependencies to latest stable versions
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m45s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details 
- Airflow 2.11 → 3.1 (BashOperator moved to providers-standard)
- Meltano 3.5 → 4.1 (meltano.yml version 2, meltanolabs target-postgres)
- dbt-postgres 1.9 → 1.10
- singer-sdk 0.39 → 0.53 (all 6 taps)
- Typesense Docker 27.1 → 30.1
- Typesense Python client >=2.0
- Python base image 3.12 → 3.13

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 09:18:11 +00:00
Tudor
e32666ae4c fix(pipeline): bump Airflow to 2.11 and dbt to 1.9 to resolve SQLAlchemy conflict
Some checks failed
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m5s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Failing after 49s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Has been skipped
Details 
Airflow 2.10 requires SQLAlchemy <2.0, but dbt-postgres 1.8+ pulls in
SQLAlchemy 2.x. Airflow 2.11 supports SQLAlchemy 2.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 09:08:21 +00:00
Tudor
8f02b5125e feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details 
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.

- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-26 08:37:53 +00:00