tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(pipeline): bump all dependencies to latest stable versions
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m4s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 57s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 1m45s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 0s
Details

Browse Source 
- Airflow 2.11 → 3.1 (BashOperator moved to providers-standard)
- Meltano 3.5 → 4.1 (meltano.yml version 2, meltanolabs target-postgres)
- dbt-postgres 1.9 → 1.10
- singer-sdk 0.39 → 0.53 (all 6 taps)
- Typesense Docker 27.1 → 30.1
- Typesense Python client >=2.0
- Python base image 3.12 → 3.13

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor
2026-03-26 09:18:11 +00:00
parent e32666ae4c
commit deb4024731
12 changed files with 19 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docker-compose.portainer.yml
View File

@@ -86,7 +86,7 @@ services:
# ── Typesense Search Engine ─────────────────────────────────────────── # ── Typesense Search Engine ───────────────────────────────────────────
typesense: typesense:
image: typesense/typesense:27.1 image: typesense/typesense:30.1
container_name: schoolcompare_typesense container_name: schoolcompare_typesense
environment: environment:
TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme} TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme}

2
docker-compose.yml
View File

@@ -77,7 +77,7 @@ services:
# Typesense — search engine # Typesense — search engine
typesense: typesense:
image: typesense/typesense:27.1 image: typesense/typesense:30.1
container_name: schoolcompare_typesense container_name: schoolcompare_typesense
ports: ports:
- "8108:8108" - "8108:8108"

2
pipeline/Dockerfile
View File

@@ -1,4 +1,4 @@
FROM python:3.12-slim FROM python:3.13-slim
WORKDIR /opt/pipeline WORKDIR /opt/pipeline

3
pipeline/dags/school_data_pipeline.py
View File

@@ -15,8 +15,7 @@ from __future__ import annotations
from datetime import datetime, timedelta from datetime import datetime, timedelta
from airflow import DAG from airflow import DAG
from airflow.operators.bash import BashOperator from airflow.providers.standard.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from airflow.utils.task_group import TaskGroup from airflow.utils.task_group import TaskGroup
PIPELINE_DIR = "/opt/pipeline" PIPELINE_DIR = "/opt/pipeline"

8
pipeline/meltano.yml
View File

@@ -1,4 +1,4 @@
version: 1 version: 2
project_id: school-compare-pipeline project_id: school-compare-pipeline
plugins: plugins:
@@ -71,8 +71,8 @@ plugins:
loaders: loaders:
- name: target-postgres - name: target-postgres
variant: transferwise variant: meltanolabs
pip_url: pipelinewise-target-postgres pip_url: meltanolabs-target-postgres
config: config:
host: $PG_HOST host: $PG_HOST
port: $PG_PORT port: $PG_PORT
@@ -84,7 +84,7 @@ plugins:
utilities: utilities:
- name: dbt-postgres - name: dbt-postgres
variant: dbt-labs variant: dbt-labs
pip_url: dbt-postgres~=1.9 pip_url: dbt-postgres~=1.10
config: config:
project_dir: $MELTANO_PROJECT_ROOT/transform project_dir: $MELTANO_PROJECT_ROOT/transform
profiles_dir: $MELTANO_PROJECT_ROOT/transform profiles_dir: $MELTANO_PROJECT_ROOT/transform

2
pipeline/plugins/extractors/tap-uk-ees/pyproject.toml
View File

@@ -8,7 +8,7 @@ version = "0.1.0"
description = "Singer tap for UK Explore Education Statistics (KS2, KS4, Census, Admissions, Phonics)" description = "Singer tap for UK Explore Education Statistics (KS2, KS4, Census, Admissions, Phonics)"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"singer-sdk~=0.39", "singer-sdk~=0.53",
"requests>=2.31", "requests>=2.31",
"pandas>=2.0", "pandas>=2.0",
] ]

2
pipeline/plugins/extractors/tap-uk-fbit/pyproject.toml
View File

@@ -8,7 +8,7 @@ version = "0.1.0"
description = "Singer tap for UK FBIT (Financial Benchmarking and Insights Tool)" description = "Singer tap for UK FBIT (Financial Benchmarking and Insights Tool)"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"singer-sdk~=0.39", "singer-sdk~=0.53",
"requests>=2.31", "requests>=2.31",
] ]

2
pipeline/plugins/extractors/tap-uk-gias/pyproject.toml
View File

@@ -8,7 +8,7 @@ version = "0.1.0"
description = "Singer tap for UK GIAS (Get Information About Schools) bulk data" description = "Singer tap for UK GIAS (Get Information About Schools) bulk data"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"singer-sdk~=0.39", "singer-sdk~=0.53",
"requests>=2.31", "requests>=2.31",
"pandas>=2.0", "pandas>=2.0",
] ]

2
pipeline/plugins/extractors/tap-uk-idaci/pyproject.toml
View File

@@ -8,7 +8,7 @@ version = "0.1.0"
description = "Singer tap for UK IDACI deprivation index" description = "Singer tap for UK IDACI deprivation index"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"singer-sdk~=0.39", "singer-sdk~=0.53",
"requests>=2.31", "requests>=2.31",
"pandas>=2.0", "pandas>=2.0",
] ]

2
pipeline/plugins/extractors/tap-uk-ofsted/pyproject.toml
View File

@@ -8,7 +8,7 @@ version = "0.1.0"
description = "Singer tap for UK Ofsted Management Information" description = "Singer tap for UK Ofsted Management Information"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"singer-sdk~=0.39", "singer-sdk~=0.53",
"requests>=2.31", "requests>=2.31",
"pandas>=2.0", "pandas>=2.0",
"odfpy>=1.4", "odfpy>=1.4",

2
pipeline/plugins/extractors/tap-uk-parent-view/pyproject.toml
View File

@@ -8,7 +8,7 @@ version = "0.1.0"
description = "Singer tap for UK Ofsted Parent View survey data" description = "Singer tap for UK Ofsted Parent View survey data"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
"singer-sdk~=0.39", "singer-sdk~=0.53",
"requests>=2.31", "requests>=2.31",
"pandas>=2.0", "pandas>=2.0",
"openpyxl>=3.1", "openpyxl>=3.1",

9
pipeline/requirements.txt
View File

@@ -1,9 +1,10 @@
# Pipeline dependencies # Pipeline dependencies
meltano==3.5.* meltano==4.1.*
dbt-postgres~=1.9 dbt-postgres~=1.10
apache-airflow==2.11.* apache-airflow==3.1.*
apache-airflow-providers-standard>=1.0
apache-airflow-providers-postgres>=5.0 apache-airflow-providers-postgres>=5.0
typesense>=0.21 typesense>=2.0,<3
requests>=2.31 requests>=2.31
openpyxl>=3.1 openpyxl>=3.1
odfpy>=1.4 odfpy>=1.4