tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(airflow): share logs volume between scheduler and api-server
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Details
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details

Browse Source 
The api-server couldn't fetch task logs because LocalExecutor runs tasks
in the scheduler process, writing logs to its local filesystem. The
api-server tried to fetch via HTTP but the scheduler's log server had
no hostname set. Fix by sharing a named volume for logs between both
containers so the api-server reads logs directly from the filesystem.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor
2026-03-26 11:55:43 +00:00
parent cd75fc4c24
commit 6b4eb08a5e
1 changed files with 5 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docker-compose.portainer.yml
View File

@@ -197,6 +197,7 @@ services:
AIRFLOW__CORE__DAGS_FOLDER: /opt/pipeline/dags AIRFLOW__CORE__DAGS_FOLDER: /opt/pipeline/dags
AIRFLOW__CORE__LOAD_EXAMPLES: "false" AIRFLOW__CORE__LOAD_EXAMPLES: "false"
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS: "${AIRFLOW_ADMIN_USER:-admin}:admin" AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS: "${AIRFLOW_ADMIN_USER:-admin}:admin"
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
PG_HOST: sc_database PG_HOST: sc_database
PG_PORT: "5432" PG_PORT: "5432"
PG_USER: ${DB_USERNAME} PG_USER: ${DB_USERNAME}
@@ -205,7 +206,7 @@ services:
TYPESENSE_URL: http://typesense:8108 TYPESENSE_URL: http://typesense:8108
TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme} TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme}
volumes: volumes:
- airflow_logs:/opt/airflow/logs
depends_on: depends_on:
sc_database: sc_database:
condition: service_healthy condition: service_healthy
@@ -230,6 +231,7 @@ services:
AIRFLOW__CORE__DAGS_FOLDER: /opt/pipeline/dags AIRFLOW__CORE__DAGS_FOLDER: /opt/pipeline/dags
AIRFLOW__CORE__LOAD_EXAMPLES: "false" AIRFLOW__CORE__LOAD_EXAMPLES: "false"
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS: "${AIRFLOW_ADMIN_USER:-admin}:admin" AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS: "${AIRFLOW_ADMIN_USER:-admin}:admin"
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
PG_HOST: sc_database PG_HOST: sc_database
PG_PORT: "5432" PG_PORT: "5432"
PG_USER: ${DB_USERNAME} PG_USER: ${DB_USERNAME}
@@ -238,7 +240,7 @@ services:
TYPESENSE_URL: http://typesense:8108 TYPESENSE_URL: http://typesense:8108
TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme} TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme}
volumes: volumes:
- airflow_logs:/opt/airflow/logs
depends_on: depends_on:
sc_database: sc_database:
condition: service_healthy condition: service_healthy
@@ -275,3 +277,4 @@ volumes:
kestra_storage: kestra_storage:
supplementary_data: supplementary_data:
typesense_data: typesense_data:
airflow_logs: