fix(airflow): share logs volume between scheduler and api-server
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s

The api-server couldn't fetch task logs because LocalExecutor runs tasks
in the scheduler process, writing logs to its local filesystem. The
api-server tried to fetch via HTTP but the scheduler's log server had
no hostname set. Fix by sharing a named volume for logs between both
containers so the api-server reads logs directly from the filesystem.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-26 11:55:43 +00:00
parent cd75fc4c24
commit 6b4eb08a5e

View File

@@ -197,6 +197,7 @@ services:
AIRFLOW__CORE__DAGS_FOLDER: /opt/pipeline/dags
AIRFLOW__CORE__LOAD_EXAMPLES: "false"
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS: "${AIRFLOW_ADMIN_USER:-admin}:admin"
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
PG_HOST: sc_database
PG_PORT: "5432"
PG_USER: ${DB_USERNAME}
@@ -205,7 +206,7 @@ services:
TYPESENSE_URL: http://typesense:8108
TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme}
volumes:
- airflow_logs:/opt/airflow/logs
depends_on:
sc_database:
condition: service_healthy
@@ -230,6 +231,7 @@ services:
AIRFLOW__CORE__DAGS_FOLDER: /opt/pipeline/dags
AIRFLOW__CORE__LOAD_EXAMPLES: "false"
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS: "${AIRFLOW_ADMIN_USER:-admin}:admin"
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
PG_HOST: sc_database
PG_PORT: "5432"
PG_USER: ${DB_USERNAME}
@@ -238,7 +240,7 @@ services:
TYPESENSE_URL: http://typesense:8108
TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme}
volumes:
- airflow_logs:/opt/airflow/logs
depends_on:
sc_database:
condition: service_healthy
@@ -275,3 +277,4 @@ volumes:
kestra_storage:
supplementary_data:
typesense_data:
airflow_logs: