fix(airflow): share logs volume between scheduler and api-server
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 32s
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m10s
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Build and Push Docker Images / Build Kestra Init (push) Successful in 31s
Build and Push Docker Images / Build Pipeline (Meltano + dbt + Airflow) (push) Successful in 31s
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
The api-server couldn't fetch task logs because LocalExecutor runs tasks in the scheduler process, writing logs to its local filesystem. The api-server tried to fetch via HTTP but the scheduler's log server had no hostname set. Fix by sharing a named volume for logs between both containers so the api-server reads logs directly from the filesystem. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -197,6 +197,7 @@ services:
|
||||
AIRFLOW__CORE__DAGS_FOLDER: /opt/pipeline/dags
|
||||
AIRFLOW__CORE__LOAD_EXAMPLES: "false"
|
||||
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS: "${AIRFLOW_ADMIN_USER:-admin}:admin"
|
||||
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
|
||||
PG_HOST: sc_database
|
||||
PG_PORT: "5432"
|
||||
PG_USER: ${DB_USERNAME}
|
||||
@@ -205,7 +206,7 @@ services:
|
||||
TYPESENSE_URL: http://typesense:8108
|
||||
TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme}
|
||||
volumes:
|
||||
|
||||
- airflow_logs:/opt/airflow/logs
|
||||
depends_on:
|
||||
sc_database:
|
||||
condition: service_healthy
|
||||
@@ -230,6 +231,7 @@ services:
|
||||
AIRFLOW__CORE__DAGS_FOLDER: /opt/pipeline/dags
|
||||
AIRFLOW__CORE__LOAD_EXAMPLES: "false"
|
||||
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS: "${AIRFLOW_ADMIN_USER:-admin}:admin"
|
||||
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
|
||||
PG_HOST: sc_database
|
||||
PG_PORT: "5432"
|
||||
PG_USER: ${DB_USERNAME}
|
||||
@@ -238,7 +240,7 @@ services:
|
||||
TYPESENSE_URL: http://typesense:8108
|
||||
TYPESENSE_API_KEY: ${TYPESENSE_API_KEY:-changeme}
|
||||
volumes:
|
||||
|
||||
- airflow_logs:/opt/airflow/logs
|
||||
depends_on:
|
||||
sc_database:
|
||||
condition: service_healthy
|
||||
@@ -275,3 +277,4 @@ volumes:
|
||||
kestra_storage:
|
||||
supplementary_data:
|
||||
typesense_data:
|
||||
airflow_logs:
|
||||
|
||||
Reference in New Issue
Block a user