[Airflow 3.1.7] Task execution fails with "Invalid auth token" using LocalExecutor and Task SDK

[Ewald-Hoffranzen]

Apache Airflow version

3.1.7

If "Other Airflow 3 version" selected, which one?

No response

What happened?

Apache Airflow version

3.1.7

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Task execution consistently fails with Invalid auth token: Signature verification failed despite Scheduler and API-Server having identical JWT secrets.

What you think should happen instead?

Tasks should execute successfully with valid JWT authentication between Scheduler and API-Server.

How to reproduce

Environment:

• Airflow: 3.1.7 (Docker image: apache/airflow:3.1.7-python3.12)
• Executor: LocalExecutor
• Database: PostgreSQL 15
• Deployment: Docker Compose

Configuration:

services:
  airflow-scheduler:
    environment:
      AIRFLOW__CORE__INTERNAL_API_URL: "http://airflow-apiserver:8080"
      AIRFLOW__CORE__EXECUTION_API_SERVER_URL: "http://airflow-apiserver:8080/execution/"
      AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
      AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
      AIRFLOW__CORE__EXECUTOR: "LocalExecutor"

  airflow-apiserver:
    environment:
      AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
      AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"

Steps:

1. Deploy Airflow 3.1.7 with above configuration
2. Verify both services have identical JWT secrets: airflow config get-value secrets jwt_secret
3. Trigger any DAG task
4. Observe immediate failure

Operating System

Ubuntu 22.04 (Docker)

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

Docker Compose with separate containers for:

• airflow-apiserver
• airflow-scheduler
• airflow-dag-processor
• airflow-triggerer
• PostgreSQL (external, version 15)

Anything else?

Error Logs:

Scheduler error:

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 966, in _on_child_started
  ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
  
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature verification failed
Correlation-id=019c6702-76ee-7344-b98b-b3890d1e64df

API-Server error:

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 328, in avalidated_claims
  self._verify_signature(signing_input, header, signature, key, algorithms)
  
jwt.exceptions.InvalidSignatureError: Signature verification failed

Observations:

1. Both services confirmed to have identical JWT secrets (86 bytes, SHA512-compliant)
2. Problem persists after deleting jwt_secret from database and complete service recreation
3. InsecureKeyLengthWarning appears intermittently, suggesting worker subprocess uses cached/different secret
4. Setting AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth is ignored by Task SDK
5. Worker subprocess in LocalExecutor creates tokens that API-Server cannot validate

Suspected Root Cause:
Worker subprocess inherits or caches a different JWT secret than API-Server uses for validation, possibly due to:

• Forked process not inheriting updated environment variables
• Task SDK hardcoded to use JWT regardless of auth_backends configuration
• Race condition in secret loading during worker subprocess creation

Impact:
Critical - Airflow 3.1.7 with LocalExecutor cannot execute any tasks.

Workarounds Attempted:

• ✗ Recreating all containers
• ✗ Deleting jwt_secret from database
• ✗ Setting Basic Auth (ignored by Task SDK)
• ✗ Ensuring identical secrets across all services

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

What you think should happen instead?

No response

How to reproduce

Apache Airflow version

3.1.7

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Task execution consistently fails with Invalid auth token: Signature verification failed despite Scheduler and API-Server having identical JWT secrets.

What you think should happen instead?

Tasks should execute successfully with valid JWT authentication between Scheduler and API-Server.

How to reproduce

Environment:

• Airflow: 3.1.7 (Docker image: apache/airflow:3.1.7-python3.12)
• Executor: LocalExecutor
• Database: PostgreSQL 15
• Deployment: Docker Compose

Configuration:

services:
  airflow-scheduler:
    environment:
      AIRFLOW__CORE__INTERNAL_API_URL: "http://airflow-apiserver:8080"
      AIRFLOW__CORE__EXECUTION_API_SERVER_URL: "http://airflow-apiserver:8080/execution/"
      AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
      AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
      AIRFLOW__CORE__EXECUTOR: "LocalExecutor"

  airflow-apiserver:
    environment:
      AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
      AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"

Steps:

1. Deploy Airflow 3.1.7 with above configuration
2. Verify both services have identical JWT secrets: airflow config get-value secrets jwt_secret
3. Trigger any DAG task
4. Observe immediate failure

Operating System

Ubuntu 22.04 (Docker)

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

Docker Compose with separate containers for:

• airflow-apiserver
• airflow-scheduler
• airflow-dag-processor
• airflow-triggerer
• PostgreSQL (external, version 15)

Anything else?

Error Logs:

Scheduler error:

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 966, in _on_child_started
  ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
  
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature verification failed
Correlation-id=019c6702-76ee-7344-b98b-b3890d1e64df

API-Server error:

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 328, in avalidated_claims
  self._verify_signature(signing_input, header, signature, key, algorithms)
  
jwt.exceptions.InvalidSignatureError: Signature verification failed

Observations:

1. Both services confirmed to have identical JWT secrets (86 bytes, SHA512-compliant)
2. Problem persists after deleting jwt_secret from database and complete service recreation
3. InsecureKeyLengthWarning appears intermittently, suggesting worker subprocess uses cached/different secret
4. Setting AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth is ignored by Task SDK
5. Worker subprocess in LocalExecutor creates tokens that API-Server cannot validate

Suspected Root Cause:
Worker subprocess inherits or caches a different JWT secret than API-Server uses for validation, possibly due to:

• Forked process not inheriting updated environment variables
• Task SDK hardcoded to use JWT regardless of auth_backends configuration
• Race condition in secret loading during worker subprocess creation

Impact:
Critical - Airflow 3.1.7 with LocalExecutor cannot execute any tasks.

Workarounds Attempted:

• ✗ Recreating all containers
• ✗ Deleting jwt_secret from database
• ✗ Setting Basic Auth (ignored by Task SDK)
• ✗ Ensuring identical secrets across all services

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

Operating System

ubuntu-20.04.3-live-server-amd64

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

this AI generated report is not helpful to help you. You likely indeed have some cache - other people do not experience similar issues, so you need to dig deeper.

[potiuk]

Converted to a discussion - you can continue posting more finding here and maybe someone will be able to help you better than the AI report that does not really go to the real cause.

[MachSilva]

Try setting AIRFLOW__API_AUTH__JWT_SECRET. Note the section api_auth

I had a similar issue. In my case, I solved by setting proper ownership of airflow.cfg.