How does a plugin/extension/dag get a local client without authenticating?

[timkpaine]

Let's imagine i want to call Pool.create_or_update_pool in my DAG. This no longer works in Airflow 3 due to no direct DB access. So instead I'd like to interact with the API via REST using the python client or airflow-ctl. But inside my DAG/plugin, I don't have a username / password or anything with which to interact with the API server. E.g. if the code is running inside my dag processor process, it really shouldn't need to authenticate as some random user. The scheduler and workers also presumably need to interact with the api server as well, is there some method to get API access when the code itself is running as part of a "trusted" process (e.g. in the dag processor, scheduler, worker, etc)?

It seems like the local client was working in this direction, but this just invokes the same Pool code that relies on direct DB access under the hood.

[timkpaine]

Here is a possible solution, which may work on the server with access to the airflow config [api_auth]:

from airflow.api_fastapi.auth.tokens import JWTGenerator, get_signing_args
from airflow.configuration import conf

# Generate a JWT token for the public REST API
generator = JWTGenerator(
    **get_signing_args(),  # Uses the same jwt_secret or private_key as the API server
    valid_for=conf.getint("api_auth", "jwt_expiration_time"),
    audience=conf.get("api", "jwt_audience", fallback="apache-airflow"),
)

# The token needs 'sub' (username) and 'role' claims for SimpleAuthManager 
token = generator.generate(extras={"sub": "internal_service", "role": "ADMIN"})

# Now use with any HTTP client
import httpx

client = httpx.Client(
    base_url=conf.get("api", "base_url", fallback="http://localhost:8080"),
    headers={"Authorization": f"Bearer {token}"}
)

# Update a pool
response = client.patch("/api/v2/pools/my_pool", json={"slots": 10})

[potiuk]

You can configure user/password in http connection and use it to run Python client https://github.com/apache/airflow-client-python which provides Pythonic API for the REST API.

[timkpaine]

shouldn't there be something like this built-in is my question? seems tough for e.g. every plugin author that needs to do anything like this to have to have their users provision an http connection. If i have 5 plugins i might very well end up having to setup 5 separate user/password combinations for each of them to be able to work, vs having some mechanism for code that the system administrator has already installed in the same place as airflow itself being able to interact with airflow.

[timkpaine]

worth noting that AWS has independently solved this problem for their platform already via boto (no username/password but they can use the iam setup to give the user a rest client).

[potiuk]

shouldn't there be something like this built-in is my question? seems tough for e.g. every plugin author that needs to do anything like this to have to have their users provision an http connection. If i have 5 plugins i might very well end up having to setup 5 separate user/password combinations for each of them to be able to work, vs having some mechanism for code that the system administrator has already installed in the same place as airflow itself being able to interact with airflow.

possibly yes. We discussed it and possibly we will introduce some built-in task authentication. But it is not implemented yet. If you want you can even attempt to contribute it.

[timkpaine]

Oh sweet, let me take a look