-
Notifications
You must be signed in to change notification settings - Fork 9
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Currently, only the connection id is passed on between DbToDbOperator, DbToDbHook, copy_db_to_db, etc. That means that every time some piece of code needs a connection, it has to retrieve the connection (hence accessing the Airflow database, or Hashicorp Vault if using that).
That happens multiple times within the scope of a single task, which generates a lot of overhead. That can be seen on task logs, where multiple events of INFO - Retrieving connection 'example_connection' for the same connection show up in the log.
We should implement some cache to the connection object so that it is reused between all those functions and is retrieved only the first time it is needed.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request