Large number of Transactions for the logs when using WASB

[ahmadtfarhan]

We are using WASB (azure storage account - file share) for the loggings and it shows that the transactions between Airflow and the file share (solely used for loggings) is very large - more than 1 million transaction per hour.

I am wondering what is happening in the app that continuously require the read/write transaction and how to reduce that number.

Currently we have 4 DAGs running including 2 only run on hourly schedule. The Airflow is deployed to Kubernetes cluster using Helm Chart (v1.10.0). We are using persistent volume to mount to storage account.

The PV and PVC looks like this

apiVersion: v1
kind: PersistentVolume
metadata:
  name: logs-pv
  namespace: namespace
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile
  azureFile:
    secretName: secret-name
    secretNamespace: namespace
    shareName: logs
    readOnly: false
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - uid=1000
    - gid=1000
    - mfsymlinks
    - cache=strict
    - actimeo=30
  claimRef:
    name: logs-pvc
    namespace: namespace

This is what we have in the values.yaml:

logs:
  persistence:
    # Enable persistent volume for storing logs
    enabled: true
    # Volume size for logs

    size: 1Gi
    # If using a custom storageClass, pass name here

    storageClassName:
    ## the name of an existing PVC to use
    existingClaim: logs-pvc

[Taragolis]

Remote shares, especially cloud based, could turned life into the pain, see: What resources might limit Scheduler’s performance

In additional there is report about this issue in SO in the past, seems like some stat calls could spawn additional Azure File Share transaction (whatever it was)

• https://stackoverflow.com/questions/66682516/too-many-transactions-to-azure-fileshare-from-airflow
• https://stackoverflow.com/questions/65274774/im-getting-a-lot-of-transactions-while-idling-airflow-and-azure-file-share

[ahmadtfarhan]

I tried changing file_parsing_sort_mode setting for scheduler to alphabetical but that has not helped. Bear in mind that we are using AKS and the file share is only used for logging and not DAGs files.
I am not sure what is there to change as well. I will appreciate any guidance.

[alex-bes-ccl]

Hi @ahmadfarhan97! did you manage to find a solution for this?

[ahmadtfarhan]

Check @carlosjourdan post
Apologies for late response!

[carlosjourdan]

We were also facing this problem using the helm chart. Turns out that the xxx-log-groomer containers in each pod (worker, triggerer, scheduler) was calling the /clean-logs script, which runs a find command in the $AIRFLOW_HOME/logs folder. This being a persistent volume with azurefile-csi storage class, a lot of transactions were sent to the storage account.

We solved it by disabling logging persistence at the helm chart level, and enabling remote logging at the airflow configuration level. We also changed all the remaining storage classes to managed-csi.

The relevant parts of the helm template look like this:

config:
  logging:
    remote_logging: "True"
    remote_log_conn_id: "wasb" #in our case we created the connection using the AzureKeyVaultBackend integration
    remote_base_log_folder: "wasb://logs@${STORAGE_ACCOUNT_NAME}.blob.core.windows.net"
  azure_remote_logging:
    remote_wasb_log_container: "logs"

logs:
  persistence:
    enabled: false

[ahmadtfarhan]

Yes thats what we did at the end, but the groomer issue is still not fixed which I believe needs to be resolved.

Many people have faced it so there should be a solution or a PR to the helm chart as the costs are ridiculously high.

[carlosjourdan]

I agree. The log groomer is very simple and isolated. Shouldn't be too hard to improve.

[62garzon]

To avoid the millions of transactions and their cost, we:

• Created a lifecyle policy in Azure to delete the logs after X days (it´s executed only once per day)
• Disabled the logGroomerSidecar in all services.

logs:
  persistence:
    enabled: true
    existingClaim: pvc-airflow-logs
    storageClassName: azureblob-fuse-premium
    
triggerer:
  logGroomerSidecar:
    enabled: false
    
scheduler:
  logGroomerSidecar:
    enabled: false     
    
workers:
  logGroomerSidecar:
    enabled: false
    
dagProcessor:
  logGroomerSidecar:
    enabled: false