DANDI S3 Log Extraction

Fast extraction of access summary data from DANDI S3 logs.

Developed for the DANDI Archive.

Installation

pip install dandi-s3-log-extraction

DANDI Usage

Usage on the DANDI archive logs requires a bit more customization than the generic package.

Begin by ensuring a special required environment variable is set:

S3_LOG_EXTRACTION_PASSWORD

• Various sensitive information on Drogon is encrypted using this password, including:
  • the regular expression for all associated Drogon IPs.
  • the IP index and geolocation caches.

This allows us to store full IP information in a persistent way (in case we need to go back and do a lookup) while still being secure.

export S3_LOG_EXTRACTION_PASSWORD="ask_yarik_or_cody_for_password"

In fresh environments, the cache should be specified as:

dandis3logextraction config cache set /mnt/backup/dandi/s3-logs-extraction-cache

To run all the steps (such as for daily updates):

dandis3logextraction extract /mnt/backup/dandi/dandiarchive-logs
dandis3logextraction update ip indexes
dandis3logextraction update ip regions
dandis3logextraction update ip coordinates
dandis3logextraction update summaries
dandis3logextraction update totals
dandis3logextraction update summaries --mode archive
dandis3logextraction update totals --mode archive