Benchmarking Workflow

[pjattke]

Some aspects for improving the current SoK benchmarking setup:

• Use for each tool a dedicated docker-entrypoint.sh script
  • Missing for SEALion
• Implement support for running same program with different configurations (e.g., varying input size)
  • Goal should be to avoid code duplication by only having code once in repository
• Run each program (not tool!) in a separate container to speed up benchmark execution time
  • Maybe it makes sense to somehow dynamically generate the docker-entrypoint.sh scripts based on a template (one per program) instead of generating a static individual script per {tool, program} configuration.
  • As this adds complexity to the Github Action: maybe it makes more sense to write a python script that is then executed in the Github Actions runner (instead of writing code directly in action just run the script).
• Use AWS CLI to send SSM command to VMs
  • Set SSM command's timeoutSeconds argument to the timeout configured in the workflow
  • Provide parameters --output-s3-bucket-name <value> and --output-s3-key-prefix <value> to write SSM output into S3 bucket (see https://docs.aws.amazon.com/cli/latest/reference/ssm/send-command.html) for easier debugging