[Bug]: warp hangs indefinitely on "Download operations"

[saidsay-so]

Hello, thank you for your work, please let me know if you need more details!

Description

Running a mixed benchmark in a distributed manner brings warp to an indefinite hang, which seem to occur when downloading the operations from the clients. The coordinator displays Benchmarking: Client [redacted]:7761 : Finished stage benchmark ..., and the clients all hang on "warp: Sending ops".
However, this hang doesn't happen anymore when adding the full: true config option or reducing the duration from 3m to less (2m). I suspect that the issue happens when collecting the results from the clients, which might be too much data given the number of clients (30), hosts (120), concurrency (128), and time segments.
A quick debug session with Delve on the coordinator shows that almost all goroutines are waiting on a optimized runtime function for runtime.futex, while the main blocks on ui.Wait.

Steps to Reproduce

warp run warp_test.yml

Configuration file

warp:
  api: v1

  # Benchmark to run.
  # Corresponds to warp [benchmark] command.
  benchmark: mixed

  # Do not print any output.
  quiet: false

  # Disable terminal color output.
  no-color: false

  # Print results and errors as JSON.
  json: false

  # Output benchmark+profile data to this file.
  # By default a unique filename is generated.
  bench-data: test-ec-{{.Date}}

  # Connect to warp clients and run benchmarks there.
  # See https://github.com/minio/warp?tab=readme-ov-file#distributed-benchmarking
  # Can be a single value or a list.
  warp-client:
    - warpclient{1..30}

  # Run MinIO server profiling during benchmark;
  # possible values are 'cpu', 'cpuio', 'mem', 'block', 'mutex', 'threads' and 'trace'.
  # Can be single value or a list.
  server-profile:

  # Remote host parameters and connection info.
  remote:
    # Specify custom region
    region: us-east-1

    # Access key and Secret key
    access-key:
    secret-key:

    # Specify one or more hosts.
    # The benchmark will be run against all hosts concurrently.
    # Multiple servers can be specified with ellipsis notation;
    # for example '10.0.0.{1...10}:9000' specifies 10 hosts.
    # See more at https://github.com/minio/warp?tab=readme-ov-file#multiple-hosts
    host:
      - cephrgw{1..30}:808{0..3} 

    # Use TLS for calls.
    tls: false

    # Allow TLS with unverified certificates.
    insecure: false

    # Stream benchmark statistics to Influx DB instance.
    # See more at https://github.com/minio/warp?tab=readme-ov-file#influxdb-output
    influxdb: ''

    # Bucket to use for benchmark data.
    #
    #  CAREFUL:    ALL DATA WILL BE DELETED IN BUCKET!
    #
    # By default, 'warp-benchmark-bucket' will be created or used.
    bucket: 

  # params specifies the benchmark parameters.
  # The fields here depend on the benchmark type.
  params:
    # Duration to run the benchmark.
    # Use 's' and 'm' to specify seconds and minutes.
    duration: 3m

    # Concurrent operations to run per warp instance.
    concurrent: 128

    # The number of objects to upload before starting the benchmark.
    # Upload enough objects to ensure that any remote caching is bypassed.
    objects: 1500

    # Properties of uploaded objects.
    obj:
      # Size of each uploaded object
      # 
      size: 32k

      # Randomize the size of each object within certain constraints.
      # See https://github.com/minio/warp?tab=readme-ov-file#random-file-sizes
      rand-size: false

      # Force specific size of each multipart part.
      # Must be '5MB' or bigger.
      part-size:

    # Use automatic termination when traffic stabilizes.
    # Can not be used with distributed warp setup.
    # See https://github.com/minio/warp?tab=readme-ov-file#automatic-termination
    autoterm:
      enabled: false
      dur: 10s
      pct: 7.5

    # Do not clear bucket before or after running benchmarks.
    no-clear: false

    # Leave benchmark data. Do not run cleanup after benchmark.
    # Bucket will still be cleaned prior to benchmark.
    keep-data: false


  # The io section specifies custom IO properties for uploaded objects.
  io:
    # Use a custom prefix
    prefix:

    # Do not use separate prefix for each thread
    no-prefix: false

    # Add MD5 sum to uploads
    md5: false

    # Disable multipart uploads
    disable-multipart: false

    # Disable calculating sha256 on client side for uploads
    disable-sha256-payload: false

    # Server-side sse-s3 encrypt/decrypt objects
    sse-s3-encrypt: false

    # Encrypt/decrypt objects (using server-side encryption with random keys)
    sse-c-encrypt: false

    # Override storage class.
    # Default storage class will be used unless specified.
    storage-class:

  analyze:
    # Display additional analysis data.
    verbose: false
    # Only output for this host.
    host: ''
    # Only output for this operation. Can be 'GET', 'PUT', 'DELETE', etc.
    filter-op: ''
    # Split analysis into durations of this length.
    # Can be '1s', '5s', '1m', etc.
    segment-duration:
    # Output aggregated data as to file.
    out:
    # Additional time duration to skip when analyzing data.
    skip-duration:
    # Max operations to load for analysis.
    limit:
    # Skip this number of operations before starting analysis.
    offset:

  advanced:
    # Stress test only and discard output.
    stress: false

    # Print requests.
    debug: true

    # Disable HTTP Keep-Alive
    disable-http-keepalive: false

    # Enable HTTP2 support if server supports it
    http2: false

    # Rate limit each instance to this number of requests per second
    rps-limit:

    # Host selection algorithm.
    # Can be 'weighed' or 'roundrobin'
    host-select: weighed

    # "Resolve the host(s) ip(s) (including multiple A/AAAA records).
    # This can break SSL certificates, use --insecure if so
    resolve-host: false

    # Specify custom write socket buffer size in bytes
    sndbuf: 32768

    # Specify custom read socket buffer size in bytes
    rcvbuf: 32768

    # When running benchmarks open a webserver to fetch results remotely, eg: localhost:7762
    serve:

[saidsay-so]

Also quick note to say that I also encountered #476 when using the "full: true" configuration option.