Abnormal phenomenon of tail sampling when match big trace

[xiangtianyu]

Component(s)

processor/tailsampling

What happened?

Description

When there is big trace (contains more than 50000 spans a trace) other traces will not be sampled

Steps to Reproduce

1. An application runs a background thread continously produce span in a trace(running serval hours and produces 50000 spans a trace)

Expected Result

since the spans are created in a long period of time, it may not fill up the cache,the other traces should be sampled correctly

Actual Result

No other traces are sampled

Collector version

0.139

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") centos
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

tail_sampling:
        decision_wait: 15s
        num_traces: 50000
        expected_new_traces_per_sec: 1024
        policies:
          [
            {
              name: policy-error,
              type: status_code,
              status_code: {status_codes: [ERROR]}
            },

            {
              name: policy-latency,
              type: latency,
              latency: {
                threshold_ms: 5000,
                upper_threshold_ms: 600000
              }
            },

            {
              name: policy-probabilistic,
              type: probabilistic,
              probabilistic: {sampling_percentage: 0.01}
            }
          ]

Log output

Additional context

A typical trace

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• processor/tailsampling: @portertech @Logiraptor

See Adding Labels via Comments if you do not have permissions to add labels yourself.