[PoC] Add support for span link traversal in TraceQL

[toddtreece]

What this PR does:

this is a PoC of adding cross-trace link traversal to TraceQL. it allows querying spans across different traces that are connected by span links.

changes:

• new TraceQL operators:
  • ->> (link-to): finds spans linked TO from the current spanset.
  • <<- (link-from): finds spans that link FROM the current spanset.
  • &->> and &<<-: union that return both the original spans and the discovered linked spans.
• the search_sharder in the frontend has been updated to handle link traversal in multiple phases with some basic limits. it's unclear how to properly enforce query limits with the union operator.

How to test:

1. make link-dev to build and run tempo and grafana in docker

2. open http://localhost:3000/explore and run example queries below. grafana does not yet support this syntax, but the queries will still execute.

   {span.service.name="gateway"} &<<- {span.service.name="backend"} &<<- {span.service.name="database"}
   
   {span.service.name="database"} &->> {span.service.name="backend"} &->> {span.service.name="gateway"}
   

[CLAassistant]

All committers have signed the CLA.

[mdisibio]

Hi, first of all I want to say that I think this is a clever approach to providing this functionality. I hadn't thought about doing it this way, and the new operator syntax is good, close to the existing operators but different and still memorable.

However I need some help understanding a few parts.

1. This is executing the 1 link search as a series of internal searches for each side and stitching them together. If the query { } ->> { } ->> { } is given, is that always executed as exactly 3 internal searches? Or could it possibly be many more, because it searches for the spanIDs in batches?

2. Direction: It looks like it searches for the last spanset first, so for the query { a } ->> { b }, it searches for b, and then a's that link to the span IDs found in the b's. I wonder if we think fan-out or fan-in is more common, if we reversed this and searched for a's first, would it be more efficient or effective? What if there are millions of b's that aren't part of links. It would be a very low chance of selecting the ones that were, but are also part of a link from an a. Maybe a better example would be a query like { a } ->> { status=error }. There are likely millions of spans with error status, and maybe only a small percent of them are a linked span, so I think the likelihood of a matching 'a' might be too rare to be effective. But if we first searched for all a's with link:spanid != nil, maybe it would be more guaranteed to find the matches?

3. What do you think about making this a command within tempo-cli to start? It seems like this doesn't actually rely on new traceql engine or storage changes, so theoretically a query with the new operators could be broken down and executed against any existing tempo cluster. That would be very interesting way to validate the approach at different scales, with less commitment/blast radius than adding to the query-frontend itself.

[toddtreece]

I hadn't thought about doing it this way

@mdisibio it may be a terrible idea/implementation 🤣, but i am mostly just mimicking how i was manually querying. i had tried originally adding it to the querier, but it wasn't clear if that would work since each piece is kind of a separate query that might need to be sharded.

What do you think about making this a command within tempo-cli to start?

before i respond to 1 & 2, i had wondered something similar. does this make more sense to do in the grafana datasource and panel plugin for traces, or do you think this makes sense to also include in tempo? to me, it seems like better UX (at least for my use case) if the panel also combines the results into a structure similar to a parent/child relationship

[mdisibio]

does this make more sense to do in the grafana datasource and panel plugin for traces, or do you think this makes sense to also include in tempo?

Was thinking more about an easy way to vet the approach on different workloads and scales, and then we could figure out the best location for it. I do agree that server-side in Tempo seems the most likely.

[github-actions]

This PR has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. This pull request will be closed in 15 days if there is no new activity.
Please apply keepalive label to exempt this Pull Request.