Tempo 3.0

[javiermolinar]

🚀 Tempo 3.0 – Upcoming Major Release

We are excited to share early details about Tempo 3.0, our next major release. This version introduces substantial architectural improvements, several breaking changes, and updates that will streamline operations and support Tempo’s evolution for years to come.

You can track progress, view planned features, and follow related issues in the Tempo 3.0 Milestone.

Tempo 3.0 focuses on four primary goals:

• Greater resilience and scalability
• Simplified operational models
• A cleaner, more maintainable codebase
• TraceQL Metrics GA: With this version, TraceQL metrics will become generally available.

Below is an overview of the key areas of work included in this major release.

A New, Decoupled Architecture

Tempo 3.0 introduces a decoupled architecture that separates ingestion and query processing. This redesign improves resilience, scalability, and operational simplicity, allowing each part of the system to scale independently and more predictably.

Under the hood, the system uses a distributed, asynchronous messaging layer to buffer data between ingestion and compaction. This ensures that heavy query loads or large batch operations do not interfere with ingestion or compaction performance.

Key benefits:

• Improved reliability: Reduces duplicated work and ensures data consistency.
• Operational simplicity: Clear separation of responsibilities simplifies deployments and upgrades.
• TraceQL Metrics GA: TraceQL metrics become generally available in this release.

Two new components will be released:

• Block-builder: Write-path component responsible for building blocks and persisting them; replaces the ingesters.
• Live-store: Read-path component responsible for serving recent data queries; also replaces the ingesters and the metrics-generator local-blocks processor.

New Compaction Architecture

Tempo 3.0 introduces a job-based compaction system, replacing the previous per-compactor approach. This new system addresses limitations in the old process and improves reliability for all queries, including TraceQL metrics.

Benefits include:

• Elimination of data duplication: Each block is processed exactly once, ensuring consistency.
• Centralized job management: A scheduler coordinates jobs, tracks progress, and automatically reschedules failed jobs, reducing operational complexity.
• Extensibility for future batch processes: While initially focused on compaction, this framework can handle other operations like trace deletion.
• Predictable batch processing: Jobs complete reliably over hours with clear completion tracking, supporting safer large-scale operations.

Storage Improvements

Tempo 3.0 adds support for vParquet5, improving storage efficiency, performance, and maintainability.

Deprecations & Breaking Changes

As a major version, Tempo 3.0 deprecates certain modules and legacy features:

• Ingester and compactor modules are deprecated in favor of the new components.
• Metrics-generator local-blocks processor is deprecated.
• Removal of older APIs, overrides, and legacy shims.
• Simplification of deployment modes and code paths incompatible with the new architecture.

Migration guides will be provided to help operators transition smoothly.

Deployment & Helm Chart Updates

Tempo 3.0 will include updates to the Helm chart to support the new architecture and operational improvements. We are currently evaluating whether to enhance the existing community chart or provide a new chart for 3.0.

In either case, detailed instructions and guidance will be provided to ensure safe deployments and upgrades.

Documentation & Migration Guides

Ahead of the release, we will publish:

• Updated architecture documentation
• Detailed migration guides
• Release notes outlining all changes and breaking updates

These resources will help operators understand the benefits of the new architecture and transition confidently to 3.0.

💬 Feedback

We welcome community feedback and questions as we move toward this major release. Please use this Announcement thread to share your thoughts and engage with the team.

[KyriosGN0]

Hey, will the old architecture continue to work in V3? (same as mimir) ? if so for how many versions ?

[joe-elliott]

The current plan is to completely retire the old architecture. We will be cutting a 2.10 early next year to give as many features/bug fixes/security patches as possible to the community before we move to 3.0.

[via-justa]

@javiermolinar, any update on when can we expect the release to happen?
We're holding the deployment of the tool in the company so we can deploy the new architecture from day one.

[javiermolinar]

Hey! We're aiming for a release in early May. Our team is busy prepping the tools and migration guides to ensure a smooth transition for everyone.

[akoshche]

Hello @javiermolinar, I cannot seems to find any compatible helm chart for Tempo 3.0
Is work still in progress for the Helm/Operator to support Tempo 3.0?

[javiermolinar]

The helm chart is maintained by the community, you can find it here

https://github.com/grafana-community/helm-charts/tree/main/charts/tempo-distributed

[Justin-DynamicD]

So the community chart is finally pushing tempo 3.x for microservices, but NOT for the monolithic chart. After some review all I can think of is "Operational simplicity" for whom?

having to deploy and maintain your own kafka/redpanda in addition to Tempo is not my definition of simple. Plus the charts refuse to handle it, they just want the to know the endpoints.

Is there any guide anywhere that helps outline what is expected other than "deploy a compatible kafka and good luck!"?

[javiermolinar]

Fair enough, I'm working a doc to provide some guidance #7714