Skip to content

feat(dataobj): Schedule LogMerge via per-tenant IndexMerge/LogMerge flip-flop#23316

Open
trevorwhitney wants to merge 7 commits into
grafana:mainfrom
trevorwhitney:sequential-index-logmerge-scheduler
Open

feat(dataobj): Schedule LogMerge via per-tenant IndexMerge/LogMerge flip-flop#23316
trevorwhitney wants to merge 7 commits into
grafana:mainfrom
trevorwhitney:sequential-index-logmerge-scheduler

Conversation

@trevorwhitney

Copy link
Copy Markdown
Collaborator

What this PR does

Changes how dataobj log compaction (LogMerge) is scheduled. Previously LogMerge only ran once a (tenant, window) had converged to a single index (len(entries) == 1 routing in the old poll loop). Under continuous ingestion a window routinely holds more than one index, so it never converged and its logs were never compacted.

This replaces the cron/poll loop with a per-tenant IndexMerge ↔ LogMerge flip-flop:

  • coordinator.Run spawns one worker goroutine per configured tenant (-dataobj.compaction.tenants) and drains them via a WaitGroup on shutdown. There is no poll loop.
  • Each worker alternates phases: IndexMerge (unchanged compactTenant) then LogMerge. On success it flips; on error it re-arms the same phase.
  • LogMerge no longer waits for convergence — it compacts the log objects behind every index in the window, one index at a time, reusing the existing compactTenantLogs per index. This is the core fix.

Why

LogMerge effectively never triggered in environments that don't converge to a single index. Decoupling it from convergence lets log compaction make progress regardless of index count.

Scope / notes

Targeted at getting LogMerge running in a dev environment for a single configured tenant. Deliberately deferred (documented for revisit before broader/production use):

  • Never-sleep loop: maximizes compaction utilization; the intended backpressure is scheduler-queue depth (not built here). An idle/converged tenant will hot-loop ToC reads — acceptable for a single active dev tenant.
  • Multi-tenant ToC-swap serialization (single shared per-window ToC object) — a process-global swap mutex is deferred; safe with one worker.
  • PollingInterval is retained as a config knob but no longer read by the scheduler (reserved for a future inter-phase throttle).
  • Metrics migration is minimal: existing counters are reused; "cycle" metrics now count per-tenant phase iterations (compacted|converged|failed).

Config

  • New: -dataobj.compaction.tenants (comma-separated) — the tenants to compact; one worker each. Static (restart to change); empty = no workers.

Testing

  • Unit tests for both phases (per-index LogMerge fan-out incl. delta-counter assertions, phaseOutcome mapping, cancellation short-circuit), plus lifecycle tests for Run/runTenantLoop (WaitGroup drain on cancel, error-keeps-phase / success-flips, tenant dedupe).
  • go test ./pkg/engine/... ./pkg/dataobj/... green; go vet and gofmt clean.

Operational acceptance (dev)

With -dataobj.compaction.enabled=true -dataobj.compaction.tenants=<tenant>, loki_dataobj_compaction_tenant_log_cycles_total{outcome="compacted"} should increment on windows with >1 index (which previously never happened), and ..._indexes_added_total / ..._tasks_total advance during log-merge phases.

@trevorwhitney
trevorwhitney requested a review from a team as a code owner July 16, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant