Bug: Loki 3.6.5 Readiness Fails (503) When Accessing From Internal Container

[irwinrex]

Describe the bug

Loki 3.6.5 intermittently returns 503 on the /ready endpoint during startup when deployed in Kubernetes (single-binary mode with -target=all).

• Pod starts successfully
• No restarts
• Readiness probe fails with HTTP 503
• External access works
• Internal access from within the Loki container fails

Readiness eventually succeeds, but by external container in same namespace

---

To Reproduce

1. Started Loki docker.io/grafana/loki:3.6.5
   -config.file=/etc/loki/config/config.yaml
   -target=all

2. (Promtail not required — readiness fails before ingestion)

3. Access readiness endpoint:
   GET http://:3100/ready

Result:
HTTP 503

Kubernetes readiness probe:
http-get http://:http-metrics/ready
delay=15s timeout=1s period=10s

Event observed:
Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503

---

Expected behavior

• /ready should return 200 once Loki process is operational
• Readiness should not return prolonged 503
• Internal and external access should behave consistently

---

Environment:

• Infrastructure: Kubernetes (StatefulSet, single-binary mode)
• Deployment tool: Helm
• Image: docker.io/grafana/loki:3.6.5
• Storage: PVC (30Gi)

---

Screenshots, Promtail config, or terminal output

Pod event during failure:
Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503

Pod eventually transitions to:
Ready: True
ContainersReady: True

Full kubectl describe pod output available upon request.

[sapn1s]

Was having same issue, but in my case I found the cause being:

func (i *Lifecycler) CheckReady(ctx context.Context) error {
    i.readyLock.Lock()
    defer i.readyLock.Unlock()

    if i.ready {
        return nil  // once ready, always ready (latch)
    }

    if err := i.checkRingHealthForReadiness(ctx); err != nil {
        i.readySince = time.Time{}  // RESET the timer on any failure
        return err
    }

    // Timer starts HERE — only after health checks pass
    if i.readySince.IsZero() {
        i.readySince = time.Now()
    }
    if time.Since(i.readySince) < i.cfg.MinReadyDuration {
        return fmt.Errorf("waiting for %v after being ready", i.cfg.MinReadyDuration)
    }

    i.ready = true  // latch — never goes back to false
    return nil
}

Which is found in:
https://raw.githubusercontent.com/grafana/dskit/main/ring/lifecycler.go

What essentially happens is, something must trigger /ready and then the default MinReadyDuration time is 15 seconds with reasoning:

f.DurationVar(&cfg.MinReadyDuration, prefix+"min-ready-duration", 15*time.Second, "Minimum duration to wait after the internal readiness checks have passed but before succeeding the readiness endpoint. This is used to slowdown deployment controllers (eg. Kubernetes) after an instance is ready and before they proceed with a rolling update, to give the rest of the cluster instances enough time to receive ring updates.")

I can't be sure you have the same issue since your issue doesn't have more specific logs.
But even then, I also checked proposed PR fix and it seems wrong, making Starting status to be assumed as good, is not good, since in real Starting state it means it is not ready to accept logs, so the requests would fail.

[JStickler]

FYI @irwinrex, we've recently published extensive Troubleshooting docs for Loki.