What is the bug?
The Loki Helm chart's ServiceMonitor templates contain a hardcoded job relabeling rule that sets the label to <namespace>/<component> (e.g. loki/write). When metrics are scraped via an OpenTelemetry Collector, this produces a doubled job label value of <namespace>/<namespace>/<component> (e.g. loki/loki/write).
This happens because the OTel Collector's Prometheus receiver stores the scraped job label as service.name, and the remote write exporter reconstructs job as <service.namespace>/<service.name> per the OpenTelemetry semantic conventions. Since service.namespace is set to the Kubernetes namespace by the k8sattributes processor, the namespace is duplicated.
The relabeling rule is hardcoded in the Helm template and cannot be overridden via values.
How to reproduce it?
- Deploy Loki via the
loki Helm chart (I deployed SSD mode) with monitoring.serviceMonitor.enabled: true
- Scrape Loki's ServiceMonitors via an OpenTelemetry Collector using the
prometheusreceiver and the TargetAllocator
- Enrich resource attributes using the
k8sattributes processor
- Remote write to a metrics storage layer via the
prometheusremotewrite exporter
- Observe
job label values such as loki/loki/write in metrics storage
What did you think would happen?
The job label should be loki/write, consistent with the intent of the Helm chart's relabeling rule.
What was your environment?
Kubernetes on EKS, Helm deployment
Kubernetes Version: 1.32
Helm Chart Version: 6.29.0
Loki: 3.4.2
Any additional context to share?
Proposed fix: Make the job relabeling rule overridable via Helm values
What is the bug?
The Loki Helm chart's ServiceMonitor templates contain a hardcoded
jobrelabeling rule that sets the label to<namespace>/<component>(e.g.loki/write). When metrics are scraped via an OpenTelemetry Collector, this produces a doubledjoblabel value of<namespace>/<namespace>/<component>(e.g.loki/loki/write).This happens because the OTel Collector's Prometheus receiver stores the scraped
joblabel asservice.name, and the remote write exporter reconstructsjobas<service.namespace>/<service.name>per the OpenTelemetry semantic conventions. Sinceservice.namespaceis set to the Kubernetes namespace by thek8sattributesprocessor, the namespace is duplicated.The relabeling rule is hardcoded in the Helm template and cannot be overridden via values.
How to reproduce it?
lokiHelm chart (I deployed SSD mode) withmonitoring.serviceMonitor.enabled: trueprometheusreceiverand the TargetAllocatork8sattributesprocessorprometheusremotewriteexporterjoblabel values such asloki/loki/writein metrics storageWhat did you think would happen?
The
joblabel should beloki/write, consistent with the intent of the Helm chart's relabeling rule.What was your environment?
Kubernetes on EKS, Helm deployment
Kubernetes Version: 1.32
Helm Chart Version: 6.29.0
Loki: 3.4.2
Any additional context to share?
Proposed fix: Make the
jobrelabeling rule overridable via Helm values