rule: log PromQL info annotations at debug instead of warn#8873
Open
Goutham-Annem wants to merge 4 commits into
Open
rule: log PromQL info annotations at debug instead of warn#8873Goutham-Annem wants to merge 4 commits into
Goutham-Annem wants to merge 4 commits into
Conversation
Goutham-Annem
force-pushed
the
rule-log-promql-info-at-debug
branch
3 times, most recently
from
July 5, 2026 14:45
0bb7fda to
46a09b8
Compare
Goutham-Annem
added a commit
to Goutham-Annem/thanos
that referenced
this pull request
Jul 5, 2026
…ssue thanos-io#8229 Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
Goutham-Annem
force-pushed
the
rule-log-promql-info-at-debug
branch
from
July 10, 2026 10:41
ca55bf2 to
a9d9658
Compare
Goutham-Annem
added a commit
to Goutham-Annem/thanos
that referenced
this pull request
Jul 10, 2026
…ssue thanos-io#8229 Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
Author
|
The failing 'Documentation' CI check is a pre-existing repo-wide issue — the dead URL at |
Goutham-Annem
added a commit
to Goutham-Annem/thanos
that referenced
this pull request
Jul 18, 2026
…ssue thanos-io#8229 Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
Goutham-Annem
force-pushed
the
rule-log-promql-info-at-debug
branch
from
July 18, 2026 17:37
a9d9658 to
784ebc6
Compare
Author
|
The failing CI check ( |
filterOutPromQLWarnings logged every PromQL annotation at warn level, including purely informational ones (e.g. "metric might not be a counter"), causing log spam on every rule evaluation cycle for rules that trigger this heuristic. Add IsPromQLInfoAnnotation to distinguish info from warning annotations, and log info annotations at debug. Fixes thanos-io#8229 Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
…ssue thanos-io#8229 Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
Goutham-Annem
force-pushed
the
rule-log-promql-info-at-debug
branch
from
July 23, 2026 02:56
784ebc6 to
2a583e0
Compare
Author
|
The failing check (Thanos end-to-end tests) appears to be a pre-existing flake unrelated to changes in pkg/rule. Could a maintainer re-run the CI when convenient? Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
filterOutPromQLWarningsincmd/thanos/rule.gologged every PromQL annotation atlevel.Warn, including purely informational ones like "metric might not be a counter" (annotations.PromQLInfo). Since this runs on every rule evaluation cycle for any rule whose query triggers the heuristic, it produces continuous log spam (see Rule: log warning spam from PromQL info "metric might not be a counter" #8229) rather than something an operator needs to act on.IsPromQLInfoAnnotationtopkg/extannotations/annotations.go, mirroring the existingIsPromQLAnnotationbut checking only theannotations.PromQLInfoprefix.filterOutPromQLWarningsnow logs info-type annotations atlevel.Debugand keeps true warning-type annotations (annotations.PromQLWarning) atlevel.Warn. Filtering behavior (which warnings get removed from the response) is unchanged — only the log level for info-type annotations changes.Verification
TestIsPromQLInfoAnnotation(pkg/extannotations/annotations_test.go) covering info vs. warning vs. unrelated strings.TestFilterOutPromQLWarnings_LogLevel(cmd/thanos/rule_test.go), which captures logger output vialog.NewLogfmtLoggerand asserts info annotations producelevel=debugwhile warning annotations producelevel=warn.TestFilterOutPromQLWarnings(filtering behavior) still passes unchanged.make go-lintpasses with 0 issues.Closes #8229