Fix: Subscriber handler errors silently swallowed causing potential data loss#3218
Open
NitinKumar004 wants to merge 4 commits intogofr-dev:developmentfrom
Open
Fix: Subscriber handler errors silently swallowed causing potential data loss#3218NitinKumar004 wants to merge 4 commits intogofr-dev:developmentfrom
NitinKumar004 wants to merge 4 commits intogofr-dev:developmentfrom
Conversation
handleSubscription was logging handler errors but returning nil, preventing the retry logic in startSubscriber from triggering. Now the error is propagated so failed messages are retried. Fixes gofr-dev#3215
9f5b862 to
6a6435d
Compare
Umang01-hash
requested changes
Apr 3, 2026
Member
Umang01-hash
left a comment
There was a problem hiding this comment.
- Previously:
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-001\",...}","topic":"order-logs",...}}
{"level":"INFO", "message":"handler received order #1: {OrderId:ORD-001 Status:FAILED}"}
{"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}
↑ ONE error log, then immediately next message:
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-BEFORE-FIX\",...}","topic":"order-logs",...}}
{"level":"INFO", "message":"handler received order #2: {...}"}
{"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}
↑ NO DELAY — tight loop!
After fix (return err — PR change):
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-001\",...}","topic":"order-logs",...}}
{"level":"INFO", "message":"handler received order #1: {OrderId:ORD-001 Status:FAILED}"}
{"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}
{"level":"ERROR","message":"error in subscription for topic order-logs: handler: downstream DB unavailable"}
↑ TWO error logs for the same error — from handleSubscription AND startSubscriber
↑ Then 2-second delay before next message is attempted
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-BEFORE-FIX\",...}","topic":"order-logs",...}}
↑ Next message 2 seconds later ✅
Every handler error fires both "error in handler for topic..." and "error in subscription for topic...". In a busy subscription loop under real failure conditions this doubles alert noise.
Maybe we can try a small fix: remove the Errorf on subscriber.go:70 and let startSubscriber's existing log (line 38) be the single error entry point.
That keeps error propagation clean and logs the error exactly once.
The handler error was being logged in both handleSubscription and startSubscriber, doubling alert noise. Remove the log from handleSubscription and let startSubscriber be the single error log entry point.
fa8abeb to
4e980a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
handleSubscriptionwas logging handler errors but returningnil, which prevented the retry logic instartSubscriberfrom triggering on handler failures.startSubscriber, enabling its existing retry-with-backoff mechanism.Fixes #3215
Test plan
TestHandleSubscription_HandlerErrorReturned— verifies handler error is returnedTestHandleSubscription_SuccessfulHandler— verifies nil on successTestHandleSubscription_SubscribeError— verifies subscribe error is returned