Skip to content

Fix: Subscriber handler errors silently swallowed causing potential data loss#3218

Open
NitinKumar004 wants to merge 4 commits intogofr-dev:developmentfrom
NitinKumar004:fix/subscriber-error-swallowed
Open

Fix: Subscriber handler errors silently swallowed causing potential data loss#3218
NitinKumar004 wants to merge 4 commits intogofr-dev:developmentfrom
NitinKumar004:fix/subscriber-error-swallowed

Conversation

@NitinKumar004
Copy link
Copy Markdown
Contributor

@NitinKumar004 NitinKumar004 commented Mar 27, 2026

Summary

  • handleSubscription was logging handler errors but returning nil, which prevented the retry logic in startSubscriber from triggering on handler failures.
  • Now the handler error is propagated back to startSubscriber, enabling its existing retry-with-backoff mechanism.
  • Added unit tests for handler error propagation, successful handler, and subscribe error scenarios.

Fixes #3215

Test plan

  • Added TestHandleSubscription_HandlerErrorReturned — verifies handler error is returned
  • Added TestHandleSubscription_SuccessfulHandler — verifies nil on success
  • Added TestHandleSubscription_SubscribeError — verifies subscribe error is returned
  • Linter passes with 0 issues

handleSubscription was logging handler errors but returning nil,
preventing the retry logic in startSubscriber from triggering.
Now the error is propagated so failed messages are retried.

Fixes gofr-dev#3215
@NitinKumar004 NitinKumar004 force-pushed the fix/subscriber-error-swallowed branch from 9f5b862 to 6a6435d Compare March 27, 2026 12:05
Copy link
Copy Markdown
Member

@Umang01-hash Umang01-hash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Previously:
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-001\",...}","topic":"order-logs",...}}                                                                                           
  {"level":"INFO", "message":"handler received order #1: {OrderId:ORD-001 Status:FAILED}"}                                                                                                        
  {"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}                                                                                         
                                               ↑ ONE error log, then immediately next message:                                                                                                    
  {"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-BEFORE-FIX\",...}","topic":"order-logs",...}}                                                                                    
  {"level":"INFO", "message":"handler received order #2: {...}"}                                                                                                                                  
  {"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}                                                                                         
                                               ↑ NO DELAY — tight loop!      

After fix (return err — PR change):

  {"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-001\",...}","topic":"order-logs",...}}                                                                                           
  {"level":"INFO", "message":"handler received order #1: {OrderId:ORD-001 Status:FAILED}"}                                                                                                      
  {"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}                                                                                         
  {"level":"ERROR","message":"error in subscription for topic order-logs: handler: downstream DB unavailable"}
                                 ↑ TWO error logs for the same error — from handleSubscription AND startSubscriber                                                                                
                                 ↑ Then 2-second delay before next message is attempted                                                                                                           
  {"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-BEFORE-FIX\",...}","topic":"order-logs",...}}                                                                                    
                                               ↑ Next message 2 seconds later ✅ 

Every handler error fires both "error in handler for topic..." and "error in subscription for topic...". In a busy subscription loop under real failure conditions this doubles alert noise.

Maybe we can try a small fix: remove the Errorf on subscriber.go:70 and let startSubscriber's existing log (line 38) be the single error entry point.
That keeps error propagation clean and logs the error exactly once.

Umang01-hash and others added 2 commits April 3, 2026 13:56
The handler error was being logged in both handleSubscription and
startSubscriber, doubling alert noise. Remove the log from
handleSubscription and let startSubscriber be the single error
log entry point.
@NitinKumar004 NitinKumar004 force-pushed the fix/subscriber-error-swallowed branch from fa8abeb to 4e980a6 Compare April 11, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix: Subscriber handler errors silently swallowed causing potential data loss

2 participants