Skip to content

Fix connection leaks in broker and transport access#1490

Open
ShubhAtWork wants to merge 1 commit intomher:masterfrom
twofourlabs:fix/connection-leaks
Open

Fix connection leaks in broker and transport access#1490
ShubhAtWork wants to merge 1 commit intomher:masterfrom
twofourlabs:fix/connection-leaks

Conversation

@ShubhAtWork
Copy link

@ShubhAtWork ShubhAtWork commented Mar 3, 2026

Summary

Several places call capp.connection() without using a context manager, creating AMQP/Redis connections that are never closed. Each leaked connection holds open a socket and file descriptor. Under sustained use (e.g. the Broker view refreshing, or /api/queues/length being polled by monitoring), this causes gradual file descriptor exhaustion, eventually leading to Too many open files errors that crash the process.

Leaks fixed

Location Problem Fix
Flower.transport property Called on every request that checks broker type. Opens connection, reads driver type, discards without closing. Cache the result (transport type never changes at runtime) + context manager for initial lookup
print_banner() app.connection().as_uri() opens a connection to format the broker URL for the startup log. Never closed. Context manager
BrokerView.get() Two separate capp.connection() calls — one for Broker creation, one for display URL. Neither closed. Single context-managed connection for both
GetQueueLengths.get() capp.connection().as_uri() opens a connection to get broker URI. Never closed. Context manager
WorkersView.get() capp.connection().as_uri() opens a connection on every non-JSON page render. Never closed. Context manager

Test plan

  • Verify existing tests pass — no behavioral change, only resource management
  • In long-running deployment, confirm no file descriptor growth over time (lsof -p <pid> | wc -l)

🤖 Generated with Claude Code

Several places call `capp.connection()` without using a context manager,
creating AMQP/Redis connections that are never closed. Each leaked
connection holds open a socket and file descriptor. Under sustained use
(e.g. the Broker view refreshing, or the `/api/queues/length` endpoint
being polled by monitoring), this causes:

- Gradual file descriptor exhaustion (hitting OS `ulimit`)
- Socket leak warnings from the broker client library
- Eventual `Too many open files` errors that crash the process

Specific leaks fixed:

1. **`Flower.transport` property** — called on every request that checks
   the broker type. Opens a connection, reads the transport driver type,
   then discards the connection object without closing. Fixed by caching
   the result (transport type never changes at runtime) and using a
   context manager for the initial lookup.

2. **`print_banner()`** — `app.connection().as_uri()` opens a connection
   to format the broker URL for the startup log. Never closed. Fixed
   with a context manager.

3. **`BrokerView.get()`** — two separate `app.capp.connection()` calls:
   one to create the Broker object, another to get the display URL.
   Neither closed. Fixed by using a single context-managed connection
   for both.

4. **`GetQueueLengths.get()`** — `app.capp.connection().as_uri()`
   opens a connection to get the broker URI. Never closed. Fixed with
   a context manager.

5. **`WorkersView.get()`** — `self.application.capp.connection().as_uri()`
   opens a connection on every non-JSON page render. Never closed.
   Fixed with a context manager.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mher
Copy link
Owner

mher commented Mar 8, 2026

I suggest doing something like this instead

  from functools import cached_property

  class Flower(tornado.web.Application):
      @cached_property
      def broker_uri(self):
          with self.capp.connection() as conn:
              return conn.as_uri()

      @cached_property
      def broker_uri_with_password(self):
          with self.capp.connection() as conn:
              return conn.as_uri(include_password=True)

      @cached_property
      def transport(self):
          with self.capp.connection() as conn:
              return getattr(conn.transport, "driver_type", None)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants