Optimize broker queue fetching with Redis pipelining, TTL cache, and pagination

[ShubhAtWork]

Summary

With 10,000+ queues, the existing RedisBase.queues() issues one LLEN command per priority step per queue sequentially. With the default 4 priority steps, that's 40,000 individual Redis round-trips per page load — taking 30-60 seconds and creating severe latency on the Broker view and /api/queues/length endpoint.

Additionally, every page load re-fetches all queue lengths from scratch, and the API endpoint returns all queues with no pagination.

Changes

1. Redis pipelining

Batch all LLEN commands into a single Redis pipeline (non-transactional). For 10,000 queues with 4 priority steps, this reduces 40,000 round-trips to 1. Measured improvement: 30-60s -> <1s.

2. Pipeline chunking

Split pipelines into chunks of 5,000 commands to prevent Redis from blocking other clients during execution of a single massive pipeline.

3. TTL cache (--queue_cache_ttl)

Cache queue stats in memory with a configurable TTL (default 5s). Multiple concurrent requests within the TTL window share the same result. Set to 0 to disable.

4. Pagination for /api/queues/length

New limit and offset query parameters, with a total field in the response. limit=0 returns an empty list (consistent semantics), negative values return 400.

5. RedisBase.close()

Properly close Redis connections when the broker client is no longer needed. Previously, Redis connections created per-request in views were never closed.

6. RabbitMQ optimizations

• Use frozenset(names) for O(1) membership testing when filtering API responses (was O(n) list scan)
• Increased API timeouts from 1s/2s to 5s/30s to handle large queue counts
• Removed premature http_client.close() in the finally block (Tornado's AsyncHTTPClient is shared/singleton)

Test plan

• pytest tests/unit/utils/test_broker_queues.py tests/unit/test_app.py — 15 tests pass
• Verify /api/queues/length?limit=10&offset=0 returns paginated results with total
• Verify Broker view loads in <2s with 10k+ queues
• Verify --queue_cache_ttl=0 disables caching

🤖 Generated with Claude Code