feat(middleware): add Redis-backed rate limiting with tier-based limits

[MohanLaksh]

Summary

Implements comprehensive Redis-backed rate limiting with tier-based endpoint protection.

Problem Statement

API endpoints currently accept unlimited requests in rapid succession. This PR adds configurable rate limiting to prevent abuse and ensure system stability.

Solution Overview

This PR implements production-grade rate limiting with:

✅ Redis-backed sliding window algorithm (graceful in-memory fallback)
✅ Endpoint-specific tiers with appropriate protection levels
✅ Multi-dimensional limiting (per-IP, per-user, per-team)
✅ SecurityLogger integration (complete audit trail)
✅ Lockout mechanism (temporary lockout after repeated violations)
✅ RFC-compliant response headers (X-RateLimit-*, Retry-After)

---

Implementation Details

1. Endpoint Tiers (Criticality-Based Protection)

Tier	Endpoints	Limit	Burst	Rationale
CRITICAL	/auth/email/*, /auth/sso/*	10/min	0	Auth endpoints - stricter limits
HIGH	/tokens/*, /oauth/*, /rbac/*	30/min	0	Admin/token management - deliberate actions
MEDIUM	/mcp, /tools/*, /llmchat/*	100/min	20	API endpoints - allow client bursts
LOW	/health, /metrics, /docs	500/min	100	Health checks - load balancer retries

Design decision: No burst allowance for auth/admin endpoints to prevent rapid repeated attempts. Burst allowed for API endpoints to support legitimate client spikes.

2. Multi-Dimensional Rate Limiting

Checks limits across three dimensions independently:

1. Per-IP (always) - Tracks request volume by source
2. Per-User (if authenticated) - Limits authenticated usage
3. Per-Team (if team context) - Enables per-tenant quotas

Example:

Client: 192.168.1.100, User: alice@example.com, Team: acme-corp

Rate limit keys checked:
- ratelimit:ip:192.168.1.100:CRITICAL
- ratelimit:user:alice@example.com:CRITICAL  
- ratelimit:team:acme-corp:CRITICAL

Block if ANY dimension exceeds limit (fail-secure)

3. Redis Sliding Window Algorithm

Uses sorted sets for precise sliding window rate limiting:

ZREMRANGEBYSCORE key 0 window_start  # Remove expired entries
ZCARD key                             # Count requests in window
ZADD key {uuid:timestamp}             # Add current request
EXPIRE key ttl                        # Auto-cleanup

Fallback: Graceful degradation to in-memory dict if Redis unavailable (single-instance dev environments).

4. SecurityLogger Integration

All rate limit violations persist to SecurityEvent table with full context:

• Event Type: rate_limit_exceeded → brute_force_attempt (on lockout)
• Severity: MEDIUM → HIGH (escalates on repeated violations)
• Context: user_id, user_email, team_id, client_ip, endpoint, tier, limit
• Threat Score: 0.5 → 0.8 (lockout detection)

Benefit: Complete audit trail for compliance, SIEM integration, pattern analysis.

5. Lockout Mechanism

After 5 rate limit violations in 5-minute window:

• ❌ 15-minute lockout (configurable)
• 📝 SecurityEvent: brute_force_attempt (HIGH severity)
• 📨 User notification: "Account locked for 15 minutes. This may indicate suspicious activity."

Design decision: Lockout instead of progressive delays to avoid blocking event loop (no asyncio.sleep() in production middleware).

6. Response Headers (RFC 6585)

All responses include rate limit headers:

X-RateLimit-Limit: 10
X-RateLimit-Remaining: 7
X-RateLimit-Reset: 1714089660

On 429 (rate limit exceeded):

Retry-After: 60
X-Lockout-Remaining: 900  (if locked out)

Benefit: Clients can implement proactive rate limiting, reducing 429 errors.

---

Technical Implementation

Architecture

Middleware Position: After HttpAuthMiddleware (Starlette LIFO execution)

• Can access request.state.user_email and request.state.team_id
• Early enough to block rate-limited requests before expensive operations

Async/Sync Bridge: ThreadPoolExecutor for sync Redis calls in async middleware

loop.run_in_executor(
    executor,
    _check_rate_limit_sync,  # Sync Redis operations
    key, limit, window
)

Pattern Reuse: Uses existing auth._get_sync_redis_client() (consistent with project patterns).

Configuration (Environment Variables)

# Enable/disable
RATE_LIMITING_ENABLED=true
RATE_LIMITING_REDIS_ENABLED=true

# Tier limits (requests per minute)
RATE_LIMIT_CRITICAL_RPM=10
RATE_LIMIT_HIGH_RPM=30
RATE_LIMIT_MEDIUM_RPM=100
RATE_LIMIT_LOW_RPM=500

# Burst allowances
RATE_LIMIT_CRITICAL_BURST=0
RATE_LIMIT_HIGH_BURST=0
RATE_LIMIT_MEDIUM_BURST=20
RATE_LIMIT_LOW_BURST=100

# Lockout configuration
RATE_LIMIT_LOCKOUT_ENABLED=true
RATE_LIMIT_LOCKOUT_THRESHOLD=5
RATE_LIMIT_LOCKOUT_DURATION_MINUTES=15

Files Changed

• mcpgateway/config.py (+25 lines) - Configuration settings
• mcpgateway/main.py (+12 lines) - Middleware registration
• mcpgateway/middleware/rate_limit_middleware.py (+407 lines) - Implementation
• tests/unit/.../test_rate_limit_middleware.py (+1051 lines) - 100+ test cases

Total: +1495 lines, 0 breaking changes

---

Testing

Coverage

✅ 100+ unit tests covering:

• Endpoint tier matching (CRITICAL/HIGH/MEDIUM/LOW)
• IP extraction (X-Forwarded-For, X-Real-IP, fallback)
• Multi-dimensional limiting (IP, User, Team)
• Redis operations + in-memory fallback
• Lockout mechanism
• SecurityLogger integration
• Response headers (success + 429)
• Configuration edge cases

Test Execution

pytest tests/unit/mcpgateway/middleware/test_rate_limit_middleware.py -v

Expected: All tests pass (100+ tests)

---

Operational Considerations

Performance Impact

Redis Path:

• Latency: ~1-2ms per request (network RTT + Redis operations)
• Acceptable overhead for added protection

In-Memory Fallback:

• Latency: ~0.1ms per request (dict lookup)
• Single-instance only (dev/test environments)

---

Backward Compatibility

✅ No breaking changes
✅ Feature flag: RATE_LIMITING_ENABLED=true (can disable if needed)
✅ Graceful degradation: In-memory fallback if Redis unavailable
✅ Existing config preserved: validation_max_requests_per_minute retained for tests

---

References

• RFC 6585: Additional HTTP Status Codes (429 Too Many Requests)
• Industry Standards: Auth0, GitHub, Stripe rate limiting patterns

---

Checklist

• Redis-backed with in-memory fallback
• SecurityLogger integration (audit trail)
• Endpoint-specific tiers (CRITICAL/HIGH/MEDIUM/LOW)
• Multi-dimensional limiting (IP + User + Team)
• Lockout mechanism (5 violations → 15 min)
• RFC-compliant response headers
• 100+ unit tests (comprehensive coverage)
• No breaking changes
• Production-ready code quality
• Documentation updated (inline comments)

---

Closes #4327

[MohanLaksh]

@brian-hussey , @ja8zyjits , Please help me review and merge this.