Skip to content

[Enhancement]: Global GC Manager for morpc - Align with gRPC/Thrift Best Practices #23424

Description

@XuPeng-SH

Is there an existing issue for the same feature request?

  • I have checked the existing issues.

Is your feature request related to a problem?

Currently, each morpc client creates 3 goroutines for GC tasks. With 104 clients in TN, this results in 312 goroutines, contributing to high goroutine count (1011 total).

Describe the feature you'd like

Implement a global GC manager following the same design pattern as mainstream RPC frameworks (gRPC, Thrift, Dubbo):

Current Design (Per-Client):

  • Each client: 3 goroutines (gcIdleTask, gcInactiveTask, createTask)
  • 104 clients → 312 goroutines

Proposed Design (Global Manager - Like gRPC):

  • Global GC manager: 3 shared goroutines
  • All clients register with global manager
  • Reduces goroutines: 312 → 3 (99% reduction)

Architecture Alignment with Mainstream Frameworks

gRPC Model

  • Global Transport layer manages all connections
  • Per-connection: 2 goroutines (read/write loops)
  • Connection-level GC: Unified management

Thrift Model

  • Global connection pool manager
  • Per-connection: 2 goroutines (read/write loops)
  • Connection-level GC: Unified management

morpc New Design (Proposed)

  • Global GC manager (similar to gRPC Transport layer)
  • Per-backend: 2 goroutines (read/write loops)
  • Connection-level GC: Unified management via global manager

Implementation Details

  1. Global GC Manager (pkg/common/morpc/client_gc.go)

    • Singleton pattern (like gRPC's Transport layer)
    • 3 shared goroutines for all clients
    • Thread-safe registration/unregistration
    • Copy-on-Write pattern for minimal lock contention
  2. Client Integration (pkg/common/morpc/client.go)

    • Remove per-client goroutines
    • Register with global manager on creation
    • Unregister on close
  3. Configuration

    • Environment: MORPC_GC_IDLE_INTERVAL, MORPC_GC_CHANNEL_BUFFER_SIZE
    • Config file: GCIdleCheckInterval, GCChannelBufferSize
    • Defaults: 1 second check interval, 4096 channel buffer
  4. Monitoring

    • Prometheus metrics: mo_rpc_gc_channel_drop_total
    • Grafana dashboard integration

Benefits

  • Aligns with gRPC, Thrift, Dubbo best practices
  • Reduces goroutines: 312 → 3 (99% reduction)
  • Reduces TN goroutines: 1011 → ~700 (30% reduction)
  • Maintains backward compatibility
  • Improves resource efficiency

Testing

  • 16 unit tests covering all GC manager functionality
  • Stress tests with 50 clients × 100 operations
  • Race condition detection with -race flag
  • All tests passing

Performance Metrics

  • Goroutine reduction: 312 → 3 (99%)
  • Lock contention: O(N) → O(1) via Copy-on-Write
  • Thundering herd prevention: ±10% Jitter

Aligns with industry best practices

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions