Skip to content

[FEATURE REQUEST] GroupMaxSize per queue #1087

@liron-navon

Description

@liron-navon

Is your feature request related to a problem? Please describe.
I work with multiple queues with one asynq server, I have multiple queues and multiple groups, we can pass "GroupMaxSize" to the asynq server, but this applies to ALL the groups, I'd like to be able to set a max size per group or per queue, since some groups might make sense to have 500 max size and others 100 max size.

Describe the solution you'd like
I'd suggest an interface like so where we can follow the "Queues" options style by setting a "Groups"
and each group can have its own options, or if it is too complicated just pass an int for the max size?

srv := asynq.NewServerFromRedisClient(
	tdb,
	asynq.Config{
		Concurrency:      10,
		GroupAggregator:  asynq.GroupAggregatorFunc(MyAggregate),
		GroupMaxDelay:    2 * time.Minute, // used as default
		GroupGracePeriod: 1 * time.Minute,  // used as default
		GroupMaxSize:     100,  // used as default
		Queues: map[string]int{
			"queue_A":         6,
			"queue_B":         2,
			"queue_C":         2
		},
		// OPTION 1
		Groups: map[string]GroupOptions{
			"queue_A": GroupOptions{
				GroupMaxDelay:    10 * time.Minute, 
				GroupGracePeriod: 2 * time.Minute,
				GroupMaxSize:     1000,
			},
			"queue_B": GroupOptions{
				GroupMaxSize:     300,
			},
		}
		// OPTION 2
		Groups: map[string]int{
			"queue_A": 1000,
			"queue_B": 300,
		}
	},
)

type GroupOptions struct {
	GroupMaxDelay      time.Duration
	GroupGracePeriod  time.Duration
	GroupMaxSize         int
}

Describe alternatives you've considered
alternative solutions are to create multiple instances of the asynq server, but that have performance panelties and makes the code more complex.

Using a single MaxSize for all the groups, but that reduces my options to optimize bulk insertions and bulk processing.

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions