Skip to content

Make Deliver policy configurable#10

Merged
dhanashritidke11 merged 9 commits intofeature/zgfrom
dhanashrit/AIP-9785/make-deliver-policy-configurable
Oct 22, 2025
Merged

Make Deliver policy configurable#10
dhanashritidke11 merged 9 commits intofeature/zgfrom
dhanashrit/AIP-9785/make-deliver-policy-configurable

Conversation

@dhanashritidke11
Copy link
Collaborator

@dhanashritidke11 dhanashritidke11 commented Oct 9, 2025

Description

This MR adds support for configurable JetStream deliver policies for sensor dependencies. Previously, sensors always used a hardcoded deliver policy when subscribing to JetStream subjects. This change allows users to configure the deliver policy on a per-dependency basis, enabling control over which messages are delivered from JetStream streams.

What's New

Users can now configure the deliverPolicy field in sensor dependencies when using JetStream as the EventBus:

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: example
spec:
  dependencies:
    - name: test-dep
      eventSourceName: webhook
      eventName: example
      jetStream:
        deliverPolicy: "all"  # New configuration option

Why is this needed?

The current consumer Deliver policy is hardcoded to DeliverNew. While this works well with the our existing DiscardOld+Limits eventbus config, it's not compatible with our proposed WorkQueue+DiscardNew policy and Sensor fails to connect to Eventbus with an error:
Screenshot 2025-10-09 at 4 19 39 PM

Supported Deliver Policies

Based on NATS JetStream documentation:

  • all: Start receiving from the earliest available message in the stream
  • last: Start with the last message added to the stream, or the last message matching the consumer's filter subject if defined
  • new: Start receiving messages created after the consumer was created (currently hardcoded to this)

Backward Compatibility

Fully backward compatible - no breaking changes:

  • Optional field: The jetStream configuration is completely optional
  • Default behavior preserved: When not configured, defaults to "new" policy (same as before)
  • Existing sensors work unchanged: Sensors without jetStream config continue to work exactly as before
  • No migration required: Existing deployments are unaffected

Migration Path

No migration needed. Users can opt-in to the new feature by adding the jetStream configuration to their sensor dependencies when desired.


Test Results

Deliver Policy = All

Screenshot 2025-10-09 at 3 56 09 PM Screenshot 2025-10-09 at 3 57 45 PM

Deliver Policy = New (Backward compatible)

Screenshot 2025-10-09 at 4 15 49 PM Screenshot 2025-10-09 at 4 18 07 PM

whynowy and others added 8 commits May 7, 2023 22:31
Signed-off-by: Derek Wang <whynowy@gmail.com>
* add .gitlab-ci.yml

* feat(EventSensor): Dead Letter Queue Trigger (argoproj#3199)
- without Sensor dlq, a feature in v1.9
(cherry picked from commit 2451789)

* comment out other builds

(cherry picked from commit 17070e7)

---------

Co-authored-by: aaronare <aaronare@zillowgroup.com>
Signed-off-by: Taleb Zeghmi <talebz@zillowgroup.com>
Add support for configuring JetStream deliver policy on a per-dependency
basis in sensors. This allows users to control how messages are delivered
from JetStream streams.

Changes:
- Add JetStreamConsumerConfig type with DeliverPolicy field to sensor API
- Add JetStreamDeliverPolicy constants: all, last, new
- Implement consumerOptionsForDependency() to apply deliver policy to NATS subscriptions
- Add JetStream field to eventbus.Dependency struct
- Generate deepcopy methods for new API types
- Add comprehensive unit tests
- Update documentation with examples
- Add example sensor configuration demonstrating all policies

Supported deliver policies (aligned with NATS JetStream):
- all: Start from earliest available message in stream
- last: Start with last message in stream
- new: Start with messages created after consumer (default)

Backward compatible: Defaults to new when not configured, preserving
existing behavior.
@dhanashritidke11 dhanashritidke11 marked this pull request as ready for review October 10, 2025 17:55
@dhanashritidke11 dhanashritidke11 self-assigned this Oct 10, 2025

When using JetStream as the EventBus, you can configure the deliver policy for each dependency using the `jetStream` field. This allows you to control how messages are delivered:

- **`all`**: Default policy. Start receiving from the earliest available message in the stream
Copy link
Collaborator

@abdulazillow abdulazillow Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from description.

Default behavior preserved: When not configured, defaults to "new" policy (same as before)

i think new is default from what i understand

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're correct. Currently, the argo-events code defaults to "new" since it is hardcoded in the project.
However, within Jetstream the default deliver policy is "all" which makes this documentation confusing.

Since we already have jetstream specific documentation in https://github.com/zillow/argo-events/pull/10/files#diff-4ab2c17d34a671056836f0920040e4f28863529225a12dcee63c8faf47e72ea6 I'll remove this section as it's too specific to jetstream to reside in Sensor

@dhanashritidke11 dhanashritidke11 merged commit 111404f into feature/zg Oct 22, 2025
1 check passed
dhanashritidke11 added a commit that referenced this pull request Oct 28, 2025
…olicy-configurable

Make  Deliver policy configurable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants