kafka pubsub bindings add controlled startup seek metadata

[h-sahli]

docs(kafka): Controlled startup seek for Kafka pub/sub and binding

Summary

This change adds an opt-in startup seek capability for Kafka consumers used by pubsub.kafka and bindings.kafka.

The feature allows a component to start consuming from:

• earliest retained offset
• latest
• a fixed offset
• an offset resolved from a Unix-millis timestamp

Default behavior is unchanged when the new metadata is not set.

New metadata

Name	Type	Default	Required	Description
seekOnStart	string	never	no	Startup seek mode: never, earliest, latest, offset, timestamp
seekValue	string/int64	n/a	yes for offset/timestamp	Offset value or Unix milliseconds, depending on seekOnStart
seekApplyWhen	string	ifNoCheckpoint	no	Apply seek always or only ifNoCheckpoint (no committed offset)
seekOnce	bool	true	no	Apply startup seek once per component instance for each group/topic/partition
seekPartition	int	all claimed partitions	no	Restrict startup seek to one partition

Validation rules

• seekOnStart=earliest|latest: seekValue is ignored.
• seekOnStart=offset: seekValue must be an integer offset >=0.
• seekOnStart=timestamp: seekValue must be Unix milliseconds.
• seekApplyWhen=ifNoCheckpoint: seek only when no committed offset exists for the partition.
• seekApplyWhen=always: seek on every startup; with seekOnce=true, apply once then skip subsequent sessions.

Examples

Example A — Start from earliest once when there’s no checkpoint

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafka-pubsub
spec:
  type: pubsub.kafka
  version: v1
  metadata:
    - name: brokers
      value: "kafka:9092"
    - name: consumerID
      value: "{appID}"  # default anyway
    - name: seekOnStart
      value: "earliest"
    - name: seekApplyWhen
      value: "ifNoCheckpoint"
    - name: seekOnce
      value: "true"

Example B — Force restart from a fixed offset (single partition)

- name: seekOnStart
  value: "offset"
- name: seekValue
  value: "12345"
- name: seekPartition
  value: "0"
- name: seekApplyWhen
  value: "always"
- name: seekOnce
  value: "true"

Example C — Start from timestamp (Unix millis) across all partitions

- name: seekOnStart
  value: "timestamp"
- name: seekValue
  value: "1735689600000"  # 2025-01-01T00:00:00Z
- name: seekApplyWhen
  value: "ifNoCheckpoint"

Compatibility and migration notes

• No metadata changes required: existing behavior remains unchanged.
• pubsub.kafka: consumerID still maps to Kafka consumer group (default app-id from runtime).
• bindings.kafka: initialOffset behavior remains unchanged when startup seek metadata is not used.
• If seekOnStart != never, startup seek takes precedence over first-start initialOffset behavior.
• earliest means earliest retained offset, not necessarily 0.
• Keep consumerGroup configured in bindings when checkpointing behavior is required.

Operational notes

• Startup seeks happen during consumer group session setup after partition claims.
• Normal offset commits (MarkMessage) are unchanged to preserve at-least-once behavior.
• If seekPartition is not assigned in the session, component logs a warning and continues.
• If seekApplyWhen=always and seekOnce=false, consumers intentionally re-read on every restart.

Validation evidence (for PR)

Unit tests (repo)

• go test ./common/component/kafka ✅
• Metadata wrappers compile path check:
  • go test ./pubsub/kafka ./bindings/kafka ✅

End-to-end validation (non-PR harness)

• Runtime: custom daprd image built with this branch’s components-contrib changes
• Scenario matrix: 33 meaningful combinations of
  • seekOnStart (earliest, latest, offset, timestamp)
  • seekApplyWhen (ifNoCheckpoint, always)
  • seekOnce (true, false)
  • seekPartition (omitted/all vs 0)
  • baseline seekOnStart=never
• Assertion result: passed=33, failed=0 ✅

[Copilot]

Pull request overview

Adds an opt-in “startup seek” capability to Kafka consumers used by pubsub.kafka and bindings.kafka, allowing controlled consumption start positions (earliest/latest/fixed offset/timestamp) without changing default behavior when unset.

Changes:

• Added new component metadata keys (seekOnStart, seekValue, seekApplyWhen, seekOnce, seekPartition) for pubsub and bindings.
• Implemented startup seek parsing + consumer-group Setup-time offset resets, including committed-offset and timestamp-based offset lookup helpers.
• Added unit tests covering metadata parsing and consumer Setup behavior for startup seek.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pubsub/kafka/metadata.yaml	Documents new startup-seek metadata for the Kafka pubsub component.
bindings/kafka/metadata.yaml	Documents new startup-seek metadata for the Kafka binding (input).
common/component/kafka/metadata.go	Adds new metadata fields + wires parsing into existing Kafka metadata decode flow.
common/component/kafka/startup_seek.go	Introduces startup-seek config parsing and helper methods for committed-offset/timestamp offset lookup.
common/component/kafka/consumer.go	Hooks startup seek into consumer group Setup to reset offsets on partition claims.
common/component/kafka/kafka.go	Stores startup seek config in the Kafka component and closes the seek client on shutdown.
common/component/kafka/metadata_test.go	Adds tests validating startup seek metadata parsing/validation.
common/component/kafka/consumer_test.go	Adds tests validating startup seek behavior in consumer Setup (earliest/offset/timestamp/partition).
docs/kafka-startup-seek-docs-pr.md	Adds a PR-scoped doc describing the feature, metadata, and examples.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.