[KIP-932] Fix for green build on CI by Ankith-Confluent · Pull Request #5352 · confluentinc/librdkafka

Ankith L (Ankith-Confluent) · 2026-03-09T06:12:04Z

Fix mock broker share session management: add per-broker session lookup by NodeId, correct epoch mismatch handling to return INVALID_SHARE_SESSION_EPOCH, and add proper ShareFetch/ShareAcknowledge validation (epoch-0 ack rejection, epoch=-1 topic add/forget rejection). Update Kafka version to 4.2.0, add CI pipeline for share consumer integration tests (0170/0171), fix compaction test segment.bytes for Kafka 4.1+.

segment.bytes=10000 → 1048576 and segment.ms=10000 → 500

As mentioned above, segment.bytes=10000 fails on Kafka 4.x due to the 1MB minimum. segment.ms was also reduced from 10s to 500ms to compensate — since segments are now much larger (1MB vs 10KB), they won't fill up as quickly from small messages, so we need the time-based roll to trigger sooner to keep the test fast.

Message size increased from 1000/1024 bytes → 60000 bytes (60KB)

The compaction test works by filling a segment with enough data to force a segment roll, which is what triggers compaction. With the old 10KB segment.bytes, tiny 1KB messages were enough to fill a segment. Now that segment.bytes is 1MB, the same small messages would require ~1000 of them to fill a segment. Instead, 60KB messages were used so that ~20 messages (~1.2MB) exceed the 1MB segment limit and force a roll, keeping the test behaviour (and runtime) roughly the same.

confluent-cla-assistant · 2026-03-09T06:12:16Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

…borted messages tests

…tests

…ating member active state logging

…it and ensure proper segment rolls

…ration

…guration

…meter

…on for all tests with SASL

…tion

…n share group tests

…lookup and management

…cript

…on timeout test

… disable until fixes are applied

…istency

…applied

…re-enabling tests

…bility

… for share group fetch mock

…hare group tests

… consumer test job name

…artbeat tests

…ance

…/m1 share consumer tests

…umer tests

…n for OSX arm64/m1

Kaushik Raina (k-raina)

First level of reviews

Kaushik Raina (k-raina) · 2026-03-17T12:52:09Z

.semaphore/semaphore.yml

+        commands:
+          - '[[ -z $DOCKERHUB_APIKEY ]] || docker login --username $DOCKERHUB_USER --password $DOCKERHUB_APIKEY'
+      jobs:
+        - name: 'Build and run share consumer tests (0170+)'


(0170+)
Will this semaphore not run all tests? Any reason why we are only using semaphore jobs to run share consumer tests seperately?

It will only run tests >= 170

This was decided to run the share consumer tests on every commit, we dont want to run all the tests for every commit since that will increase the cost/time required for green build.

Kaushik Raina (k-raina) · 2026-03-17T12:53:52Z

tests/0156-share_consumer_fetch_mock.c

Should tests be renamed with 170x numbering?

This was decided to have this change in a separate PR.

Kaushik Raina (k-raina) · 2026-03-17T12:54:42Z

.semaphore/semaphore.yml

+        commands:
+          - '[[ -z $DOCKERHUB_APIKEY ]] || docker login --username $DOCKERHUB_USER --password $DOCKERHUB_APIKEY'
+      jobs:
+        - name: 'Build and run share consumer tests (0170+)'


Suggested change

- name: 'Build and run share consumer tests (0170+)'

- name: 'Build and run share consumer tests'

Kaushik Raina (k-raina) · 2026-03-17T12:56:02Z

.semaphore/semaphore.yml

+            - export JAVA_HOME=$(/usr/libexec/java_home -v 21)
+            - python3 -m pip install tests/trivup/trivup-0.14.0.tar.gz jsoncomment
+            - cache restore trivup-kafka-osx-${KAFKA_VERSION}-${CACHE_TAG}
+            - ./configure --install-deps --source-deps-only --enable-devel


Why is --enable-devel needed?

I had added it for testing purposes, I will remove it

Kaushik Raina (k-raina) · 2026-03-17T12:56:50Z

.semaphore/semaphore.yml

+            - ./configure --install-deps --source-deps-only --enable-devel
+            - make -j all
+            - make -j -C tests build
+            - (cd tests && python3 -m trivup.clusters.KafkaCluster --kraft


Some semaphore blocks are using run-share-consumer-tests.sh and some or not. Why this inconsistency ?

For mac I wad using trivup directly, I will change it to use run-share-consumer-tests.sh

Kaushik Raina (k-raina) · 2026-03-17T13:04:22Z

src/rdkafka_mock_sharegrp.c

                                continue;

-                        state->state = RD_KAFKA_MOCK_SGRP_RECORD_AVAILABLE;
+                        /* Per KIP-932: if delivery count has reached


KIP-932 Maybe remove for rest of files in PR?

Kaushik Raina (k-raina) · 2026-03-17T13:09:22Z

tests/0077-compaction.c

            "--config cleanup.policy=compact "
-            "--config segment.ms=10000 "
-            "--config segment.bytes=10000 "
+            "--config segment.ms=500 "


Could you please mention in PR summary details, reason for these changes? Will help in review and future context

Kaushik Raina (k-raina) · 2026-03-17T13:10:10Z

tests/0129-fetch_aborted_msgs.c

        test_admin_create_topic(rk, topic, 1, 1,
                                (const char *[]) {"max.message.bytes", "10000",
-                                                  "segment.bytes", "20000",
+                                                  "segment.bytes", "1048576",


Was there any change in AK or Clients which requires increase in segment bytes?

Yes AK 4.2 requires segment.bytes to be 1MB minimum

Kaushik Raina (k-raina) · 2026-03-17T13:12:59Z

tests/0155-share_group_heartbeat_mock.c

-
-        TEST_CALL_ERR__(rd_kafka_assignment(c1, &c1_assignment));
-        TEST_ASSERT(c1_assignment->cnt == 3,
+        cnt = wait_assignment_count(share_c1, 3, 15000);


cnt = wait_assignment_count(share_c1, 3, 15000);

In other calls we use 10000 as timeout, any reason why we use 15000 here?

I had added it while it was getting stuck, I will revert

Kaushik Raina (k-raina) · 2026-03-17T13:24:31Z

tests/test.c

    _TEST(0155_share_group_heartbeat_mock, TEST_F_LOCAL),
-    _TEST(0156_share_group_fetch_mock, TEST_F_LOCAL),
-    _TEST(0157_share_group_ack_mock, TEST_F_LOCAL),
+    _TEST(0156_share_group_fetch_mock, TEST_F_MANUAL),


share_group_fetch_mock Maybe change to share_consumer_fetch_mock?

…ompatibility

…andling functions

- Updated test file names from `share_group_fetch_mock` to `share_consumer_fetch_mock` - Updated test file names from `share_group_ack_mock` to `share_consumer_ack_mock` - Adjusted corresponding test declarations in `test.c` to reflect the new naming convention

…ency

airlock-confluentinc bot force-pushed the dev_kip-932_ci_integration_tests_fixes branch 3 times, most recently from a777d8a to 7a23d48 Compare March 13, 2026 16:17

Ankith L (Ankith-Confluent) marked this pull request as ready for review March 13, 2026 17:14

Ankith L (Ankith-Confluent) requested a review from a team as a code owner March 13, 2026 17:14

Ankith L (Ankith-Confluent) changed the title ~~Increase segment.bytes configuration to 1MB in compaction and fetch a…~~ [KIP-932] Fix for green build on CI Mar 14, 2026

Ankith L (Ankith-Confluent) added 23 commits March 17, 2026 13:58

Increase segment.bytes configuration to 1MB in compaction and fetch a…

2b5cd3e

…borted messages tests

Replace usleep with rd_usleep for better precision in session expiry …

6d93639

…tests

Update Kafka version to 4.2.0 in CI configurations and tests

313a1cc

Refactor mock group member handling by removing common header and upd…

ecbc8e8

…ating member active state logging

Add new mock tests for group heartbeat, fetch, and acknowledgment

1ea98c1

Increase message size in compaction tests to exceed segment.bytes lim…

ac3b552

…it and ensure proper segment rolls

Make TEST_SASL parameter required and add default value in CI configu…

362ede5

…ration

Make TEST_SASL parameter required in CI configuration

5343217

Add SASL test blocks for x86_64 and aarch64 architectures in CI confi…

bccac02

…guration

Make TEST_SASL parameter optional in CI configuration

05be8e3

Make TEST_SASL parameter optional in CI configuration

70394bd

Refactor SASL test configuration and remove deprecated TEST_SASL para…

31da574

…meter

Refactor SASL test configuration: rename pipeline and add new promoti…

80b0603

…on for all tests with SASL

Remove SASL test configuration and related parameters from CI integra…

69cadfe

…tion

Refactor ShareFetch error handling and session validation in mock tests

c981e89

Test 155 fixes

4adfdd0

Add wait functions for assignment count and fatal error propagation i…

6d994b3

…n share group tests

Enhance ShareFetch session validation to include Node ID for session …

0c42958

…lookup and management

Refactor tests to use SUB_TEST_QUICK for faster execution

02f282e

Reduce wait times in share group tests for improved execution speed

71a3555

style fixes

a7de3fb

Add rdunittest_fetcher.c to the list of source files in CMakeLists.txt

f7a5ece

Add share consumer tests to CI configuration and create test runner s…

00d0b27

…cript

Ankith L (Ankith-Confluent) added 19 commits March 17, 2026 13:58

tests: Initialize share_c1_initial and share_c2_initial to 0 in sessi…

9e4b217

…on timeout test

tests: Update test flags for share group heartbeat and fetch mocks to…

675e442

… disable until fixes are applied

style: Fix spacing in test macro definition for consistency

4b0ad33

tests: Update test macros for consistency and improve readability

18d3dec

tests: Refactor subscription macros for improved readability and cons…

08805de

…istency

tests: Add high volume and multi-topic tests for share consumer

428336c

tests: Update share group mock tests to run manually until fixes are …

787c6a1

…applied

tests: Improve heartbeat and assignment handling in share consumer tests

900fb58

tests: Enhance share consumer tests by adjusting response delays and …

dac9db1

…re-enabling tests

tests: Update share group fetch mock to run manually for improved sta…

d8933fc

…bility

tests: Add SUB_TEST macros for improved test structure and clarity

66a9b56

tests: Adjust message production count and update test execution mode…

250f155

… for share group fetch mock

tests: Replace SUB_TEST with SUB_TEST_QUICK for faster execution in s…

0436f51

…hare group tests

tests: Update share group fetch mock to run manually and adjust share…

ecd20b4

… consumer test job name

tests: Refactor polling comments to improve clarity in share group he…

8cf9f34

…artbeat tests

style fix

268af9f

tests: Add share consumer tests for OSX arm64/m1 and Linux aarch64

8c148c3

tests: Update Python package installation for test dependencies

fed57c0

tests: Enhance caching and setup for OSX arm64/m1 Kafka tests

e007663

airlock-confluentinc bot force-pushed the dev_kip-932_ci_integration_tests_fixes branch from 213b104 to e007663 Compare March 17, 2026 08:32

Ankith L (Ankith-Confluent) added 5 commits March 17, 2026 14:03

tests: Replace SUB_TEST with SUB_TEST_QUICK for improved test perform…

db2f63f

…ance

tests: Update Java installation for share consumer tests to OpenJDK 21

0cd6af3

tests: Add Java version check and improve error logging for OSX arm64…

0eda3c7

…/m1 share consumer tests

tests: Update Python package installation for OSX arm64/m1 share cons…

2ddafb7

…umer tests

tests: Remove redundant Kafka download step and improve test executio…

fb9d444

…n for OSX arm64/m1

Kaushik Raina (k-raina) requested changes Mar 17, 2026

View reviewed changes

Ankith L (Ankith-Confluent) added 4 commits March 17, 2026 19:23

tests: Refactor share consumer tests setup for improved clarity and c…

fad45ea

…ompatibility

tests: Remove KIP-932 references from comments for clarity in share h…

8026076

…andling functions

tests: Rename main functions in share consumer mock tests for consist…

2c1c3e7

…ency

	- name: 'Build and run share consumer tests (0170+)'
	- name: 'Build and run share consumer tests'

Conversation

Ankith L (Ankith-Confluent) commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

confluent-cla-assistant bot commented Mar 9, 2026

Uh oh!

Kaushik Raina (k-raina) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Ankith L (Ankith-Confluent) commented Mar 9, 2026 •

edited

Loading