-
-
Notifications
You must be signed in to change notification settings - Fork 253
Description
Describe the bug
The test test_read_write_serde_v2 fails when run with Zstandard compression on environments where python-module-cramjam version 2.11.0-rc4 or newer is installed. The test fails with an AssertionError because the actual CRC value from the compressed data does not match the hardcoded expected value.
Failure Log Example:
=========================== FAILURES ===========================
__________________ test_read_write_serde_v2[zstd] __________________
request = <FixtureRequest for <Function test_read_write_serde_v2>>
compression_type = 2, crc = 1714138923
@pytest.mark.parametrize("compression_type,crc", _test_serde_v2_params)
@pytest.mark.asyncio
async def test_read_write_serde_v2(
request, compression_type: int, crc: Optional[int]
):
...
if crc is not None:
> assert reader.crc == crc
E assert 1679657554 == 1714138923
E + where 1679657554 = <aiokafka.record.default_records._DefaultRecordBatchPy object at 0x...>
tests/record/test_default_records.py:68: AssertionError
Expected behaviour
The CRC check should pass regardless of the cramjam version used, or the test should be adapted to handle the version-dependent CRC values. The core functionality of compressing and decompressing data should remain compatible.
Cause
A change in cramjam 2.11.0-rc4 alters the Zstandard compression output by including the original data size in the compressed stream's metadata. This modification causes the CRC to change from 1714138923 to 1679657554. The current hardcoded CRC value in the test suite is no longer valid for newer cramjam versions.
Proposed Solution
I've created a patch that modifies the test to handle both CRC values based on the cramjam version. This maintains backward compatibility and ensures the test passes in all environments.
Suggested change:
Dynamically set the expected CRC based on the cramjam version.
try:
_cramjam_version = tuple(map(int, cramjam.__version__.split('.')))
_zstd_crc = 1679657554 if _cramjam_version >= (2, 11, 0) else 1714138923
except (AttributeError, ValueError):
_zstd_crc = 1714138923
...
pytest.param(DefaultRecordBatch.CODEC_ZSTD, _zstd_crc, id="zstd"),Environment (please complete the following information):
- aiokafka version: 0.12.0
- Kafka Broker version: Not applicable
- Python version: 3.12.11
- cramjam version: 2.11.0
Reproducible example
Run pytest tests/record/test_default_records.py in an environment with cramjam>=2.11.0. The test will fail as shown in the log above.