Migrate Kafka transporter from kafkajs to @platformatic/kafka#1336
Migrate Kafka transporter from kafkajs to @platformatic/kafka#1336
Conversation
Co-authored-by: icebob <306521+icebob@users.noreply.github.com>
Co-authored-by: icebob <306521+icebob@users.noreply.github.com>
commit: |
Pull Request Test Coverage Report for Build 23388474009Details
💛 - Coveralls |
|
@copilot Fix the E2E errors: |
Co-authored-by: icebob <306521+icebob@users.noreply.github.com>
... Fixed in commit c8c4e3f. The issue was that |
|
@copilot Fix the following E2E error: |
Co-authored-by: icebob <306521+icebob@users.noreply.github.com>
... Fixed in commit 57a2e9d. The error occurs when topics already exist from a previous run. Now the code catches |
Co-authored-by: AndreMaz <9802754+AndreMaz@users.noreply.github.com>
... Implemented in commit 9f1e6d7. Now using |
|
@copilot E2E error: |
Co-authored-by: icebob <306521+icebob@users.noreply.github.com>
... Fixed in commit aeb4d5f. The error occurred because the Producer didn't have metadata for topics created via the Admin client. Enabled |
This error seems to be related to: platformatic/kafka#109 |
@copilot Check the related issue, and fix the E2E error. |
Co-authored-by: icebob <306521+icebob@users.noreply.github.com>
... Fixed in commit 7d8a2e1. After investigating the issue, the problem occurs when topics are created via the Admin client but the Producer doesn't immediately have their metadata. Added explicit metadata refresh call ( Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
|
@copilot update @platformatic/kafka to 1.18.0 that contains fix for the platformatic/kafka#109 that I've mentioned here: #1336 (comment) After that trigger the CI again current version in the PR - "@platformatic/kafka": "^1.17.0", |
Co-authored-by: AndreMaz <9802754+AndreMaz@users.noreply.github.com>
Updated to @platformatic/kafka 1.18.0 in commit f3c2d7e. This version includes the fix for the metadata handling issue referenced in platformatic/kafka#109. All unit tests pass successfully. |
- Remove duplicate types/kafka.d.ts, update src/transporters/kafka.d.ts with new options - Fix bootstrapBrokers string-to-array normalization bug - Add connection validation via admin.listTopics() in connect() - Save consumer stream reference and close it in disconnect() - Remove undocumented autocreateTopics and producer.metadata() calls - Add test for string bootstrapBrokers normalization Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The autocreateTopics option is documented in @platformatic/kafka and needed so brokers auto-create missing topics during produce. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Migration from kafkajs to @platformatic/kafka - COMPLETE ✅
Summary of Changes
The Kafka transporter has been successfully migrated from the unmaintained
kafkajspackage to the actively maintained@platformatic/kafkapackage. Key changes include:client.brokersto flatbootstrapBrokersstructurelistTopics()to check existing topics and only create missing ones (prevents errors)autocreateTopicsfor Producer to automatically handle topic metadata fetchingAll unit tests pass successfully (121 test suites, 2160 tests) and the code follows the project's linting standards.
Original prompt
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.