Kafka Output Bulk Flush Configuration Enhancement

[gogodjzhu]

The current Kafka output implementation in Beats lacks essential batch control parameters, specifically Flush.Bytes and Flush.Messages from the sarama client. This limitation causes performance issues in production environments:

Current Implementation

// libbeat/outputs/kafka/config.go#newSaramaConfig()
// Only configures MaxMessages and Frequency
k.Producer.Flush.MaxMessages = config.BulkMaxSize
if config.BulkFlushFrequency > 0 {
    k.Producer.Flush.Frequency = config.BulkFlushFrequency
}

Issues Identified

• Small Batch Requests: No byte-based or message-count-based triggers cause frequent small requests
• High Kafka CPU Usage: Many small batches increase broker processing overhead

I would like to add two new configuration options to expose sarama's Flush.Bytes and Flush.Messages.

[botelastic]

This issue doesn't have a Team:<team> label.