Hortonworks Cybersecurity Platform
Also available as:
PDF
loading table of contents...

Fastcapa-Kafka Configuration File

The Kafka configuration file defines how Fastcapa interacts with librdkafka.

You specify the path to the configuration file with the -c command-line argument. The file can contain any global or topic-specific, producer-focused configuration values accepted by Librdkafka.

The configuration file is a .ini-like Glib configuration file. Place the global configuration values under a [kafka-global] header and place topic-specific values under [kafka-topic].

A minimally viable configuration file only needs to include the Kafka broker to connect to:

                              [kafka-global]
metadata.broker.list = kafka-broker1:9092, kafka-broker2:9092

                           

The configuration parameters that are important for either basic functioning or performance tuning of Fastcapa include the following.

Name Description Default
metadata.broker.list Initial list of brokers as a CSV list of broker host or host:port NA
client.id Client identifier.
queue.buffering.max.messages Maximum number of messages allowed on the producer queue 100000
queue.buffering.max.ms Maximum time, in milliseconds, for buffering data on the producer queue 1000
message.copy.max.bytes Maximum size for the message to be copied to buffer. Messages larger than this are passed by reference (zero-copy) at the expense of larger iovecs. 65535
batch.num.messages Maximum number of messages batched in one MessageSet 10000
statistics.interval.ms How often statistics are emitted; 0 = never 0
compression.codec Compression codec to use for compressing message sets: none, gzip, snappy, lz4 none

Local global configuration values under the [kafka-global] header.

Locate topic configuration values under the [kafka-topic] header.

Description Default
compression.codec Compression codec to use for compressing message sets: none, gzip, snappy, lz4 none
request.required.acks How many acknowledgements the leader broker must receive from ISR brokers before responding to the request; { 0 = no ack, 1 = leader ack, -1 = all ISRs } 1
message.timeout.ms Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A value of 0 represents infinity. 300000
queue.buffering.max.kbytes Maximum total message size sum allowed on the producer queue none