Tune Batch Indexing Kafka Partitions
The first batch indexing variable that you should determine is the minimum number of Kafka partitions required.
Use the following formula to determine the minimum number of Kafka partitions for
the enrichment toplogy:
Num Partitions=Max( t/p,t/c)
where
t
is the desired throughputp
is the maximum throughout using a single producerc
is the maximum throughput using a single consumer
Note | |
---|---|
You can estimate the minimum number of required partitions based on the
original calculations performed for enrichment topologies. However, this might or
may might be suitable as the message size increases as it progresses through
Metron topologies. This increased message size can affect throughput. For this
reason, we recommend that you perform the same steps to calculate the value for
the batch indexing topic. |