Administration
Also available as:
PDF
loading table of contents...

PCAP Tuning

PCAP is a Spout-only topology. Both Kafka topic consumption and HDFS writing is performed within a spout to avoid the additional network hop required if using an additional bolt.

General Storm topology properties

topology.workers=16
topology.ackers.executors: 0


                                +__Spout and Bolt properties__
                                +
                                +kafkaSpout
                                +    parallelism: 128
                                +    poll.timeout.ms=100
                                +    offset.commit.period.ms=30000
                                +    session.timeout.ms=39000
                                +    max.uncommitted.offsets=200000000
                                +    max.poll.interval.ms=10
                                +    max.poll.records=200000
                                +    receive.buffer.bytes=431072
                                +    max.partition.fetch.bytes=10000000
                                +    enable.auto.commit=false
                                +    setMaxUncommittedOffsets=20000000
                                +    setOffsetCommitPeriodMs=30000
                                +
                                +writerConfig
                                +    withNumPackets=1265625
                                +    withMaxTimeMS=0
                                +    withReplicationFactor=1
                                +    withSyncEvery=80000
                                +    withHDFSConfig
                                +        io.file.buffer.size=1000000
                                +        dfs.blocksize=1073741824
                                +
                                +