Introduction to Tuning HCP

Tuning your Hortonworks Cybersecurity Platform (HCP) architecture can help maximize the performance of the Apache Metron Storm topologies.

In the simplest terms, HCP powered by Apache Metron is a streaming architecture created on top of Kafka and three main types of Storm topologies: parsers, enrichment, and indexing. Each parser has its own topology and there is also a highly performant, specialized spout-only topology for streaming PCAP data to HDFS.

Streaming data is a supply and demand issue. The cluster capacity must be able to handle peak EPS (events per second) loads. When event sources produce more events than ingest can handle, the topology can fall behind. Increased unprocessed event counts can cause lag and the toplogy cannot catch up until the EPS dips below capacity.

The HCP architecture can be tuned almost exclusively using a few primary Storm and Kafka parameters along with a few Metron-specific options. You can think of the data flow as being similar to water flowing through a pipe, and the majority of these options assist in tweaking the various pipe widths in the system.