Also available as:
loading table of contents...

Enriching Telemetry Events

After you parse and normalize the raw security telemetry events, you must enrich the data elements of the normalized event. Enrichments add external data from data stores (such as Apache HBase) to make the data from the normalized event more useful and relevant.

Threat intelligence is another type of enrichment. For information about threat intelligence see Using Threat Intelligence.

Examples of enrichments are GEO where an external IP address is enriched with GeoIP information (lat/long coordinates + City/State/Country) and HOST where an IP gets enriched with Host details (for example, IP corresponds to Host X which is part of a web server farm for an e-commerce application). This information makes the data more useful and relevant.

The telemetry data sources for which HCP includes parsers (for example, Bro, Snort, and YAF) already include enrichment topologies that activate when you start the data sources in HCP.

HCP provides the following enrichment sources, but you can add your own enrichment sources to suit your needs:

  • Asset

  • GeoIP

  • User

One of the advantages of the enrichment topology is that it groups messages by HBase key. Whenever you execute a Stellar function, you can add a caching layer, thus decreasing the need to call HBase for every event.

Prior to enabling an enrichment capability within HCP, you must load the enrichment store (which for HCP is primarily HBase) with enrichment data. You can load enrichment data from the local file system or HDFS, or you can use the parser framework to stream data into the enrichment store. The enrichment loader transforms the enrichment into a JSON format that Apache Metron can use. Additionally, the loading framework can detect and remove old data from the enrichment store automatically.

After you load the stores, you can incorporate into the enrichment topology an enrichment bolt to a specific field or tag within a Metron message. The bolt can detect when it can enrich a field and then take an enrichment from the enrichment store and tag the message with it. The enrichment is then stored within the bolt's in-memory cache. HCP uses the underlying Storm routing capabilities to ensure that similar enrichment values are sent to the appropriate bolts that already have these values cached in-memory.

To configure an enrichment source, complete the following tasks:

For more information about the Metron enrichment framework, see Enrichment Framework.