Enriching With Threat Intelligence Information
Also available as:
PDF

Bulk Loading CSV Threat Intelligence Sources

Although Hortonworks Cybersecurity Platform (HCP) is designed to work with STIX/Taxii threat feeds, you can also be bulk loaded with threat data from a CSV file.

CSV Flatfile Loader

The shell script $METRON_HOME/bin/flatfile_loader.sh reads data from local disk and loads the threat intelligence data into an HBase table. This loader uses the special configuration parameter inputFormatHandler to specify how to consider the data. The two implementations are BY_LINE and org.apache.metron.dataloads.extractor.inputformat. WholeFileFormat.

The default is BY_LINE, which makes sense for a list of CSVs in which each line indicates a unit of information to be imported. However, if you are importing a set of STIX documents, then you want each document to be considered as input to the Extractor.

Start the user parser topology by running the following:

$METRON_HOME/bin/start_parser_topology.sh -s user -z $ZOOKEEPER_HOST:2181 -k $KAKFA_HOST:6667

The parser topology listens for data streaming in and pushes the data to HBase. Now you have data flowing into the HBase table, but you need to ensure that the enrichment topology can be used to enrich the data flowing past.

The parameters for the utility are as follows:

Short Code Long Code Is Required? Description
-h No Generate the help screen/set of options
-e --extractor_config Yes JSON document describing the extractor for this input data source
-t --hbase_table Yes The HBase table to import into
-c --hbase_cf Yes The HBase table column family to import into
-i --input Yes The input data location on local disk. If this is a file, then that file will be loaded. If this is a directory, then the files will be loaded recursively under that directory.
-l --log4j No The log4j properties file to load
-n --enrichment_config No The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified.

HDFS via MapReduce

The shell script $METRON_HOME/bin/flatfile_loader.sh will kick off the MapReduce job to load data stated in HDFS into an HBase table. The following is as example of the syntax:
                           $METRON_HOME/bin/flatfile_loader.sh -i /tmp/top-10k.csv -t enrichment -c t -e ./extractor.json -m MR
                        

The parameters for the utility are as follows:

Short Code Long Code Is Required? Description
-h No Generate the help screen/set of options
-e --extractor_config Yes JSON document describing the extractor for this input data source
-t --hbase_table Yes The HBase table to import into
-c --hbase_cf Yes The HBase table column family to import into
-i --input Yes The input data location on local disk. If this is a file, then that file will be loaded. If this is a directory, then the files will be loaded recursively under that directory.
-l --log4j No The log4j properties file to load
-n --enrichment_config No The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified.