Run the CSV Threat Intel Loader
After you define the threat intelligence source, threat intelligence extractor, and threat intelligence mapping configuration, you must run the loader to move the data from the threat intelligence source to the Metron threat intelligence store and to store the enrichment configuration in ZooKeeper.
$HOST_WITH_ENRICHMENT_TAGas root and navigate to
Use the loader to move the enrichment source to the enrichment store in
$METRON_HOME/bin/flatfile_loader.sh -n threatintel_config.json -i domainblocklist_ref.csv -t threatintel -c t -e threatintel_extractor_config.jsonThis command modifies the Squid enrichment config in ZooKeeper to include the threat intel enrichment.The parameters for the utility are as follows:
The data is populated into an HBase table called
- -b,--batchSize <SIZE>
- The batch size to use for HBase puts
- -c,--hbase)cf <CF>
- HBase column family to ingest the data into. This must be column family t.
- -e,--extractor_config <JSON_FILE>
- JSON Document describing the extractor for this input data source
- Generate Help screen
- -i,--input <FILE>
- The CSV File to load
- -l,--log4j <FILE>
- The log4j properties file to load
- -m,--import_mode <MODE>
- The Import mode to use: LOCAL,MR.Default: LOCAL
- -n,--enrichment_config <JSON_FILE>
- JSON Document describing the enrichment configuration details. This is used to associate an enrichment type with a field type in ZooKeeper.
- -p,--threads <NUM_THREADS>
- The number of threads to use when extracting data. The default is the number of cores of your machine.
- Do not update progress
- -t,--hbase_table <TABLE>
- HBase table to ingest the data into.
- Verify that the logs are properly ingested to HBase:
hbase shell scan 'threatintel'You should see a configuration for the sensor that looks something like the following:
- Generate some data to populate the Metron dashboard.