Bulk Loading Sources
Bulk loading is used to load information that does not change frequently. For example, bulk loading is ideal for loading from an asset database on a daily or even weekly basis because you don't typically change the number of assets on a network very often.
Enrichment data can be bulk loaded from the local file system, HDFS. The enrichment loader transforms the enrichment into a JSON format that is understandable to Metron. The loading framework has additional capabilities for aging data out of the enrichment stores based on time. Once the stores are loaded, you can incorporate an enrichment bolt, that can interact with the enrichment store, into the enrichment topology.
You can bulk load enrichment information from the following sources:
-
CSV Flat File Ingestion
-
HDFS via MapReduce
-
Taxii Loader
CSV File
The shell script $METRON_HOME/bin/flatfile_loader.sh
reads data from
local disk and loads the enrichment data into an HBase table. This loader uses the special
configuration parameter inputFormatHandler to specify how to consider the data. The two
implementations are BY_LINE
and
org.apache.metron.dataloads.extractor.inputformat. WholeFileFormat
.
The default is BY_LINE
, which makes sense for a list of CSVs in which
each line indicates a unit of information to be imported. However, if you are importing a
set of STIX documents, then you want each document to be considered as input to the
Extractor.
The parameters for the utility are as follows:
Short Code | Long Code | Required? | Description |
---|---|---|---|
-h | No | Generates the help screen or set of options | |
-e | --extractor_config | Yes | JSON document describing the extractor for this input data source |
-t | --hbase_table | Yes | The HBase table to import into |
-c | --hbase_cf | Yes | The HBase table column family to import into |
-i | --input | Yes | The input data location on local disk. If this is a file, then that file is loaded. If this is a directory, then the files are loaded recursively under that directory. |
-l | --log4j | No | The log4j properties file to load |
-n | --enrichment_config | No | The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. |
HDFS Through MapReduce
$METRON_HOME/bin/flatfile_loader.sh
starts the MapReduce job to load
data from HDFS to an HBase table. The following is as example of the
syntax:$METRON_HOME/bin/flatfile_loader.sh -i /tmp/top-10k.csv -t enrichment -c t -e ./extractor.json -m MR
The parameters for the utility are as follows:
Short Code | Long Code | Is Required? | Description |
---|---|---|---|
-h | No | Generate the help screen or set of options | |
-e | --extractor_config | Yes | JSON document describing the extractor for this input data source |
-t | --hbase_table | Yes | The HBase table to import to |
-c | --hbase_cf | Yes | The HBase table column family to import to |
-i | --input | Yes | The input data location on local disk. If this is a file, then that file is loaded. If this is a directory, then the files are loaded recursively under that directory. |
-l | --log4j | No | The log4j properties file to load |
-n | --enrichment_config | No | The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. |
Taxii Loader
You can use the shell script $METRON_HOME/bin/threatintel_taxii_load.sh
to poll a Taxii server for STIX documents and ingest them into HBase.
This Taxii server is often an aggregation server such as Soltra Edge.
This loader requires a configuration file describing the connection information to the Taxii server and Enrichment and Extractor configurations. The following is an example of a configuration file:
{
"endpoint" : "http://localhost:8282/taxii-discovery-service"
,"type" : "DISCOVER"
,"collection" : "guest.Abuse_ch"
,"table" : "threat_intel"
,"columnFamily" : "cf"
,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}
- endpoint
-
The URL of the endpoint
- type
-
POLL
orDISCOVER
, depending on the endpoint - collection
-
The Taxii collection to ingest.
- table
-
The HBase table to import to.
- columnFamily
-
The column family to import to.
- allowedIndicatorTypes
-
An array of acceptable threat intelligence types
The parameters for the utility are as follows:
Short Code | Long Code | Is Required? | Description |
---|---|---|---|
-h | No | Generate the help screen or set of options | |
-e | --extractor_config | Yes | JSON document describing the extractor for this input data source |
-c | --taxii_connection_config | Yes | The JSON configuration file to configure the connection |
-p | --time_between_polls | No | The time between polling the Taxii server, in milliseconds. Default: 1 hour |
-b | --begin_time | No | Start time to poll the Taxii server (all data from that point will be gathered in the first pull). The format for the date is yyyy-MM-dd HH:mm:ss. |
-l | --log4j | No | The Log4j properties to load |
-n | --enrichment_config | No | The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. |