Setting Up Global Configuration
Global configurations are applied to all data sources as opposed to other configurations that are applied to a specific sensor. In other words, every message from every sensor is validated against global configuration rules. The format of the global enrichment is a JSON string-to-object map that is stored in ZooKeeper. For example:
{ "es.clustername": "metron", "es.ip": "node1", "es.port": "9300", "es.date.format": "yyyy.MM.dd.HH", "fieldValidations" : [ { "input" : [ "ip_src_addr", "ip_dst_addr" ], "validation" : "IP", "config" : { "type" : "IPV4" } } ] }
where
- es.ip
A single or collection of elastic search master nodes. They may be specified via the widely accepted
hostname:port
syntax. If a port is not specified, then a separate global propertyes.port
is required:Example:
es.ip
: [ “10.0.0.1:1234”, “10.0.0.2:1234”]Example:
es.ip
: “10.0.0.1” (thus requiringes.port
to be specified as well)Example:
es.ip
: “10.0.0.1:1234” (thus not requiringes.port
to be specified)- es.port
The port of the elastic search master node. This is not strictly required if the port is specified in the
es.ip global
property as described above. It is expected that this be an integer or a string representation of an integer.Example:
es.port
: “1234"Example:
es.port
: 1234- es.clustername
The elastic search cluster name to which you want to write.
Example:
es.clustername
: “metron” (providing your ES cluster is configured to have metron be a valid cluster name)- es.date.format
We shard the indices first by sensor and then by date. This provides the granularity time-wise that we shard.
Example:
es.date.format
: “yyyy.MM.dd.HH” (this would shard by hour creating, for example, a Bro shard of bro_2016.01.01.01, bro_2016.01.01.02, etc.)Example:
es.date.format
: “yyyy.MM.dd” (this would shard by day, creating, for example, a Bro shard of bro_2016.01.01, bro_2016.01.02, etc.)