Administration
Also available as:
PDF
loading table of contents...

Global Configuration

Global configurations are applied to all data sources as opposed to other configurations that are applied to a specific sensor. For example, every message from every sensor is validated against global configuration rules.

Various parts of the HCP stack use the global configuration and methods for using and modifying the global configuration properties are documented throughout the HCP documentation. The following is an index of the global configuration properties and their associated Ambari properties if they are managed by Ambari.

[Important]Important

Any property that is managed by Ambari should only be modified via Ambari. Otherwise, when you restart a service, Ambari might overwrite your updates. For more information, see Updating Properties.

Table 3.1. Global Configuration Properties

Property NameSubsystemTypeAmbari Property
es.clusternameIndexingStringes_cluster_name
es.ipIndexingStringes_hosts
es.portIndexingStringes_port
es.date.formatIndexingStringes_date_format
fieldValidationsParsingObjectN/A
parser.error.topicParsingStringN/A
stellar.function.pathsStellarCSV StringN/A
stellar.function.resolver.includesStellarCSV StringN/A
stellar.function.resolver.excludesStellarCSV StringN/A
profiler.period.durationProfilerIntegerprofiler_period_duration
profiler.period.duration.unitsProfilerStringprofiler_period_units
update.hbase.tableREST/IndexingStringupdate_hbase_table
update.hbase.cfREST-IndexingStringupdate_hbase_cf
geo.hdfs.fileEnrichmentStringgeo_hdfs_file

  1. To configure a global configuration file, create a file called global.json at $METRON_HOME/config/zookeeper.

  2. Populate the file with enrichment configurations you want to apply to all sensors.

    The file should have the following format:

    {
      "es.clustername": "metron",
      "es.ip": "node1",
      "es.port": "9300",
      "es.date.format": "yyyy.MM.dd.HH",
      "fieldValidations" : [
                  {
                    "input" : [ "ip_src_addr", "ip_dst_addr" ],
                    "validation" : "IP",
                    "config" : {
                        "type" : "IPV4"
                               }
                  }
                           ]
    }

    where

    es.ip

    A single or collection of elastic search master nodes.

    They may be specified via the widely accepted hostname:port syntax. If a port is not specified, then a separate global property es.port is required:

    • Example: es.ip : [ “10.0.0.1:1234”, “10.0.0.2:1234”]

    • Example: es.ip : “10.0.0.1” (thus requiring es.port to be specified as well)

    • Example: es.ip : “10.0.0.1:1234” (thus not requiring es.port to be specified)

    es.port

    The port of the elastic search master node.

    This is not strictly required if the port is specified in the es.ip global property as described above. It is expected that this be an integer or a string representation of an integer.

    • Example: es.port : “1234"

    • Example: es.port : 1234

    es.clustername

    The elastic search cluster name to which you want to write.

    • Example: es.clustername : “metron” (providing your ES cluster is configured to have metron be a valid cluster name)

    es.date.format

    We shard the indices first by sensor and then by date.

    This provides the granularity time-wise that we shard.

    • Example: es.date.format : “yyyy.MM.dd.HH” (this would shard by hour creating, for example, a Bro shard of bro_2016.01.01.01, bro_2016.01.01.02, etc.)

    • Example: es.date.format : “yyyy.MM.dd” (this would shard by day, creating, for example, a Bro shard of bro_2016.01.01, bro_2016.01.02, etc.)

    fieldValidations

    A validation framework that enables you to construct validation rules that cross all sensors.

    This is done in the form of validation plugins where assertions about fields or whole messages can be made.

    input

    An array of input fields or a single field. If this is omitted, then the whole messages is passed to the validator.

    config

    A String to Object map for validation configuration. This is optional if the validation function requires no configuration.

    validation

    The validation function to be used. This is one of the following:

    STELLAR

    Execute a Stellar Language statement. Expects the query string in the condition field of the config.

    IP

    Validates that the input fields are an IP address. By default, if no configuration is set, it assumes IPV4, but you can specify the type by passing in type with either IPV6 or IPV4 or by passing in a list [IPV4,IPV6] in which case the input(s) will be validated against both.

    DOMAIN

    Validates that the fields are all domains.

    EMAIL

    Validates that the fields are all email addresses.

    URL

    Validates that the fields are all URLs.

    DATE

    Validates that the fields are a date. Expects format in the config.

    INTEGER

    Validates that the fields are an integer. String representation of an integer is allowed.

    REGEX_MATCH

    Validates that the fields match a regex. Expects pattern in the config.

    NOT_EMPTY

    Validates that the fields exist and are not empty (after trimming.)

You can also create a validation using Stellar. The following validation uses Stellar to validate an ip_src_addr similar to the "validation":"IP"" example above:

"fieldValidations" : [
              {
                "validation" : "STELLAR",
                "config" : {
                    "condition" : "IS_IP(ip_src_addr, 'IPV4')"
                           }
              }
                       ]