Run Book
Also available as:

Creating a Profile

To create a profile, complete the following steps:

  1. Create a table within HBase that will store the profile data.

    The table name and column family must match the Profiler's configuration.

    $ /usr/hdp/current/hbase-client/bin/hbase shell
    hbase(main):001:0> create 'profiler', 'P'
  2. Define the profile in a file located at $METRON_HOME/config/zookeeper/profiler.json.

    The following example JSON creates a profile that simply counts the number of messages per ip_src_addr during each sampling interval.

      "profiles": [
          "profile": "test",
          "foreach": "ip_src_addr",
          "init":    { "count": 0 },
          "update":  { "count": "count + 1" },
          "result":  "count"

    Table 2.1. Profile Elements

    profileRequiredA unique name identifying the profile. The field is treated as a string.

    A separate profile is maintained 'for each' of these. This is effectively the entity that the profile is describing. The field is expected to contain a Stellar expression whose result is the entity name.

    For example, if ip_src_addr then a separate profile would be maintained for each unique IP source address in the data;,, etc.

    onlyifOptionalAn expression that determines if a message should be applied to the profile. A Stellar expression that returns a Boolean is expected. A message is only applied to a profile if this expression is true. This allows a profile to filter the messages that get applied to it.

    One or more Stellar expressions used to group the profile measurements when persisted. This is intended to sort the Profile data to allow for a contiguous scan when accessing subsets of the data.

    The 'groupBy' expressions can refer to any field within aorg.apache.metron.profiler.ProfileMeasurement. A common use case would be grouping by day of week. This allows a contiguous scan to access all profile data for Mondays only. Using the following definition would achieve this.

    "groupBy": [ "DAY_OF_WEEK()" ] 

    One or more expressions executed at the start of a window period. A map is expected where the key is the variable name and the value is a Stellar expression. The map can contain 0 or more variables/expressions. At the start of each window period the expression is executed once and stored in a variable with the given name.

    "init": {
      "var1": "0",
      "var2": "1"

    One or more expressions executed when a message is applied to the profile. A map is expected where the key is the variable name and the value is a Stellar expression. The map can include 0 or more variables/expressions. When each message is applied to the profile, the expression is executed and stored in a variable with the given name.

    "update": {
      "var1": "var1 + 1",
      "var2": "var2 + 1"

    A Stellar expression that is executed when the window period expires. The expression is expected to summarize the messages that were applied to the profile over the window period. The expression must result in a numeric value such as a Double, Long, Float, Short, or Integer.

    For more advanced use cases, a profile can generate two types of results. A profile can define one or both of these result types at the same time.

    • profile: A required expression that defines a value that is persisted for later retrieval.

    • triage: An optional expression that defines values that are accessible within the Threat Triage process.

    expiresOptionalA numeric value that defines how many days the profile data is retained. After this time, the data expires and is no longer accessible. If no value is defined, the data does not expire.
  3. Upload the profile definition to ZooKeeper:

    $ cd /$METRON_HOME/
    $ bin/ -m PUSH -i config/zookeeper/ -z $ZOOKEEPER_HOST:2181
  4. Start the Profiler topology:

    $ bin/
  5. Ensure that test messages are being sent to the Profiler's input topic in Kafka.

    The Profiler will consume messages from the inputTopic defined in the Profiler's configuration.

  6. Check the HBase table to validate that the Profiler is writing the profile.

    Remember that the Profiler is flushing the profile every 15 minutes. You will need to wait at least this long to start seeing profile data in HBase.

    $ /usr/hdp/current/hbase-client/bin/hbase shell
    hbase(main):001:0> count 'profiler'
    *** Output Information ***
  7. Use the Profiler Client to read the profile data.

    The following example PROFILE_GET command reads data written by the sample profile given above, if is one of the input values for ip_src_addr. For more information on using the API client, refer to Accessing Profiles.

    $ bin/stellar -z $ZOOKEEPER_HOST:2181
    [Stellar]>>> PROFILE_GET( "test", "", PROFILE_FIXED(30, "MINUTES"))
    [451, 448]