Analytics
Also available as:
PDF

Developing Profiles

Troubleshooting issues when programming against a live stream of data can be difficult. The Stellar REPL (an interactive top level or language shell) is a powerful tool to help work out the kinds of enrichments and transformations that are needed. The Stellar REPL can also be used to help when developing profiles for the Profiler.

Follow these steps in the Stellar REPL to see how it can be used to help create profiles.

  1. Take a first pass at defining your profile.

    As an example, in the editor copy/paste the basic "Hello, World" profile below.

    [Stellar]>>> conf := SHELL_EDIT()
    [Stellar]>>> conf
    {
      "profiles": [
        {
          "profile": "hello-world",
          "onlyif":  "exists(ip_src_addr)",
          "foreach": "ip_src_addr",
          "init":    { "count": "0" },
          "update":  { "count": "count + 1" },
          "result":  "count"
        }
      ]
    }
    
  2. Initialize the Profiler.

    [Stellar]>>> profiler := PROFILER_INIT(conf)
    [Stellar]>>> profiler
    org.apache.metron.profiler.StandAloneProfiler@4f8ef473
    
  3. Create a message to simulate the type of telemetry that you expect to be profiled.

    As an example, in the editor copy/paste the JSON below.

    [Stellar]>>> message := SHELL_EDIT()
    [Stellar]>>> message
    {
      "ip_src_addr": "10.0.0.1",
      "protocol": "HTTPS",
      "length": "10",
      "bytes_in": "234"
    }
    
  4. Apply some telemetry messages to your profiles. The following applies the same message 3 times.

    [Stellar]>>> PROFILER_APPLY(message, profiler)
    org.apache.metron.profiler.StandAloneProfiler@4f8ef473
    
    [Stellar]>>> PROFILER_APPLY(message, profiler)
    org.apache.metron.profiler.StandAloneProfiler@4f8ef473
    
    [Stellar]>>> PROFILER_APPLY(message, profiler)
    org.apache.metron.profiler.StandAloneProfiler@4f8ef473
    
  5. Flush the Profiler to see what has been calculated.

    A flush is what occurs at the end of each 15 minute period in the Profiler. The result is a list of profile measurements. Each measurement is a map containing detailed information about the profile data that has been generated.

    [Stellar]>>> values := PROFILER_FLUSH(profiler)
    [Stellar]>>> values
    [{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000}, 
       profile=hello-world, groups=[], value=3, entity=10.0.0.1}]
    

    This profile counts the number of messages by IP source address. Notice that the value is '3' for the entity '10.0.0.1' as we applied 3 messages with an 'ip_src_addr' of '10.0.0.1'. There will always be one measurement for each [profile, entity] pair.

  6. If you are unhappy with the data that has been generated, then 'wash, rinse and repeat' this process. After you are satisfied with the data being generated by the profile, then follow the Metron Profiler readme to use the profile against your live, streaming data in a Metron cluster.