Troubleshooting issues when programming against a live stream of data can be difficult.
The Stellar REPL (an interactive top level or language shell) is a powerful tool to help work
out the kinds of enrichments and transformations that are needed. The Stellar REPL can also be
used to help when developing profiles for the Profiler.
Follow these steps in the Stellar REPL to see how it can be used to help create
profiles.
-
Take a first pass at defining your profile.
As an example, in the editor copy/paste the basic "Hello, World" profile below.
[Stellar]>>> conf := SHELL_EDIT()
[Stellar]>>> conf
{
"profiles": [
{
"profile": "hello-world",
"onlyif": "exists(ip_src_addr)",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
-
Initialize the Profiler.
[Stellar]>>> profiler := PROFILER_INIT(conf)
[Stellar]>>> profiler
org.apache.metron.profiler.StandAloneProfiler@4f8ef473
-
Create a message to simulate the type of telemetry that you expect to be profiled.
As an example, in the editor copy/paste the JSON below.
[Stellar]>>> message := SHELL_EDIT()
[Stellar]>>> message
{
"ip_src_addr": "10.0.0.1",
"protocol": "HTTPS",
"length": "10",
"bytes_in": "234"
}
-
Apply some telemetry messages to your profiles. The following applies the same message
3 times.
[Stellar]>>> PROFILER_APPLY(message, profiler)
org.apache.metron.profiler.StandAloneProfiler@4f8ef473
[Stellar]>>> PROFILER_APPLY(message, profiler)
org.apache.metron.profiler.StandAloneProfiler@4f8ef473
[Stellar]>>> PROFILER_APPLY(message, profiler)
org.apache.metron.profiler.StandAloneProfiler@4f8ef473
-
Flush the Profiler to see what has been calculated.
A flush is what occurs at the end of each 15 minute period in the Profiler. The result
is a list of profile measurements. Each measurement is a map containing detailed
information about the profile data that has been generated.
[Stellar]>>> values := PROFILER_FLUSH(profiler)
[Stellar]>>> values
[{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000},
profile=hello-world, groups=[], value=3, entity=10.0.0.1}]
This profile counts the number of messages by IP source address. Notice that the value
is '3' for the entity '10.0.0.1' as we applied 3 messages with an 'ip_src_addr' of
'10.0.0.1'. There will always be one measurement for each [profile, entity] pair.
-
If you are unhappy with the data that has been generated, then 'wash, rinse and
repeat' this process. After you are satisfied with the data being generated by the
profile, then use the profile against your live, streaming data in a Metron cluster.