Stellar Examples

Consider the basic example of taking a message and applying a couple of enrichments, such as converting the hostname field to lowercase. For this conversion, you must specify the transformation inside of the config file for the stellar fieldMap option. Two syntaxes are supported, specifying the transformations as a map with the key as the field and the value as the Stellar expression:

"fieldMap": {
       ...
      "stellar" : {
        "config" : {
          "hostname" : "To_LOWER(hostname)"
        }
      }
    }

Let's consider a situation where you have a message containing field ip_src_addr and you want to determine if the src address is one of a few subnet ranges. You also want to store the information in a variable called is_local:

"fieldMap": {
               ...
               "stellar" : {
               "config" : {
                 "is_local := IN_SUBNET( ip_src_addr, '192.168.0.0/16', '10.10.0.0/8')"
            }
         }
      }

Now, let's consider a situation where you want to determine if the top level domain of a domain name, stored in a field called domain, is from a specific set of whitelisted TLDs:

is_government := DOMAIN_TO_TLD(domain) in [ 'mil', 'gov' ] 

Let’s assume further that the data coming in is known to be spotty with possible spaces and a dot at the end periodically due to a known upstream data ingest mistake. You can do that with three Stellar statements, the first two sanitizing the domain field and the final statement performing the whitelist check:

sanitized_domain := TRIM(domain)

sanitized_domain := if ENDS_WITH(sanitized_domain, '.') then CHOP(sanitized_domain) else sanitized_domain 

is_government := DOMAIN_TO_TLD( sanitized_domain ) in [ 'mil', 'gov' ]

Now, let’s consider a situation where you have a blacklist of known malicious domains. You can use the HCP data importer to store this data in HBase under the enrichment type malicious_domains. As data streams by, you will want to indicate whether a domain is malicious or not. Further, as before, you still have some ingestion cruft to adjust:

sanitized_domain := TRIM(domain)

sanitized_domain := if ENDS_WITH(sanitized_domain, '.') then CHOP(sanitized_domain) else sanitized_domain 

in_blacklist := ENRICHMENT_EXISTS('malicious_domains', sanitized_domains, 'enrichments', 't')