Create a Parser for Your New Data Source by Using the Management UI

To add a new data source, you must create a parser that transforms the data source data into JSON messages suitable for downstream enrichment and indexing by HCP. Although HCP supports both Java and general-purpose parsers, you can learn the general process of creating parsers by viewing an example using the general-purpose parser Grok.

1. Determine the format of the new data source’s log entries, so you can parse them:
   1. Use ssh to access the host for the new data source.
   2. View the different log files and determine which to parse:

      sudo su - 
      cd /var/log/$NEW_DATASOURCE 
      ls

      The file you want is typically the access.log, but your data source might use a different name.
   3. Generate entries for the log that needs to be parsed so that you can see the format of the entries:

      timestamp | time elapsed | remotehost | code/status | bytes | method | URL rfc931 peerstatus/peerhost | type

2. Create a Grok statement file that defines the Grok expression for the log type you identified in Step 1.

   	Important
   	You must include timestamp in the Grok expression to ensure that the system uses the event time rather than the system time.

   Refer to the Grok documentation for additional details.
3. Launch the HCP Management module from $METRON_MANAGEMENT_UI_HOST:4200, or follow these steps:
   1. From the Ambari Dashboard, click Metron.
   2. Select the Quick Links.
   3. Select Metron Management UI.
4. Launch the Management UI.
5. Under Operations, click Sensors.
6. Click to view the new sensor panel:

   
   
   

7. In the NAME field, enter the name of the new sensor.
8. In the Kafka Type field, enter the name of the new sensor.
9. In the Parser Type field, choose the type of parser for the new sensor (in this example task, Grok).
   Don't worry if you see "No Matching Kafka Topic." The Kafka topic will be created automatically when you save.
10. Enter a Grok statement for the new parser:
    1. In the Grok Statement box, click (expand window) to display the Grok validator panel:
       
    2. For SAMPLE, enter a sample log entry for the data source.
    3. For STATEMENT, enter the Grok statement you created for the data source.
       The Management UI automatically completes partial words in your Grok statement as you enter them.

       	Note
       	You must include timestamp to ensure that the system uses the event time rather than the system time.

    4. Click TEST.

       If the validator finds an error, it displays the error information; otherwise, the valid mapping displays in the PREVIEW field.

       Consider repeating substeps a through c to ensure that your Grok statement is valid for all sensor logs.
    5. Click SAVE to save the sensor information and add it to the list of sensors.
11. Click the pencil icon to edit the sensor you just added.
12. Scroll down to the Parser Config section.
13. In the first open field, indicated by enter field, enter timestampField.
    
14. In next open field, enter timestamp.
15. Click Save.
16. Continue to build and test the Grok statement until you have entries for each element in the log entry.