Also available as:
loading table of contents...

Creating a NiFi Flow to Stream Events to HCP

This section provides instructions to create a flow to capture events from the new data source and push them into HCP.

  1. Drag the first icon on the toolbar (the processor icon) to your workspace.

    NiFi displays the Add Processor dialog box.

  2. Select the TailFile type of processor and click Add.

    NiFi displays a new TailFile processor.

    Figure 3.1. New TailFile Processor

  3. Right-click the processor icon and select Configure to display the Configure Processor dialog box.

    • In the Settings tab, change the name to Ingest $DATASOURCE Events.

      Figure 3.2. Configure Processor Dialog Box Settings Tab

    • In the Properties tab, enter the path to the datasource file in the Value column for the File(s) to Tail property:

      Figure 3.3. NiFi Configure Processor

  4. Add another processor by dragging the Processor icon to the main window.

  5. Select the PutKafka type of processor and click Add.

  6. Right-click the processor and select Configure.

  7. In the Settings tab, change the name to Stream to Metron and then click the relationship check boxes for failure and success.

    Figure 3.4. Configure Processor Settings Tab

  8. In the Properties tab, set the following three properties:

    • Known Brokers: $KAFKA_HOST:6667

    • Topic Name: $DATAPROCESSOR

    • Client Name: nifi-$DATAPROCESSOR

    Figure 3.5. Configure Processor Properties Tab

  9. Create a connection by dragging the arrow from the Ingest $DATAPROCESSOR Events processor to the Stream to Metron processor.

    NiFi displays a Create Connection dialog box.

    Figure 3.6. nifi_create_connection.png

  10. Click Add to accept the default settings for the connection.

  11. Press the Shift key and draw a box around both parsers to select the entire flow; then click the play button (green arrow).

    You should see all of the processor icons turn into green arrows.

    Figure 3.7. NiFi Dataflow

  12. Click (Start button in the Operate panel.

    Figure 3.8. Operate Panel

  13. Generate some data using the new data processor client.

    You should see metrics on the processor of data being pushed into Metron.

  14. Look at the Storm UI for the parser topology and you should see tuples coming in.

  15. After about five minutes, you should see a new Elastic Search index called $DATAPROCESSOR_index* in the Elastic Admin UI.

For more information about creating a NiFi data flow, see the NiFi documentation.