Also available as:
loading table of contents...

Streaming Data Using NiFi

NiFi provides a highly intuitive streaming user interface that is compatible with most types of data sources.

  1. Drag the icon to your workspace.

    NiFi displays the Add Processor dialog box.

  2. Select the TailFile type of processor and click Add.

    NiFi displays a new TailFile processor:

  3. Right-click (processor icon) and select Configure to display the Configure Processor dialog box:

    1. In the Settings tab, change the name to Ingest $DATASOURCE Events:

    2. In the Properties tab, enter the path to the data source file in the Value column for the File(s) to Tail property:

      Figure 3.2. NiFi Configure Processor

  4. Add another processor by dragging the Processor icon to your workspace.

  5. Select the PutKafka type of processor and click Add.

  6. Right-click the processor and select Configure.

  7. In the Settings tab, change the name to Stream to Metron and then select the relationship check boxes for failure and success.

    Figure 3.3. Configure Processor Settings Tab

  8. In the Properties tab, set the following three properties:

    Known Brokers


    Topic Name


    Client Name


    Figure 3.4. Configure Processor Properties Tab

  9. Create a connection by dragging the arrow from the Ingest $DATAPROCESSOR Events processor to the Stream to Metron processor.

    NiFi displays the Create Connection dialog box.

    Figure 3.5. Create Connection Dialog Box

  10. Click Add to accept the default settings for the connection.

  11. Press Shift and draw a box around both parsers to select the entire flow; then click the green arrow.

    All of the processor icons turn into green arrows:

    Figure 3.6. NiFi Dataflow

  12. In the Operate panel, click the arrow icon.

    Figure 3.7. Operate Panel

  13. Generate some data using the new data processor client.

  14. Look at the Storm UI for the parser topology and confirm that tuples are coming in.

  15. After about five minutes, you see a new index called $DATAPROCESSOR_index* in either the Solr Admin UI or the Elastic Admin UI.

For more information about creating a NiFi data flow, see the NiFi documentation.