Stream Data Using NiFi
NiFi provides a highly intuitive streaming user interface that is compatible with most types of data sources.
- Drag the icon to your workspace.
NiFi displays the Add Processor dialog box.
- Select the TailFile type of processor and click
NiFi displays a new TailFile processor:
- Right-click (processor icon) and select
Configure to display the Configure Processor dialog box:
In the Settings tab, change the name to
Ingest $DATASOURCE Events:
In the Properties tab, enter the path to the data
source file in the Value column for the
File(s) to Tail property:
- In the Settings tab, change the name to Ingest $DATASOURCE Events:
- Add another processor by dragging the Processor icon to your workspace.
- Select the PutKafka type of processor and click Add.
- Right-click the processor and select Configure.
- In the Settings tab, change the name to Stream to
Metron and then select the relationship check boxes for
failure and success.
- In the Properties tab, set the following three properties:
- Known Brokers
- Topic Name
- Client Name
- Create a connection by dragging the arrow from the Ingest $DATAPROCESSOR Events processor to
the Stream to Metron processor.
NiFi displays Create Connection dialog box.
- Click Add to accept the default settings for the connection.
- Press Shift and draw a box around both parsers to select the entire
flow; then click the green arrow.
All of the processor icons turn into green arrows:
- In the Operate panel, click the arrow icon.
- Generate some data using the new data processor client.
- Look at the Storm UI for the parser topology and confirm that tuples are coming in.
- After about five minutes, you see a new index called $DATAPROCESSOR_index* in either the Solr Admin UI or the Elastic Admin UI.