MiNiFi Java Agent Administration
Also available as:
PDF

Example Config File

Below are two example config YAML files. The first tails the minifi-app.log, send the tailed log and provenance data back to a secure instance of NiFi. The second uses a series of processors to tail the app log, routes off only lines that contain "WriteAheadFlowFileRepository" and puts it as a file in the "./" directory.

MiNiFi Config Version: 1
Flow Controller:
 name: MiNiFi Flow
 comment:

Core Properties:
 flow controller graceful shutdown period: 10 sec
 flow service write delay interval: 500 ms
 administrative yield duration: 30 sec
 bored yield duration: 10 millis

FlowFile Repository:
 partitions: 256
 checkpoint interval: 2 mins
 always sync: false
 Swap:
 threshold: 20000
 in period: 5 sec
 in threads: 1
 out period: 5 sec
 out threads: 4

Provenance Repository:
 provenance rollover time: 1 min

Content Repository:
 content claim max appendable size: 10 MB
 content claim max flow files: 100
 always sync: false

Component Status Repository:
 buffer size: 1440
 snapshot frequency: 1 min

Security Properties:
 keystore: /tmp/ssl/localhost-ks.jks
 keystore type: JKS
 keystore password: localtest
 key password: localtest
 truststore: /tmp/ssl/localhost-ts.jks
 truststore type: JKS
 truststore password: localtest
 ssl protocol: TLS
 Sensitive Props:
 key:
 algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
 provider: BC

Processors:
 - name: TailFile
 class: org.apache.nifi.processors.standard.TailFile
 max concurrent tasks: 1
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 1 sec
 penalization period: 30 sec
 yield period: 1 sec
 run duration nanos: 0
 auto-terminated relationships list:
 Properties:
 File to Tail: logs/minifi-app.log
 Rolling Filename Pattern: minifi-app*
 Initial Start Position: Beginning of File

Connections:
 - name: TailToS2S
 source name: TailFile
 source relationship name: success
 destination name: 8644cbcc-a45c-40e0-964d-5e536e2ada61
 max work queue size: 0
 max work queue data size: 1 MB
 flowfile expiration: 60 sec
 queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer

Remote Processing Groups:
 - name: NiFi Flow
 comment:
 url: https://localhost:8090/nifi
 timeout: 30 secs
 yield period: 10 sec
 Input Ports:
 - id: 8644cbcc-a45c-40e0-964d-5e536e2ada61
 name: tailed log
 comments:
 max concurrent tasks: 1
 use compression: false

Provenance Reporting:
 comment:
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 30 sec
 destination url: https://localhost:8090/
 port name: provenance
 originating url: http://${hostname(true)}:8081/nifi
 use compression: true
 timeout: 30 secs
 batch size: 1000
Flow Controller:
 name: MiNiFi Flow
 comment:

Core Properties:
 flow controller graceful shutdown period: 10 sec
 flow service write delay interval: 500 ms
 administrative yield duration: 30 sec
 bored yield duration: 10 millis
 max concurrent threads: 1

FlowFile Repository:
 partitions: 256
 checkpoint interval: 2 mins
 always sync: false
 Swap:
 threshold: 20000
 in period: 5 sec
 in threads: 1
 out period: 5 sec
 out threads: 4

Content Repository:
 content claim max appendable size: 10 MB
 content claim max flow files: 100
 always sync: false

Component Status Repository:
 buffer size: 1440
 snapshot frequency: 1 min

Security Properties:
 keystore: /tmp/ssl/localhost-ks.jks
 keystore type: JKS
 keystore password: localtest
 key password: localtest
 truststore: /tmp/ssl/localhost-ts.jks
 truststore type: JKS
 truststore password: localtest
 ssl protocol: TLS
 Sensitive Props:
 key:
 algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
 provider: BC

Processors:
 - name: TailAppLog
 class: org.apache.nifi.processors.standard.TailFile
 max concurrent tasks: 1
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 10 sec
 penalization period: 30 sec
 yield period: 1 sec
 run duration nanos: 0
 auto-terminated relationships list:
 Properties:
 File to Tail: logs/minifi-app.log
 Rolling Filename Pattern: minifi-app*
 Initial Start Position: Beginning of File
 - name: SplitIntoSingleLines
 class: org.apache.nifi.processors.standard.SplitText
 max concurrent tasks: 1
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 0 sec
 penalization period: 30 sec
 yield period: 1 sec
 run duration nanos: 0
 auto-terminated relationships list:
 - failure
 - original
 Properties:
 Line Split Count: 1
 Header Line Count: 0
 Remove Trailing Newlines: true
 - name: RouteErrors
 class: org.apache.nifi.processors.standard.RouteText
 max concurrent tasks: 1
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 0 sec
 penalization period: 30 sec
 yield period: 1 sec
 run duration nanos: 0
 auto-terminated relationships list:
 - unmatched
 - original
 Properties:
 Routing Strategy: Route to 'matched' if line matches all conditions
 Matching Strategy: Contains
 Character Set: UTF-8
 Ignore Leading/Trailing Whitespace: true
 Ignore Case: true
 Grouping Regular Expression:
 WALFFR: WriteAheadFlowFileRepository
 - name: PutFile
 class: org.apache.nifi.processors.standard.PutFile
 max concurrent tasks: 1
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 0 sec
 penalization period: 30 sec
 yield period: 1 sec
 run duration nanos: 0
 auto-terminated relationships list:
 - failure
 - success
 Properties:
 Directory: ./
 Conflict Resolution Strategy: replace
 Create Missing Directories: true
 Maximum File Count:
 Last Modified Time:
 Permissions:
 Owner:
 Group:

Connections:
 - name: TailToSplit
 source name: TailAppLog
 source relationship name: success
 destination name: SplitIntoSingleLines
 max work queue size: 0
 max work queue data size: 1 MB
 flowfile expiration: 60 sec
 queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer
 - name: SplitToRoute
 source name: SplitIntoSingleLines
 source relationship name: splits
 destination name: RouteErrors
 max work queue size: 0
 max work queue data size: 1 MB
 flowfile expiration: 60 sec
 queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer
 - name: RouteToS2S
 source name: RouteErrors
 source relationship name: matched
 destination name: PutFile
 max work queue size: 0
 max work queue data size: 1 MB
 flowfile expiration: 60 sec
 queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer

Provenance Reporting:
 comment:
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 30 sec
 destination url: https://localhost:8080/
 port name: provenance
 originating url: http://${hostname(true)}:8081/nifi
 use compression: true
 timeout: 30 secs
 batch size: 1000