Upgrade
Also available as:
PDF

Upgrading a NiFi cluster

The NiFi software configuration described in this procedure applies to both standalone and cluster node instances of NiFi software.

  1. Download the latest NiFi release.

  2. Install the new release in /opt. Your directory structure should look similar to:

    /
    |--> opt/
    |    |--> nifi-current_version
    |    |    |--> LICENSE
    |    |    |--> NOTICE
    |    |    |--> README
    |    |    |--> bin/
    |    |    |--> conf/
    |    |    |--> docs/
    |    |    |--> lib/
    |    |    |--> work/
    |    |
    |    |--> nifi-1.0.0
    |    |    |--> LICENSE
    |    |    |--> NOTICE
    |    |    |--> README
    |    |    |--> bin/
    |    |    |--> conf/
    |    |    |--> docs/
    |    |    |--> lib/
    |    |    |--> work/
    |    |        
    |    |--> Configuration_resources/
    |    |    |--> authorized-users.xml
    |    |    |--> flow.xml.gz
    |    |    |--> templates/
    |    |    |--> custom_lib/
    |
    |--> database_repository/
    |--> content_repository_1/
    |--> content_repository_N/
    |--> flowfile_repository/
    |--> NiFi-logs/
    |--> provenance_repository_1/
    |--> provenance_repository_N/
    
  3. Save the authorized-users.xml from your older NiFi version, so that NiFi 1.0.0 can use it to populate the values of the authorizations.xml and users.xml files.

  4. Using the values already configured in the following files inside the previous NiFi conf directory, update the corresponding lines in the same files under target NiFi directory (nifi-1.0.0):

    • bootstrap.conf

    • logback.xml

    • nifi.properties

    [Note]Note
    • Ensure that you make no typos when you are configuring the various paths to your existing repos and the path to the flow.xml.gz file in the nifi.properties file in the newer release.

    • Ensure that you have the same Run As user configured in the bootstrap.conf file.

    • If no new lines were added to these files, you can copy them directly from an older version to a newer. If you do copy the nifi.properties file, update the nifi.version number.

  5. Verify that all file and directory ownerships for your target NiFi directory match what you set on the previous version.

  6. Stop the NCM and all nodes running the previous NiFi version.

    [Note]Note
    • NiFi 1.0.0 implements zero-master clustering and there is no NCM in NiFi 1.0.0.

    • It is important to stop everything. You cannot upgrade and restart one node at a time. You could end up with mismatched versions of NiFi connected to your cluster NCM.

    • It is OK to stop your NiFi cluster while files still exist in your dataflows.

  7. Before you start the upgraded NiFi nodes, add the location of your authorized-users.xml file to authorizations.xml, if you want to map your previous authorized users to the new NiFi authentication model.

    <authorizer>
    <identifier>file-provider</identifier>
    <class>org.apache.nifi.authorization.FileAuthorizer</class>
    <property name="Authorizations File">./conf/authorizations.xml</property>
    <property name="Users File">./conf/users.xml</property>
    <property name="Initial Admin Identity"></property>
    <property name="Legacy Authorized Users File"></property>
  8. To support NiFi's new zero-master clustering model, ensure that the following properties are set on each node in your cluster:

    • nifi.web.http.port=<node port>

    • nifi.cluster.is.node=true

    • nifi.cluster.node.address=<fully qualified hostname of the node>

    • nifi.cluster.node.protocol.port=<node protocol port>

    • nifi.state.management.embedded.zookeeper.start=true

    • nifi.state.management.provider.cluster=zk-provider

    • nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

    • nifi.zookeeper.connect.string=<A comma-separated list of host:port pairs to connect to ZooKeeper. For example, my-zk-server1:2181,my-zk-server2:2181,my-zk-server3:2183>

    Configuration details can be found in the Clustering Configuration, the Cluster Common Properties, the Cluster Node Properties, and the ZooKeeper Properties sections of Hortonworks DataFlow Administration.

  9. Set the following NiFi properties to ensure that the cluster coordinator and other nodes have time to select the correct dataflow:

    • nifi.cluster.flow.election.max.wait.time=5 mins

    • nifi.cluster.flow.election.max.candidates=<number of NiFi nodes in the cluster>

  10. Start the upgraded NiFi nodes.

    [Note]Note

    Check the nifi-app.log for errors and for any nodes that fail to join the upgraded cluster.

  11. If you configured NiFi to run as a service, ensure that any path or links for that service are updated to point at the newly installed version executables.

  12. Configure the state-management.xml and zookeeper.properties file on every node with the following information:

    • Specify whether you want to use an embedded or external ZooKeeper server.

    • If you are using an embedded server, identify the nodes on which you want to run the Zookper server. For these nodes, be sure to also edit the state management porition of the nifi.properties file.

    See the Hortonworks DataFlow Administration for additional information on state management and using the ZooKeeper server.

  13. If you want to start using LDAP user authentication, configure the login-identity-providers.xml file, and update the nifi.properties file.

  14. If you want to setup NiFi JVM lifecycle event notifications, configure the bootstrap-notification.services.xml file and update the relevant settings in the bootstrap.conf file.

  15. Once you have verified that the new version has started, is processing data, and can be accessed via the UI, you can delete the older version.