Hortonworks Docs » DataFlow 3.3.1 » Apache NiFi Toolkit
Apache NiFi Toolkit
Also available as:
PDF

Migrating Between Source and Destination ZooKeepers

Before you begin, confirm that:

  • You have installed the destination ZooKeeper cluster.

  • You have installed and configured a NiFi cluster to use the destination ZooKeeper cluster.

  • If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, you have already followed appropriate NiFi upgrade steps.

  • You have configured Kerberos as needed.

  • You have not started processing any dataflow (to avoid duplicate data processing).

  • If one of the ZooKeeper clusters you are using is configured with Kerberos, you are running the ZooKeeper Migrator from a host that has access to NiFi's ZooKeeper client jaas configuration file.

ZooKeeper Migration Steps

  1. Collect the following information:

    Required Information

    Description

    Source ZooKeeper hostname (sourceHostname)

    The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the server.N properties can be used.

    Destination ZooKeeper hostname (destinationHostname)

    The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the server.N properties can be used.

    Source ZooKeeper port (sourceClientPort)

    This can be found in <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the clientPort property.

    Destination ZooKeeper port (destinationClientPort)

    This can be found in <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the clientPort property.

    Export data path

    Determine the path that will store a json file containing the export of data from ZooKeeper. It must be readable and writable by the user running the zk-migrator tool.

    Source ZooKeeper Authentication Information

    This information is in <NiFi installation dir>/conf/state-management.xml. For NiFi 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the -a,--auth argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the -k,--krb-conf argument.

    If the state-management.xml specifies Open, no authentication is required.

    Destination ZooKeeper Authentication Information

    This information is in <NiFi installation dir>/conf/state-management.xml. For NiFi 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the -a,--auth argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the -k,--krb-conf argument.

    If the state-management.xml specifies Open, no authentication is required.

    Root path to which NiFi writes data in Source ZooKeeper (sourceRootPath)

    This information can be found in <NiFi installation dir>/conf/state-management.xml under the Root Node property in the cluster-provider element. (default: /nifi)

    Root path to which NiFi writes data in Destination ZooKeeper (destinationRootPath)

    This information can be found in <NiFi installation dir>/conf/state-management.xml under the Root Node property in the cluster-provider element.

  2. Stop all processors in the NiFi flow. If you are migrating between two NiFi installations, the flows on both must be stopped.

  3. Export the NiFi component data from the source ZooKeeper. The following command reads from the specified ZooKeeper running on the given hostname:port, using the provided path to the data, and authenticates with ZooKeeper using the given username and password. The data read from ZooKeeper is written to the file provided.

    • For NiFi 0.x

      • For an open ZooKeeper:

        • zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -f /path/to/export/zk-source-data.json

      • For a ZooKeeper using username:password for authentication:

        • zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -a <username:password> -f /path/to/export/zk-source-data.json

    • For NiFi 1.x

      • For an open ZooKeeper:

        • zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -f /path/to/export/zk-source-data.json

      • For a ZooKeeper using Kerberos for authentication:

        • zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-source-data.json

  4. (Optional) If you have used the new NiFi installation to do any processing, you can also export its ZooKeeper data as a backup prior to performing the migration.

    • For an open ZooKeeper:

      • zk-migrator.sh -r -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/zk-destination-backup-data.json

    • For a ZooKeeper using Kerberos for authentication:

      • zk-migrator.sh -r -z destinationHostname:destinationClientPort/destinationRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-destination-backup-data.json

  5. Migrate the ZooKeeper data to the destination ZooKeeper. If the source and destination ZooKeepers are the same, the --ignore-source option can be added to the following examples.

    • For an open ZooKeeper:

      • zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/zk-source-data.json

    • For a ZooKeeper using Kerberos for authentication:

      • zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-source-data.json

  6. Once the migration has completed successfully, start the processors in the NiFi flow. Processing should continue from the point at which it was stopped when the NiFi flow was stopped.

© 2012–2018, Hortonworks, Inc.
Document licensed under the Creative Commons Attribution ShareAlike 4.0 License.
Hortonworks.com | Documentation | Support | Community