Upgrade
Also available as:
PDF

Chapter 2. Upgrade Kafka

Upgrade Apache Kafka

Upgrade each Apache Kafka node, one at a time.

You can stop each Kafka broker and upgrade the component without downtime if you have enough replication for your topic.

You can stop one Kafka broker at a time, upgrade the Kafka binaries to HDP 2.6, and start the broker, before stopping the next broker.

[Note]Note

If you are willing to accept downtime, you can take all of the brokers down, update the code, and restart all of the brokers. By default, the brokers start with the new protocol.

You can bump the protocol version and restart, at any time after the brokers are upgraded.

  1. Shut down the current Kafka daemon, switch to the new version, and start the daemon:

    su - kafka -c "/usr/hdf/current/kafka-broker/bin/kafka stop"
    hdf-select set kafka-broker 2.5.4.0-2633
    su - kafka -c "usr/hdf/current/kafka-broker/bin/kafka start"
  2. To verify that the Kafka daemon joined the cluster, create a topic and submit it to Kafka. Send a test message for that topic, and then validate that it was received by a consumer.

  3. If the upgrade process fails, follow the steps in "Downgrading Kafka" to return to your previous version of Kafka.

Downgrading Kafka

Downgrade each Kafka node one at a time. You can stop each Kafka broker and upgrade the component without downtime if you have enough replication for your topic.

  1. Shut down the current Kafka daemon, switch to the previous version, and start the daemon:

    su - kafka -c "/usr/hdp/current/kafka-broker/bin/kafka stop"
    hdp-select set kafka-broker 2.6.0.0-2041
    su - kafka -c "/usr/hdp/current/kafka-broker/bin/kafka start"
    su - kafka -c "/usr/hdf/current/kafka-broker/bin/kafka stop"
    hdf-select set kafka-broker 2.6.0.0-2041
    su - kafka -c "/usr/hdf/current/kafka-broker/bin/kafka start"
  2. To verify that the Kafka daemon joined the cluster, create a topic and submit it to Kafka. Send a test message for that topic, and then validate that it was received by a consumer.