Apache Ambari Upgrade
Also available as:
PDF

Preparing for Upgrade to Kafka 1.0

About This Task

If you are upgrading and HDP cluster including Kafka, you must perform some manual steps to support the upgrade to Kafka 1.0 before you begin an Ambari-managed Rolling or Express upgrade.

Steps for a Rolling Upgrade

  1. In Custom kafka-broker section in Ambari Configs, click Add Property and create inter.broker.protocol.version with the following values and click Save.

    Key: inter.broker.protocol.version
    Value: CURRENT_KAFKA_VERSION
    

    Where CURRENT_KAFKA_VERSION is the version of Kafka you are currently running. For example, 0.10.1.

  2. In Custom kafka-broker section in Ambari Configs, click Add Property and createlog.message.format.version with the following values and click Save.

    Key: log.message.format.version
    Value: CURRENT_MESSAGE_FORMAT_VERSION

    Where CURRENT_MESSAGE_FORMAT_VERSION is the version of the message format you are currently using. For example, 0.10.1.

  3. Perform a rolling restart of all Kafka Brokers.

  4. Perform a Rolling upgrade of the HDP Stack to HDP 2.6.5.

  5. After the Rolling upgrade is complete, change the value for inter.broker.protocol.version to 1.0 and save the updated value.

  6. Perform rolling restart of all Kafka Brokers.

  7. After the Rolling restart is complete, change value for log.message.format.version to 1.0 and save the updated value.

  8. Perform a final Rolling restart of Kafka Brokers.

Steps for an Express Upgrade

  1. Stop Kafka.

  2. In Custom kafka-broker section in Ambari Configs, click Add Property and create inter.broker.protocol.version with the following values and click Save.

    Key: inter.broker.protocol.version
    Value: CURRENT_KAFKA_VERSION
    

    Where CURRENT_KAFKA_VERSION is the version of Kafka you are currently running. For example, 0.10.1.

  3. In Custom kafka-broker section in Ambari Configs, click Add Property and createlog.message.format.version with the following values and click Save.

    Key: log.message.format.version
    Value: CURRENT_MESSAGE_FORMAT_VERSION

    Where CURRENT_MESSAGE_FORMAT_VERSION is the version of the message format you are currently using. For example, 0.10.1.

  4. Restart all Kafka Brokers.

  5. Perform an Express upgrade to HDP 2.6.5.

  6. After the Express upgrade is complete, change the value for inter.broker.protocol.version to 1.0 and save the updated value.

  7. Perform rolling restart of all Kafka Brokers.

  8. After the Rolling restart is complete, change value for log.message.format.version to 1.0 and save the updated value.

  9. Perform a final Rolling restart of Kafka Brokers.

[Note]Note

Older Scala consumers do not support the new message format introduced in Kafka 0.11. The newer Java consumer must be used to avoid performance issues or leverage exactly-once semantics.

Additional Upgrade Guidance

In Kafka versions 0.11.0.0 and later, unclean leader election is disabled by default. If you want to retain the previous default behavior, set the broker configuration unclean.leader.election.enable to true.

If your existing cluster contains topics with the replication factor equal to one, you are encouraged to enable unclean leader election for those topics before upgrading the cluster. You can use kafka-configs.sh command to add topic specific configurations:

bin/kafka-configs.sh 
--zookeeper zk.host.name:2181 
--entity-type topics 
--entity-name TEST_TOPIC  
--alter 
--add-config unclean.leader.election.enable=true

More Information

Apache Kafka 1.0 Upgrade Guidance