Upgrading HDP Manually
Also available as:
PDF
loading table of contents...

Configure and Start Apache Tez

[Note]Note

The su commands in this section use keywords to represent the Service user. For example, "hdfs" is used to represent the HDFS Service user. If you are using another name for your Service users, you will need to substitute your Service user name in each of the su commands.

[Important]Important

If you have a secure server, you will need Kerberos credentials for hdfs user access.

To upgrade Apache Tez:

  1. Copy your previously backed-up copy of tez-site.xml into the /etc/tez/conf directory.

  2. Check to see if the HDP-2.3.4 Tez tarball libraries are already in the /hdp/apps/<hdp 2.3>/tez/ version> directory - if so, then skip this step. If not, put the HDP-2.3.4 Tez tarball libraries in the /hdp/apps directory in HDFS, so that submitted Tez applications to this cluster will have access to these shared Tez tarball libraries. Execute this step on any host that has the Tez client installed.

    su - hdfs
    hdfs dfs -mkdir -p /hdp/apps/<hdp_version>/tez/
    hdfs dfs -put /usr/hdp/<hdp_version>/tez/lib/tez.tar.gz /hdp/apps/<hdp_version>/tez/
    hdfs dfs -chown -R hdfs:hadoop /hdp
    hdfs dfs -chmod -R 555 /hdp/apps/<hdp_version>/tez
    hdfs dfs -chmod -R 444 /hdp/apps/<hdp_version>/tez/tez.tar.gz

    Where <hdp_version> is the current HDP version, for example 2.3.4-$BUILD.

  3. Edit the tez.lib.uris property in the tez-site.xml file to point to /hdp/apps/<hdp_version>/tez/tez.tar.gz

    ...
    <property>
     <name>tez.lib.uris</name>
     <value>/hdp/apps/<hdp_version>/tez/tez.tar.gz</value>
    </property>
    ...

    Where <hdp_version> is the current HDP version, for example 2.3.4-$BUILD.

  4. Optional Earlier releases of Tez did not have access control. In the current version of Tez, the default behavior restricts the ability to view the Tez history to only the owner of the job. To retain unrestricted access for non-secure clusters, set tez.am.view-acls set to "*".

  5. Update the following tez-site.xml properties to their new names.

    Old Property NameNew Property Name

    tez.am.java.opts

    tez.am.launch.cmd-opts

    tez.am.env

    tez.am.launch.env

    tez.am.shuffle-vertex-manager.min-src-fraction

    tez.shuffle-vertex-manager.min-src-fraction

    tez.am.shuffle-vertex-manager.max-src-fraction

    tez.shuffle-vertex-manager.max-src-fraction

    tez.am.shuffle-vertex-manager.enable.auto-parallel

    tez.shuffle-vertex-manager.enable.auto-parallel

    tez.am.shuffle-vertex-manager.desired-task-input-size

    tez.shuffle-vertex-manager.desired-task-input-size

    tez.am.shuffle-vertex-manager.min-task-parallelism

    tez.shuffle-vertex-manager.min-task-parallelism

    tez.am.grouping.split-count

    tez.grouping.split-count

    tez.am.grouping.by-length

    tez.grouping.by-length

    tez.am.grouping.by-count

    tez.grouping.by-count

    tez.am.grouping.max-size

    tez.grouping.max-size

    tez.am.grouping.min-size

    tez.grouping.min-size

    tez.am.grouping.rack-split-reduction

    tez.grouping.rack-split-reduction

    tez.am.am.complete.cancel.delegation.tokens

    tez.cancel.delegation.tokens.on.completion

    tez.am.max.task.attempts

    tez.am.task.max.failed.attempts

    tez.generate.dag.viz

    tez.generate.debug.artifacts

    tez.runtime.intermediate-output.key.comparator.class

    tez.runtime.key.comparator.class

    tez.runtime.intermediate-output.key.class

    tez.runtime.key.class

    tez.runtime.intermediate-output.value.class

    tez.runtime.value.class

    tez.runtime.intermediate-output.should-compress

    tez.runtime.compress

    tez.runtime.intermediate-output.compress.codec

    tez.runtime.compress.codec

    tez.runtime.intermediate-input.key.secondary.comparator.class

    tez.runtime.key.secondary.comparator.class

    tez.runtime.broadcast.data-via-events.enabled

    tez.runtime.transfer.data-via-events.enabled

    tez.runtime.broadcast.data-via-events.max-size

    tez.runtime.transfer.data-via-events.max-size

    tez.runtime.shuffle.input.buffer.percent

    tez.runtime.shuffle.fetch.buffer.percent

    tez.runtime.task.input.buffer.percent

    tez.runtime.task.input.post-merge.buffer.percent

    tez.runtime.job.counters.max

    tez.am.counters.max.keys

    tez.runtime.job.counters.group.name.max

    tez.am.counters.group-name.max.keys

    tez.runtime.job.counters.counter.name.max

    tez.am.counters.name.max.keys

    tez.runtime.job.counters.groups.max

    tez.am.counters.groups.max.keys

    tez.task.merge.progress.records

    tez.runtime.merge.progress.records

    tez.runtime.metrics.session.id

    tez.runtime.framework.metrics.session.id

    tez.task.scale.memory.additional.reservation. fraction.per-io

    tez.task.scale.memory.additional-reservation.fraction.per-io

    tez.task.scale.memory.additional.reservation. fraction.max

    tez.task.scale.memory.additional-reservation.fraction.max

    tez.task.initial.memory.scale.ratios

    tez.task.scale.memory.ratios

    tez.resource.calculator.process-tree.class

    tez.task.resource.calculator.process-tree.class

    For more information on setting Tez configuration parameters in HDP-2.3.4, see Installing and Configuring Apache Tez in the Non-Ambari Cluster Installation Guide.

  6. Change the value of the tez.tez-ui.history-url.base property to the url for the upgraded Tez View. For more details, see Using the Tez View in the HDP Ambari Views Guide.