HDP-2.3.6 Release Notes
Also available as:
PDF

Tez

In addition to any Apache patches ported over from the 2.4.x core, HDP 2.3.6 provides Tez 0.7.0 and the following Apache patches:

  • TEZ-2097: TEZ-UI Add dag logs backend support.

  • TEZ-3114: Shuffle OOM due to EventMetaData flood.

  • TEZ-3276: Tez Example MRRSleep job fails when tez.staging-dir fs is not same as default FS.

  • TEZ-3289: Tez Example MRRSleep job does not set Staging dir correctly on secure cluster.

  • TEZ-3295: TestOrderedWordCount should handle relative input/output paths.

HDP 2.3.6 provides Tez 0.7.0 and ports the following Apache patches from the 2.4.x core:

  • TEZ-1961: Remove misleading exception "No running dag" from AM logs.

  • TEZ-2307: Possible wrong error message when submitting new DAG.

  • TEZ-2863: Container, node, and logs not available in UI for tasks that fail to launch.

  • TEZ-2886: Ability to merge AM credentials with DAG credentials.

  • TEZ-2898: Tez tools: swimlanes.py is broken.

  • TEZ-2900: Ignore V_INPUT_DATA_INFORMATION when vertex is in Failed/Killed/Error.

  • TEZ-2963: RecoveryService#handleSummaryEvent exception with HDFS transparent encryption & Kerberos authentication.

  • TEZ-3017: HistoryACLManager does not have a close method for cleanup.

  • TEZ-3025: InputInitializer creation should use the DAG GUI.

  • TEZ-3032: Incorrect start time in different events for DAG history events.

  • TEZ-3037: History URL should be set regardless of which history logging service is enabled.

  • TEZ-3066: TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services.

  • TEZ-3101: Tez UI: Task attempt log link doesn't have the correct protocol.

  • TEZ-3117: Deadlock in Edge and Vertex code.

  • TEZ-3123: Containers can get re-used even with conflicting local resources.

  • TEZ-3126: Log reason for not reducing parallelism.

  • TEZ-3128: Avoid stopping containers on the AM shutdown thread.

  • TEZ-3137: Tez task failed with illegal state exception.

  • TEZ-3156: Tez client keeps trying to talk to RM even if RM does not know about the application.

  • TEZ-3177: Non-DAG events should use the session domain or no domain if the data does not need protection.

  • TEZ-3189: Pre-warm dags should not be counted in submitted dags count by DAGAppMaster.

HDP 2.3.4.7 provided Tez 0.7.0 and the following Apache patches:

  • TEZ-3017: Thread leaks from TimelineClient.

  • TEZ-3025: InputInitializer creation should use the dag UGI.

HDP 2.3.4 provided Tez 0.7.0 and the following Apache patches:

  • TEZ-1314: Port MAPREDUCE-5821 to Tez.

  • TEZ-2398: Flaky test: TestFaultTolerance.

  • TEZ-2436: Tez UI: Add cancel button in column selector.

  • TEZ-2538: ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple partitions.

  • TEZ-2553: Tez UI: Tez UI Nits.

  • TEZ-2561: Port for TaskAttemptListenerImpTezDag should be configurable.

  • TEZ-2758: Remove append API in RecoveryService after TEZ-1909..

  • TEZ-2781: Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails.

  • TEZ-2787: Tez AM should have java.io.tmpdir=./tmp to be consistent with tasks.

  • TEZ-2808: Race condition between preemption and container assignment.

  • TEZ-2829: Tez UI: minor fixes to in-progress update of UI from AM.

  • TEZ-2850: Tez MergeManager OOM for small Map Outputs.

  • TEZ-2851: Support a way for upstream applications to pass in a caller context to Tez.

  • TEZ-2866: Tez UI: Newly added columns wont be displayed by default in tables.

  • TEZ-2874: Improved logging for caller context.

  • TEZ-2878: Tez UI: AM error handling - Make the UI handle cases in which AM returns unexpected/no data.

  • TEZ-2882: Consider improving fetch failure handling.

  • TEZ-2885: Remove counter logs from AMWebController.

  • TEZ-2887: Tez build failure due to missing dependency in pom files.

  • TEZ-2893: Tez UI: Retain vertex info displayed in DAG details page even after completion.

  • TEZ-2894: Tez UI: Disable sorting for few columns while in progress. Display an alert on trying to sort them.

  • TEZ-2895: Tez UI: Add option to enable and disable in-progress.

  • TEZ-2896: Fix thread names used during Input/Output initialization.

  • TEZ-2899: Backport graphical view fix from TEZ-2899.

  • TEZ-2907: NPE in IFile.Reader.getLength during final merge operation.

  • TEZ-2908: Tez UI: Errors are logged, but not displayed in the UI when AM fetch fails.

  • TEZ-2909: Tez UI: Application link in All DAGs table is disable when applicationhistory is unavailable.

  • TEZ-2910: Tez should invoke HDFS Client API to set up caller context.

  • TEZ-2915: Tez UI: Getting back to the DAG details page is difficult.

  • TEZ-2923: Tez Live UI counters view empty for vertices, tasks, attempts.

  • TEZ-2927: Tez UI: Graciously fail when system-metrics-publisher is disabled.

  • TEZ-2929: Tez UI: Dag details page displays vertices to be running even when dag have completed.

  • TEZ-2930: Tez UI: Parent controller is not polling at times.

  • TEZ-2933: Tez UI: Load application details from RM when available.

  • TEZ-2936: Support HDFS-based Timeline writer.

  • TEZ-2946: Tez UI: At times RM return a huge error message making the yellow error bar to fill the whole screen.

  • TEZ-2947: Tez UI: Timeline, RM : AM requests gets into a consecutive loop in counters page without any delay.

  • TEZ-2949: Allow duplicate dag names within session for Tez.

  • TEZ-2960: Tez UI: Move hardcoded URL namespace to the configuration file.

  • TEZ-2968: Counter limits exception causes AM to crash.

  • TEZ-2970: Re-localization in TezChild does not use correct UGI.

  • TEZ-2975: Bump up Apache commons dependency.

  • TEZ-2995: Timeline primary filter should only be on callerId and not type.

HDP 2.3.2 provided Tez 0.7.0 and the following Apache patches:

IMPROVEMENTS

  • TEZ-2719 Consider reducing logs in unordered fetcher with shared-fetch option.

  • TEZ-2767 Make TezMxBeanResourceCalculator the default resource calculator.

  • TEZ-2789 Backport events added in TEZ-2612 to branch-0.7.

  • TEZ-2812 Tez UI, Update task and attempt tables while in progress.

  • TEZ-2813 Tez UI, add counter data for rest api calls to AM Web Services v2.

  • TEZ-2817 Tez UI, update in progress counter data for the dag vertices and tasks table.

  • TEZ-2817 Tez UI, update in progress counter data for the dag vertices and tasks table.

  • TEZ-2830 Backport TEZ-2774 to branch-0.7. Improvements to logging in the AM and part of the runtime.

  • TEZ-2844 Backport TEZ-2775 to branch-0.7. Improve and consolidate logging in Runtime components.

  • TEZ-2876 Tez UI, Update vertex, task and attempt details page while in progress.

BUG FIXES

  • TEZ-2211 Tez UI. Allow users to configure timezone.

  • TEZ-2291 TEZ UI. Improper vertex name in tables.

  • TEZ-2483 Tez UI, Show error if in progress fails due to AM not reachable.

  • TEZ-2549 Reduce Counter Load on the Timeline Server.

  • TEZ-2602 Throwing EOFException when launching MR job.

  • TEZ-2660 Tez UI, need to show application page even if system metrics publish is disabled.

  • TEZ-2663 SessionNotRunning exceptions are wrapped in a ServiceException from a dying AM.

  • TEZ-2745 ClassNotFound in InputInitializer causes AM to crash.

  • TEZ-2752 logUnsuccessful completion in Attempt should write original finish time to ATS.

  • TEZ-2754 Tez UI. StartTime and EndTime is not displayed with right format in Graphical View.

  • TEZ-2761 Addendum fix build failure for java 6.

  • TEZ-2761 Tez UI. update the progress on the dag and vertices pages with info from AM.

  • TEZ-2766 Tez UI. Add vertex in-progress info in DAG details.

  • TEZ-2768 Log a useful error message when the summary stream cannot be closed when shutting down an AM.

  • TEZ-2780 Tez UI, Update All Tasks page while in progress.

  • TEZ-2792 Add AM web service API for tasks.

  • TEZ-2792 Addendum fix build failure for java 6.

  • TEZ-2807 Log data in the finish event instead of the start event.

  • TEZ-2812 Preemption sometimes does not respect heartbeats between pre-emptions.

  • TEZ-2816 Preemption sometimes does not respect heartbeats between pre-emptions.

  • TEZ-2825 Report progress in terms of completed tasks to reduce load on AM for Tez UI.

  • TEZ-2834 Make Tez preemption resilient to incorrect free resource reported by YARN.

  • TEZ-2842 Tez UI, Update Tez App details page while in-progress.

  • TEZ-2847 Tez UI, Task details doesn't gets updated on manual refresh after job complete.

  • TEZ-2853 Tez UI, task attempt page is coming empty.

  • TEZ-814 Improve heuristic for determining a task has failed outputs.

  • TEZ-814 Improve heuristic for determining a task has failed outputs.

INCOMPATIBLE CHANGES

  • TEZ-2768 Log a useful error message when the summary stream cannot be closed when shutting down an AM.

HDP 2.3.0 provided Tez 0.7.0 and the following Apache patches:

IMPROVEMENTS

  • TEZ-2076 Tez framework to extract/analyze data stored in ATS for specific dag.

  • TEZ-2461 tez-history-parser compile fails with hadoop-2.4.

BUG FIXES

  • TEZ-1529 ATS and TezClient integration in secure Kerberos enabled cluster.

  • TEZ-2391 TestVertexImpl timing out at times on Jenkins builds.

  • TEZ-2409 Allow different edges to have different routing plugins

  • TEZ-2447 Tez UI: Generic changes based on feedbacks.

  • TEZ-2453 Tez UI: show the dagInfo is the application has set the same.

  • TEZ-2455 Tez UI: Dag view caching, error handling and minor layout changes

  • TEZ-2460 Temporary solution for issue due to YARN-2560

  • TEZ-2474 The old taskNum is logged incorrectly when parallelism is changed

  • TEZ-2475 Fix a potential hang in Tez local mode caused by incorrectly handled interrupts.

  • TEZ-2478 Move OneToOne routing to store events in Tasks.

  • TEZ-2481 Tez UI: graphical view does not render properly on IE11

  • TEZ-2482 Tez UI: Mouse events not working on IE11

  • TEZ-2489 Disable warn log for Timeline ACL error when tez.allow.disabled.timeline-domains set to true.

  • TEZ-2504 Tez UI: tables - show status column without scrolling, numeric 0 shown as Not available

  • TEZ-2505 PipelinedSorter uses Comparator objects concurrently from multiple threads.

  • TEZ-2509 YarnTaskSchedulerService should not try to allocate containers if AM is shutting down.

  • TEZ-2513 Tez UI: Allow filtering by DAG ID on All dags table.

  • TEZ-2523 Tez UI: derive applicationId from dag/vertex id instead of relying on json data

  • TEZ-2527 Tez UI: Application hangs on entering erroneous RegEx in counter table search box

  • TEZ-2528 Tez UI: Column selector buttons gets clipped, and table scroll bar not visible in mac.

  • TEZ-2535 Tez UI: Failed task attempts link in vertex details page is broken.

  • TEZ-2539 Tez UI: Pages are not updating in IE.

  • TEZ-2541 DAGClientImpl enable TimelineClient check is wrong.

  • TEZ-2546 Tez UI: Fetch Hive query text from timeline if dagInfo is not set.

  • TEZ-2547 Tez UI: Download Data fails on secure, cross-origin clusters

  • TEZ-2548 TezClient submitDAG can hang if the AM is in the process of shutting down.

  • TEZ-2554 Tez UI: View log link does not correctly propagate login credential to read log from YARN web.

  • TEZ-2568 auto_sortmerge_join_5 fails in Tez mode