Release Notes
Also available as:
PDF

Tez

HDP 2.5.0 provides Tez 0.7.0 and the following Apache patches:

  • TEZ-814: Improve heuristic for determining a task has failed output.

  • TEZ-1248: Reduce slow-start should special case 1 reducer runs.

  • TEZ-1314: Port MAPREDUCE-5821 to Tez.

  • TEZ-1529: ATS and TezClient integration in secure kerberos enabled cluster.

  • TEZ-1911: MergeManager's unconditionalReserve() should check for memory limits before allocating.

  • TEZ-1961: Remove misleading exception "No running dag" from AM log.

  • TEZ-2076: Tez framework to extract/analyze data stored in ATS for specific dag.

  • TEZ-2097: TEZ-UI Add dag logs backend support.

  • TEZ-2198: Fix sorter spill count.

  • TEZ-2211: Tez UI: Allow users to configure timezone.

  • TEZ-2291: TEZ UI: Improper vertex name in tables.

  • TEZ-2307: Possible wrong error message when submitting new dag.

  • TEZ-2311: AM can hang if kill received while recovering from previous attempt.

  • TEZ-2391: TestVertexImpl timing out at times on Jenkins builds.

  • TEZ-2398: Flaky test: TestFaultTolerance.

  • TEZ-2409: Allow different edges to have different routing plugin.

  • TEZ-2436: Tez UI: Add cancel button in column selector.

  • TEZ-2440: Sorter should check for indexCacheList.size() in flush(.

  • TEZ-2447: Tez UI: Generic changes based on feedbacks.

  • TEZ-2453: Tez UI: show the dagInfo is the application has set the same.

  • TEZ-2455: Tez UI: Dag view caching, error handling and minor layout change.

  • TEZ-2460: Temporary solution for issue due to YARN-256.

  • TEZ-2461: tez-history-parser compile fails.

  • TEZ-2468: Change the minimum Java version to Java 7.

  • TEZ-2474: The old taskNum is logged incorrectly when parallelism is changed.

  • TEZ-2475: Fix a potential hang in Tez local mode caused by incorrectly handled interrupts.

  • TEZ-2478: Move OneToOne routing to store events in Tasks.

  • TEZ-2481: Tez UI: graphical view does not render properly on IE1.

  • TEZ-2482: Tez UI: Mouse events not working on IE1.

  • TEZ-2483: TEZ-2843 Tez UI: Show error if in progress fails due to AM not reachable.

  • TEZ-2489: Disable warn log for Timeline ACL error when tez.allow.disabled.timeline-domains set to true.

  • TEZ-2504: Tez UI: tables - show status column without scrolling, numeric 0 shown as Not available.

  • TEZ-2505: PipelinedSorter uses Comparator objects concurrently from multiple threads.

  • TEZ-2509: YarnTaskSchedulerService should not try to allocate containers if AM is shutting down.

  • TEZ-2513: Tez UI: Allow filtering by DAG ID on All dags table.

  • TEZ-2523: Tez UI: derive applicationId from dag/vertex id instead of relying on json date.

  • TEZ-2527: Tez UI: Application hangs on entering erroneous RegEx in counter table search bot.

  • TEZ-2528: Tez UI: Column selector buttons gets clipped, and table scroll bar not visible in mac.

  • TEZ-2535: Tez UI: Failed task attempts link in vertex details page is broken.

  • TEZ-2538: ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple partition.

  • TEZ-2539: Tez UI: Pages are not updating in IE.

  • TEZ-2541: DAGClientImpl enable TimelineClient check is wrong.

  • TEZ-2545: It is not necessary to start the vertex group commit when DAG is in TERMINATING

  • TEZ-2546: Tez UI: Fetch hive query text from timeline if dagInfo is not set.

  • TEZ-2547: Tez UI: Download Data fails on secure, cross-origin cluster.

  • TEZ-2548: TezClient submitDAG can hang if the AM is in the process of shutting down.

  • TEZ-2549: Reduce Counter Load on the Timeline Server.

  • TEZ-2552: CRC errors can cause job to run for very long time in large jobs.

  • TEZ-2553: Tez UI: Tez UI Nits.

  • TEZ-2554: Tez UI: View log link does not correctly propagate login credential to read log from yarn web.

  • TEZ-2560: Fix Tez-ui build for maven 3.3+.

  • TEZ-2561: Port for TaskAttemptListenerImpTezDag should be configurable.

  • TEZ-2567: Tez UI: download dag data does not work within ambari.

  • TEZ-2568: V_INPUT_DATA_INFORMATION may happen after vertex is initialized.

  • TEZ-2575: Handle KeyValue pairs size which do not fit in a single block in PipelinedSorte.

  • TEZ-2579: Incorrect comparison of TaskAttemptId.

  • TEZ-2602: Throwing EOFException when launching MR job.

  • TEZ-2629: LimitExceededException in Tez client when DAG has exceeds the default max counters.

  • TEZ-2635: Limit number of attempts being downloaded in unordered fetch.

  • TEZ-2636: MRInput and MultiMRInput should work for cases when there are 0 physical inputs.

  • TEZ-2660: Tez UI: need to show application page even if system metrics publish is disabled.

  • TEZ-2662: Provide a way to check whether AM or task opts are valid and error if not.

  • TEZ-2663: SessionNotRunning exceptions are wrapped in a ServiceException from a dying AM.

  • TEZ-2687: Tez should release/kill all held containers before stopping services during the shutdown phase.

  • TEZ-2719: Consider reducing logs in unordered fetcher with shared-fetch option.

  • TEZ-2730: tez-api missing dependency on org.codehaus.jettison for json.

  • TEZ-2732: DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffer.

  • TEZ-2734: Add a test to verify the filename generated by OnDiskMerge.

  • TEZ-2742: VertexImpl.finished() terminationCause hides member var of the same name.

  • TEZ-2745: ClassNotFound in InputInitializer causes AM to crash.

  • TEZ-2752: logUnsuccessful completion in Attempt should write original finish time to ATS.

  • TEZ-2754: Tez UI: StartTime & EndTime is not displayed with right format in Graphical View.

  • TEZ-2756: MergeManager close should not try merging files on close if invoked after a shuffle exception.

  • TEZ-2758: Remove append API in RecoveryService after TEZ-190.

  • TEZ-2761: Addendum fix build failure for java.

  • TEZ-2761: Tez UI: update the progress on the dag and vertices pages with info from AM.

  • TEZ-2766: Tez UI: Add vertex in-progress info in DAG detail.

  • TEZ-2767: Make TezMxBeanResourceCalculator the default resource calculator.

  • TEZ-2768: Log a useful error message when the summary stream cannot be closed when shutting down an AM.

  • TEZ-2780: Tez UI: Update All Tasks page while in progress.

  • TEZ-2781: Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails.

  • TEZ-2787: Tez AM should have java.io.tmpdir=./tmp to be consistent with tasks.

  • TEZ-2789: Backport events added in TEZ-2612.

  • TEZ-2792: Add AM web service API for task.

  • TEZ-2792: Addendum fix build failure for java.

  • TEZ-2807: Log data in the finish event instead of the start event.

  • TEZ-2808: Race condition between preemption and container assignment.

  • TEZ-2812: Tez UI: Update task & attempt tables while in progress.

  • TEZ-2813: Tez UI: add counter data for rest api calls to AM Web Services v2.

  • TEZ-2816: Preemption sometimes does not respect heartbeats between preemption.

  • TEZ-2817: Tez UI: update in progress counter data for the dag vertices and tasks table.

  • TEZ-2825: Report progress in terms of completed tasks to reduce load on AM for Tez U.

  • TEZ-2829: Tez UI: minor fixes to in-progress update of UI from AM.

  • TEZ-2830: Backport TEZ-2774. Improvements to logging in the AM and part of the runtime.

  • TEZ-2834: Make Tez preemption resilient to incorrect free resource reported by YARN.

  • TEZ-2842: Tez UI: Update Tez App details page while in-progress.

  • TEZ-2844: Backport TEZ-2775. Improve and consolidate logging in Runtime components.

  • TEZ-2846: Flaky test: TestCommit.testVertexCommit_OnDAGSuccess.

  • TEZ-2847: Tez UI: Task details doesn't gets updated on manual refresh after job complete.

  • TEZ-2850: Tez MergeManager OOM for small Map Outputs.

  • TEZ-2851: Support a way for upstream applications to pass in a caller context to Tez.

  • TEZ-2853: Tez UI: task attempt page is coming empty.

  • TEZ-2857: Fix flakey tests in TestDAGImpl.

  • TEZ-2863: Container, node, and logs not available in UI for tasks that fail to launch.

  • TEZ-2866: Tez UI: Newly added columns wont be displayed by default in table.

  • TEZ-2868: Fix setting Caller Context in Tez Examples.

  • TEZ-2874: Improved logging for caller context.

  • TEZ-2876: Tez UI: Update vertex, task & attempt details page while in progress.

  • TEZ-2878: Tez UI: AM error handling - Make the UI handle cases in which AM returns unexpected/no date.

  • TEZ-2882: Consider improving fetch failure handling.

  • TEZ-2885: Remove counter logs from AMWebController.

  • TEZ-2886: Ability to merge AM credentials with DAG credentials.

  • TEZ-2887: Tez build failure due to missing dependency in pom files.

  • TEZ-2893: Tez UI: Retain vertex info displayed in DAG details page even after completion.

  • TEZ-2894: Tez UI: Disable sorting for few columns while in progress. Display an alert on trying to sort them.

  • TEZ-2895: Tez UI: Add option to enable and disable in-progress.

  • TEZ-2896: Fix thread names used during Input/Output initialization.

  • TEZ-2898: Tez tools : swimlanes.py is broken.

  • TEZ-2899: Backport graphical view fix from TEZ-2899.

  • TEZ-2900: Ignore V_INPUT_DATA_INFORMATION when vertex is in Failed/Killed/Error.

  • TEZ-2907: NPE in IFile.Reader.getLength during final merge operation.

  • TEZ-2908: Tez UI: Errors are logged, but not displayed in the UI when AM fetch fails.

  • TEZ-2909: Tez UI: Application link in All DAGs table is disable when applicationhistory is unavailable.

  • TEZ-2910: Tez should invoke HDFS Client API to set up caller context.

  • TEZ-2915: Tez UI: Getting back to the DAG details page is difficult.

  • TEZ-2923: Tez Live UI counters view empty for vertices, tasks, attempt.

  • TEZ-2927: Tez UI: Graciously fail when system-metrics-publisher is disabled.

  • TEZ-2929: Tez UI: Dag details page displays vertices to be running even when dag have completed.

  • TEZ-2930: Tez UI: Parent controller is not polling at time.

  • TEZ-2933: Tez UI: Load application details from RM when available.

  • TEZ-2936: Support HDFS-based Timeline writer.

  • TEZ-2946: Tez UI: At times RM return a huge error message making the yellow error bar to fill the whole screen.

  • TEZ-2947: Tez UI: Timeline, RM & AM requests gets into a consecutive loop in counters page without any delay.

  • TEZ-2949: Allow duplicate dag names within session for Tez.

  • TEZ-2960: Tez UI: Move hardcoded url namespace to the configuration file.

  • TEZ-2963: RecoveryService#handleSummaryEvent exception with HDFS transparent encryption + kerberos authentication.

  • TEZ-2968: Counter limits exception causes AM to crash.

  • TEZ-2970: Re-localization in TezChild does not use correct UGI.

  • TEZ-2975: Bump up apache commons dependency.

  • TEZ-2988: DAGAppMaster:shutdownTezAM should return with a no-op if it has been invoked earlier.

  • TEZ-2995: Timeline primary filter should only be on callerId and not type.

  • TEZ-2997: Tez UI: Support searches by CallerContext ID for DAGs.

  • TEZ-3017: HistoryACLManager does not have a close method for cleanup.

  • TEZ-3025: InputInitializer creation should use the dag UGI.

  • TEZ-3032: Incorrect start time in different events for DAG history events.

  • TEZ-3036: Tez AM can hang on startup with no indication of error.

  • TEZ-3037: History URL should be set regardless of which history logging service is enabled.

  • TEZ-3052: Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED.

  • TEZ-3063: Tez UI : Display Input, Output, Processor, Source and Sink configurations under a vertex.

  • TEZ-3066: TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services.

  • TEZ-3086: Tez UI: Backward compatibility changes.

  • TEZ-3101: Tez UI: Task attempt log link doesn't have the correct protocol.

  • TEZ-3103: Shuffle can hang when memory to memory merging enabled.

  • TEZ-3105: TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures.

  • TEZ-3107: tez-tools: Log warn messages in case ATS has wrong values (e.g. startTime > finishTime).

  • TEZ-3114: Shuffle OOM due to EventMetaData flood.

  • TEZ-3117: Deadlock in Edge and Vertex code.

  • TEZ-3123: Containers can get re-used even with conflicting local resources.

  • TEZ-3126: Log reason for not reducing parallelism.

  • TEZ-3128: Avoid stopping containers on the AM shutdown thread.

  • TEZ-3131: Support a way to override test_root_dir for FaultToleranceTestRunner.

  • TEZ-3137: Tez task failed with illegal state exception.

  • TEZ-3147: Intermediate mem-to-mem: Fix early exit when only one segment can fit into memory.

  • TEZ-3155: Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits.

  • TEZ-3156: Tez client keeps trying to talk to RM even if RM does not know about the application.

  • TEZ-3166: Counters aren't fully updated and sent for failed task.

  • TEZ-3173: Update Tez AM REST APIs for more information for each vertex.

  • TEZ-3175: Add tez client submit host.

  • TEZ-3177: Non-DAG events should use the session domain or no domain if the data does not need protection.

  • TEZ-3189: Pre-warm dags should not be counted in submitted dags count by DAGAppMaster.

  • TEZ-3192: IFile#checkState creating unnecessary objects though auto-boxing.

  • TEZ-3193: Deadlock in AM during task commit request.

  • TEZ-3196: java.lang.InternalError from decompression codec is fatal to a task during shuffle.

  • TEZ-3202: Reduce the memory need for jobs with high number of segments.

  • TEZ-3203: DAG hangs when one of the upstream vertices has zero tasks.

  • TEZ-3213: Uncaught exception during vertex recovery leads to invalid state transition loop.

  • TEZ-3223: Support a NullHistoryLogger to disable history logging if needed.

  • TEZ-3224: User payload is not initialized before creating vertex manager plugin.

  • TEZ-3233: Tez UI: Have LLAP information reflect in Tez UI.

  • TEZ-3254: Tez UI: Consider downloading Hive/Pig explain plan.

  • TEZ-3255: Tez UI: Hide swimlane while displaying running DAGs from old versions of Tez.

  • TEZ-3256: [Backport HADOOP-11032] Remove Guava Stopwatch dependency.

  • TEZ-3258: JVM Checker does not ignore DisableExplicitGC when checking JVM GC options.

  • TEZ-3259: Tez UI: Build issue - File saver package is not working well with bower.

  • TEZ-3262: Tez UI : zip.js is not having a bower friendly versioning system.

  • TEZ-3264: Tez UI: UI discrepancies.

  • TEZ-3276: Tez Example MRRSleep job fails when tez.staging-dir fs is not same as default FS.

  • TEZ-3281: Tez UI: Swimlane improvement.

  • TEZ-3286: Allow clients to set processor reserved memory per vertex (instead of per container).

  • TEZ-3288: Tez UI: Display more details in the error bar.

  • TEZ-3289: Tez Example MRRSleep job does not set Staging dir correctly on secure cluster.

  • TEZ-3291: Optimize splits grouping when locality information is not available.

  • TEZ-3292: Tez UI: UTs breaking with timezone change.

  • TEZ-3293: Fetch failures can cause a shuffle hang waiting for memory merge that never starts.

  • TEZ-3294: DAG.createDag() does not clear local state on repeat calls.

  • TEZ-3295: TestOrderedWordCount should handle relative input/output path.

  • TEZ-3297: Deadlock scenario in AM during ShuffleVertexManager auto reduce.

  • TEZ-3304: TestHistoryParser fails with Hadoop 2.7.

  • TEZ-3305: TestAnalyzer fails with Hadoop 2.7.

  • TEZ-3308: Add counters to capture input split length.

  • TEZ-3314: Double counting input bytes in MultiMRInput.

  • TEZ-3318: Tez UI: Polling is not restarted after RM recovery.

  • TEZ-3325: Flaky test in TestDAGImpl.testCounterLimits.

  • TEZ-3327: ATS Parser: Populate config details available in dag.

  • TEZ-3329: Tez ATS data is incomplete for a vertex which fails or gets killed before initialization.

  • TEZ-3331: Add operation specific HDFS counters to ATS.

  • TEZ-3333: Tez UI: Handle cases where Vertex/Task/Task Attempt data is missing.

  • TEZ-3337: Do not log empty fields of TaskAttemptFinishedEvent to avoid confusion.

  • TEZ-3357: Change TimelineCachePlugin to handle DAG grouping.

  • TEZ-3370: Tez UI: Display the log link as N/A if the app does not provide a log line.

  • TEZ-3374: Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.

  • TEZ-3376: Fix groupId generation to account for dagId starting with 1.

  • TEZ-3379: Tez analyzer: Move sysout to log4j.

  • TEZ-3382: Tez analyzer: Should be resilient to new counters.

  • TEZ-3398: Tez UI: Bread crumb link to Application from Application details dag/configuration tab is broken