Release Notes
Also available as:
PDF

Hadoop

This release provides Hadoop Common 2.7.3 and the following Apache patches:

  • HADOOP-12173: NetworkTopology::add calls toString always.

  • HDFS-11499: Decommissioning stuck because of failing recovery.

  • HDFS-11545: Propagate DataNode's slow disks info to the NameNode via Heartbeat.

  • HDFS-11547: Add logs for slow BlockReceiver while writing data to disk.

  • HDFS-11551: Handle SlowDiskReport from DataNode at the NameNode. .

  • HDFS-11560: Expose slow disks via NameNode JMX.

  • HDFS-11570: Unit test for NameNodeStatusMXBean.

  • HDFS-11603: Improve slow mirror/disk warnings in BlockReceiver.

  • HDFS-11722: Change Datanode file IO profiling sampling to percentage.

  • HDFS-11842: TestDataNodeOutlierDetectionViaMetrics UT fails intermittently.

  • MAPREDUCE-6702: Fixed failures in TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv.

  • YARN-5368: Fixed a memory leak in Timeline server.

HDP 2.6.0 provided Hadoop Common 2.7.3 the following Apache patches:

  • HADOOP-1023: Add more logging in WASB page blob read function.

  • HADOOP-1026: Logging bug in WASB causing out of index exception.

  • HADOOP-1540: breaks backward compatibilty.

  • HADOOP-1540: Support file exclusion list in distcp.

  • HADOOP-3733: "s3x:" URLs break when Secret Key contains a slash, even if encoded.

  • HADOOP-5323: Trash documentation should describe its directory structure and configurations.

  • HADOOP-5470: RunJar.unJar() should write the last modified time found in the jar entry to the uncompressed file.

  • HADOOP-7139: Allow appending to existing SequenceFiles

  • HADOOP-7256: Resource leak during failure scenario of closing of resources.

  • HADOOP-7713: dfs -count -q should label output column

  • HADOOP-7713: dfs -count -q should label output column .

  • HADOOP-7817: RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos()

  • HADOOP-7930: Kerberos relogin interval in UserGroupInformation should be configurable

  • HADOOP-8151: Error handling in snappy decompressor throws invalid exceptions.

  • HADOOP-8436: NPE In getLocalPathForWrite

  • HADOOP-8437: getLocalPathForWrite should throw IOException for invalid paths.

  • HADOOP-8751: NPE in Token.toString() when Token is constructed using null identifier.

  • HADOOP-8818: Use equals instead == in MD5MD5CRC32FileChecksum and TFileDumper.

  • HADOOP-8934: Shell command ls should include sort options

  • HADOOP-9121: InodeTree.java has redundant check for vName while throwing exception.

  • HADOOP-9242: Duplicate surefire plugin config in hadoop-common.

  • HADOOP-9477: Add posixGroups support for LDAP groups mapping service.

  • HADOOP-10035: Cleanup TestFilterFileSystem.

  • HADOOP-10048: LocalDirAllocator should avoid holding locks while accessing the filesystem.

  • HADOOP-10300: Allowed deferred sending of call responses.

  • HADOOP-10365: BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block.

  • HADOOP-10406: TestIPC.testIpcWithReaderQueuing may fail.

  • HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.

  • HADOOP-10597: RPC Server signals backoff to clients when all request queues are full.

  • HADOOP-1062:added hadoop-azure-datalake and azure-data-lake-store-sdk to all the wasb locations.

  • HADOOP-10775: Shell operations to fail with meaningful errors on windows if winutils.exe not found.

  • HADOOP-10823: TestReloadingX509TrustManager is flaky.

  • HADOOP-10965: Print fully qualified path in CommandWithDestination error messages.

  • HADOOP-10971: Add -C flag to make `hadoop fs -ls` print filenames only.

  • HADOOP-11031: Design Document for Credential Provider API.

  • HADOOP-11098: [JDK8] Max Non Heap Memory default changed between JDK7 and 8.

  • HADOOP-11212: NetUtils.wrapException to handle SocketException explicitly.

  • HADOOP-11218: Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.

  • HADOOP-11252: RPC client does not time out by default

  • HADOOP-11262: Enable YARN to use S3A.

  • HADOOP-11361: Fix a race condition in MetricsSourceAdapter.updateJmxCache.

  • HADOOP-11404: Clarify the "expected client Kerberos principal is null" authorization message.

  • HADOOP-11491: HarFs incorrectly declared as requiring an authority.

  • HADOOP-11581: Multithreaded correctness Warnings #org.apache.hadoop.fs.shell.Ls

  • HADOOP-11613: Remove commons-httpclient dependency from hadoop-azure.

  • HADOOP-11628: SPNEGO auth does not work with CNAMEs in JDK8.

  • HADOOP-11685: StorageException complaining "no lease ID" during HBase distributed log splitting.

  • HADOOP-11687: Ignore x-* and response headers when copying an Amazon S3 object.

  • HADOOP-11692: Improve authentication failure WARN message to avoid user confusion.

  • HADOOP-11713: ViewFileSystem should support snapshot methods.

  • HADOOP-11717: Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth

  • HADOOP-11772: RPC Invoker relies on static ClientCache which has synchronized(this) blocks.

  • HADOOP-11780: Prevent IPC reader thread death.

  • HADOOP-11785: Reduce the number of listStatus operation in distcp buildListing

  • HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm

  • HADOOP-11812: Implement listLocatedStatus for ViewFileSystem to speed up split calculation

  • HADOOP-11827: Speed-up distcp buildListing() using threadpool

  • HADOOP-11859: PseudoAuthenticationHandler fails with httpcomponents v4.4.

  • HADOOP-11864: JWTRedirectAuthenticationHandler breaks java8 javadocs.

  • HADOOP-11876: Refactor code to make it more readable, minor maybePrintStats bug

  • HADOOP-11901: BytesWritable fails to support 2G chunks due to integer overflow.

  • HADOOP-11918: Listing an empty s3a root directory throws FileNotFound.

  • HADOOP-11924: Tolerate JDK-8047340-related exceptions in Shell#isSetSidAvailable preventing class init.

  • HADOOP-11932: MetricsSinkAdapter may hang when being stopped.

  • HADOOP-11960: Enable Azure-Storage Client Side logging.

  • HADOOP-12001: Fixed LdapGroupsMapping to include configurable Posix UID and GID attributes during the search.

  • HADOOP-12006: Remove unimplemented option for from document in branch-2.7.

  • HADOOP-12009: Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. backport to 2.8.

  • HADOOP-12017: Hadoop archives command should use configurable replication factor when closing

  • HADOOP-12043: Display warning if defaultFs is not set when running fs commands.

  • HADOOP-12045: Enable LocalFileSystem#setTimes to change atime.

  • HADOOP-12049: Control http authentication cookie persistence via configuration.

  • HADOOP-12052:IPC client downgrades all exception types to IOE, breaks callers trying to use them.

  • HADOOP-12054: RPC client should not retry for InvalidToken exceptions.

  • HADOOP-12056: Use DirectoryStream in DiskChecker#checkDirs to detect errors when listing a directory.

  • HADOOP-12058: Fix dead links to DistCp and Hadoop Archives pages.

  • HADOOP-12059: S3Credentials should support use of CredentialProvider.

  • HADOOP-12061: Incorrect command in single cluster setup document.

  • HADOOP-12073: Azure FileSystem PageBlobInputStream does not return -1 on EOF.

  • HADOOP-12074: in Shell.java#runCommand() rethrow InterruptedException as InterruptedIOException

  • HADOOP-12076: Incomplete Cache Mechanism in CredentialProvider API.

  • HADOOP-12089: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

  • HADOOP-12095: org.apache.hadoop.fs.shell.TestCount fails.

  • HADOOP-12097: Allow port range to be specified while starting webapp.

  • HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission.

  • HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing.

  • HADOOP-12107: long running apps may have a huge number of StatisticsData instances under FileSystem

  • HADOOP-12119: hadoop fs -expunge does not work for federated namespace

  • HADOOP-12158: Improve error message in TestCryptoStreamsWithOpensslAesCtrCryptoCodec when OpenSSL is not installed.

  • HADOOP-12159: Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces

  • HADOOP-12161: Add getStoragePolicy API to the FileSystem interface

  • HADOOP-12169:ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse.

  • HADOOP-12185: NetworkTopology is not efficient adding/getting/removing nodes.

  • HADOOP-12186: ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times

  • HADOOP-12189: Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.

  • HADOOP-12191: Bzip2Factory is not thread safe.

  • HADOOP-12213: Interrupted exception can occur when Client#stop is called.

  • HADOOP-12229:Fix inconsistent subsection titles in filesystem.md.

  • HADOOP-12230: hadoop-project declares duplicate, conflicting curator dependencies.

  • HADOOP-12232: Upgrade Tomcat dependency to 6.0.44.

  • HADOOP-12239: StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

  • HADOOP-12242: Add in-page TOC to filesystem specification pages.

  • HADOOP-12252: LocalDirAllocator should not throw NPE with empty string configuration.

  • HADOOP-12258: Need translate java.nio.file.NoSuchFileException to FileNotFoundException to avoid regression.

  • HADOOP-12259: Utility to Dynamic port allocation

  • HADOOP-12269: Update aws-sdk dependency to 1.10.6

  • HADOOP-12280: Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded

  • HADOOP-12284: UserGroupInformation doAs can throw misleading exception

  • HADOOP-12291: Add support for nested groups in LdapGroupsMapping.

  • HADOOP-12292: Make use of DeleteObjects optional.

  • HADOOP-12296: when setnetgrent returns 0 in linux, exception should be thrown.

  • HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.

  • HADOOP-12317: Applications fail on NM restart on some linux distro because NM container recovery declares AM container as LOST

  • HADOOP-12324: Better exception reporting in SaslPlainServer.

  • HADOOP-12325: RPC Metrics : Add the ability track and log slow RPCs.

  • HADOOP-12334: Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries.

  • HADOOP-12345: Pad hostname correctly in CredentialsSys.java.

  • HADOOP-12346: Increase some default timeouts / retries for S3a connector.

  • HADOOP-12348: MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.

  • HADOOP-12350: WASB Logging: Improve WASB Logging around deletes, reads and writes.

  • HADOOP-12358: Add -safely flag to rm to prompt when deleting many files.

  • HADOOP-12359: hadoop fs -getmerge doc is wrong.

  • HADOOP-12366: expose calculated paths.

  • HADOOP-12374: Description of hdfs expunge command is confusing.

  • HADOOP-12386: RetryPolicies.RETRY_FOREVER should be able to specify a retry interval.

  • HADOOP-12404: Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.

  • HADOOP-12406: Fixed AbstractMapWritable.readFields to use the thread's ClassLoader to load class instead of System ClassLoader.

  • HADOOP-12407: Test failing: hadoop.ipc.TestSaslRPC.

  • HADOOP-12413: AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.

  • HADOOP-12415: Add io.netty dependency to hadoop-nfs and to hadoop-hdfs, needed to build Bigtop successfuly, see BIGTOP-2049.

  • HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.

  • HADOOP-12416: Trash messages should be handled by Logger instead of being delivered on System.out.

  • HADOOP-12423: Handle failure of registering shutdownhook by ShutdownHookManager in static block

  • HADOOP-12426: Add Entry point for Kerberos health check.

  • HADOOP-12437: Allow SecurityUtil to lookup alternate hostnames.

  • HADOOP-12438: TestLocalFileSystem tests can fail on Windows after HDFS-8767 fix for handling pipe.

  • HADOOP-12440: TestRPC#testRPCServerShutdown did not produce the desired thread states before shutting down.

  • HADOOP-12441: Fixed kill-command behavior to work correctly across OSes by using bash shell built-in.

  • HADOOP-12442: Display help if the command option to 'hdfs dfs' is not valid

  • HADOOP-12444: Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel.

  • HADOOP-12449: TestDNS and TestNetUtils failing if no network.

  • HADOOP-12453: Support decoding KMS Delegation Token with its own Identifier.

  • HADOOP-12457: [JDK8] Fix a failure of compiling common by javadoc.

  • HADOOP-12463: Fix TestShell.testGetSignalKillCommand failure on windows.

  • HADOOP-12464: Interrupted client may try to fail-over and retry.

  • HADOOP-12465: Incorrect javadoc in WritableUtils.java.

  • HADOOP-12469: distcp should not ignore the ignoreFailures option.

  • HADOOP-12472: Make GenericTestUtils.assertExceptionContains robust.

  • HADOOP-12478: Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath().

  • HADOOP-12481: JWTRedirectAuthenticationHandler doesn't Retain Original Query String.

  • HADOOP-12482: Race condition in JMX cache update.

  • HADOOP-12483: Maintain wrapped SASL ordering for postponed IPC responses.

  • HADOOP-12484: Single File Rename Throws Incorrectly In Potential Race Condition Scenarios.

  • HADOOP-12508: delete fails with exception when lease is held on blob.

  • HADOOP-12520: Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests.

  • HADOOP-12526: there are duplicate dependency definitions in pom's

  • HADOOP-12533: Introduce FileNotFoundException in WASB for read and seek API.

  • HADOOP-12535: Run FileSystem contract tests with hadoop-azure.

  • HADOOP-12537: S3A to support Amazon STS temporary credentials.

  • HADOOP-12540: TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.

  • HADOOP-12542: TestDNS fails on Windows after HADOOP-12437.

  • HADOOP-12545: Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp.

  • HADOOP-12548: Read s3a creds from a Credential Provider.

  • HADOOP-12551: Introduce FileNotFoundException for WASB FileSystem API

  • HADOOP-12555: WASB to read credentials from a credential provider.

  • HADOOP-12559: KMS connection failures should trigger TGT renewal.

  • HADOOP-12564: Upgrade JUnit3 TestCase to JUnit 4 in org.apache.hadoop.io package.

  • HADOOP-12565: Replace DSA with RSA for SSH key type in SingleCluster.md.

  • HADOOP-12568: Update core-default.xml to describe posixGroups support.

  • HADOOP-12570: HDFS Secure Mode Documentation updates.

  • HADOOP-12573: TestRPC.testClientBackOff failing.

  • HADOOP-12577: Bump up commons-collections version to 3.2.2 to address a security flaw.

  • HADOOP-12588: Fix intermittent test failure of TestGangliaMetrics.

  • HADOOP-12589: Fix intermittent test failure of TestCopyPreserveFlag

  • HADOOP-12590: TestCompressorDecompressor failing without stack traces

  • HADOOP-12602: TestMetricsSystemImpl#testQSize occasionally fails.

  • HADOOP-12603: TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail.

  • HADOOP-12604: Exception may be swallowed in KMSClientProvider.

  • HADOOP-12605: Fix intermittent failure of TestIPC.testIpcWithReaderQueuingi.

  • HADOOP-12608: Fix exception message in WASB when connecting with anonymous credential.

  • HADOOP-12609: Fix intermittent failure of TestDecayRpcScheduler.

  • HADOOP-12613: TestFind.processArguments occasionally fails.

  • HADOOP-12622: Improve the loggings in RetryPolicies and RetryInvocationHandler.

  • HADOOP-12634: Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy.

  • HADOOP-12635: Adding Append API support for WASB.

  • HADOOP-12636: Prevent ServiceLoader failure init for unused FileSystems.

  • HADOOP-12653: Use SO_REUSEADDR to avoid getting "Address already in use" when using kerberos and attempting to bind to any port on the local IP address.

  • HADOOP-12659: Incorrect usage of config parameters in token manager of KMS.

  • HADOOP-12663: Remove Hard-Coded Values From FileSystem.java.

  • HADOOP-12666: Support Microsoft Azure Data Lake - as a file system in Hadoop.

  • HADOOP-12667:s3a to support createNonRecursive API.

  • HADOOP-12668: Support excluding weak Ciphers in HttpServer2 through ssl- server.conf.

  • HADOOP-12672: RPC timeout should not override IPC ping interval

  • HADOOP-12675: Fix description about retention period in usage of expunge command.

  • HADOOP-12678: Handle empty rename pending metadata file during atomic rename in redo path.

  • HADOOP-12682: Fix TestKMS#testKMSRestart* failure.

  • HADOOP-12688: Fix deadlinks in Compatibility.md.

  • HADOOP-12689: S3 filesystem operations stopped working correctly

  • HADOOP-12691: Add CSRF Filter for REST APIs to Hadoop Common.

  • HADOOP-12691: Move files to correct location.

  • HADOOP-12696: Add tests for S3Filesystem Contract

  • HADOOP-12699: TestKMS#testKMSProvider intermittently fails during 'test rollover draining'.

  • HADOOP-12700: Remove unused import in TestCompressorDecompressor.java.

  • HADOOP-12706: TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally.

  • HADOOP-12711: Remove dependency on commons-httpclient for ServletUtil

  • HADOOP-12715: TestValueQueue#testgetAtMostPolicyALL fails intermittently.

  • HADOOP-12716: KerberosAuthenticator#doSpnegoSequence use incorrect class to determine isKeyTab in JDK8.

  • HADOOP-12717: NPE when trying to rename a directory in Windows Azure Storage FileSystem.

  • HADOOP-12723:S3A: Add ability to plug in any AWSCredentialsProvider.

  • HADOOP-12731: Remove useless boxing/unboxing code.

  • HADOOP-12736: TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out.

  • HADOOP-12738: Create unit test to automatically compare Common related classes and core-default.xml.

  • HADOOP-12751: While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple.

  • HADOOP-12752: Improve diagnostics/use of envvar/sysprop credential propagation.

  • HADOOP-12758: Extend CSRF Filter with UserAgent Checks.

  • HADOOP-12761: incremental maven build is not really incremental

  • HADOOP-12765: HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections.

  • HADOOP-12767: Update apache httpclient version to 4.5.2; httpcore to 4.4.4.

  • HADOOP-12772: NetworkTopologyWithNodeGroup.getNodeGroup() can loop infinitely for invalid 'loc' values.

  • HADOOP-12773: HBase classes fail to load with client/job classloader enabled

  • HADOOP-12774: s3a should use UGI.getCurrentUser.getShortname() for username.

  • HADOOP-12776: Remove getaclstatus call for non-acl commands in getfacl.

  • HADOOP-12780: During WASB atomic rename handle crash when one directory has been renamed but not file under it.

  • HADOOP-12782: Faster LDAP group name resolution with ActiveDirectory.

  • HADOOP-12786: "hadoop key" command usage is not documented.

  • HADOOP-12787: KMS SPNEGO sequence does not work with WEBHDFS.

  • HADOOP-12788: OpensslAesCtrCryptoCodec should log which random number generator is used.

  • HADOOP-12789: log classpath of ApplicationClassLoader at INFO level.

  • HADOOP-12792: TestUserGroupInformation#testGetServerSideGroups fails in chroot.

  • HADOOP-12793: Write a new group mapping service guide

  • HADOOP-12794: Support additional compression levels for GzipCodec.

  • HADOOP-12795: KMS does not log detailed stack trace for unexpected errors.

  • HADOOP-12800: Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs.

  • HADOOP-12801: Suppress obsolete S3FileSystem tests.

  • HADOOP-12804: Read Proxy Password from Credential Providers in S3 FileSystem.

  • HADOOP-12805: Annotate CanUnbuffer with @InterfaceAudience.Public

  • HADOOP-12807: S3AFileSystem should read AWS credentials from environment variables.

  • HADOOP-12810: FileSystem#listLocatedStatus causes unnecessary RPC calls

  • HADOOP-12817: Enable TLS v1.1 and 1.2

  • HADOOP-12825: Log slow name resolutions.

  • HADOOP-12828: Print user when services are started.

  • HADOOP-12829: StatisticsDataReferenceCleaner swallows interrupt exceptions

  • HADOOP-12831: LocalFS/FSOutputSummer NPEs in constructor if bytes per checksum set to 0

  • HADOOP-12844:Recover when S3A fails on IOException in read().

  • HADOOP-12846: Credential Provider Recursive Dependencies.

  • HADOOP-12847: hadoop daemonlog should support https and SPNEGO for Kerberized cluster.

  • HADOOP-12849: TestSymlinkLocalFSFileSystem fails intermittently.

  • HADOOP-12851: S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders.

  • HADOOP-12853: Change WASB documentation regarding page blob support.

  • HADOOP-12870: Fix typo admininistration in CommandsManual.md.

  • HADOOP-12871: Fix dead link to NativeLibraries.html in CommandsManual.md.

  • HADOOP-12872: Fix formatting in ServiceLevelAuth.md.

  • HADOOP-12875: [Azure Data Lake] Support for contract test and unit test cases.

  • HADOOP-12878: KMS SPNEGO sequence does not work with WEBHDFS.

  • HADOOP-12886: Exclude weak ciphers in SSLFactory through ssl-server.xml.

  • HADOOP-12891: S3AFileSystem should configure Multipart Copy threshold and chunk size.

  • HADOOP-12893: Update LICENSE.txt and NOTICE.txt.

  • HADOOP-12893: Update LICENSE.txt and NOTICE.txt.

  • HADOOP-12893: Update LICENSE.txt and NOTICE.txt".

  • HADOOP-12895: SSLFactory#createSSLSocketFactory exception message is wrong.

  • HADOOP-12901: Add warning log when KMSClientProvider cannot create a connection to the KMS server.

  • HADOOP-12902: JavaDocs for SignerSecretProvider are out-of-date in AuthenticationFilter.

  • HADOOP-12903: IPC Server should allow suppressing exception logging by type,.

  • HADOOP-12903: IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages.

  • HADOOP-12906: AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.

  • HADOOP-12916: Allow RPC scheduler/callqueue backoff using response times.

  • HADOOP-12926: lz4.c does not detect 64-bit mode properly.

  • HADOOP-12929: JWTRedirectAuthenticationHandler must accommodate null expiration time.

  • HADOOP-12943: Add -w -r options in dfs -test command.

  • HADOOP-12947: Update documentation Hadoop Groups Mapping to add static group mapping, negative cache.

  • HADOOP-12950: ShutdownHookManager should have a timeout for each of the Registered shutdown hook.

  • HADOOP-12954: Add a way to change hadoop.security.token.service.use_ip

  • HADOOP-12958: PhantomReference for filesystem statistics can trigger OOM.

  • HADOOP-12962: KMS key names are incorrectly encoded when creating key.

  • HADOOP-12963:Allow using path style addressing for accessing the s3 endpoint.

  • HADOOP-12964: Http server vulnerable to clickjacking.

  • HADOOP-12973: Make DU pluggable.

  • HADOOP-12975: Add jitter to CachingGetSpaceUsed's thread

  • HADOOP-12977:s3a to handle delete("/", true) robustly.

  • HADOOP-12982:Document missing S3A and S3 properties.

  • HADOOP-12984: Add GenericTestUtils.getTestDir method and use it for emporary directory in tests.

  • HADOOP-12985: Support MetricsSource interface for DecayRpcScheduler Metrics.

  • HADOOP-12989: Some tests in org.apache.hadoop.fs.shell.find occasionally time out.

  • HADOOP-12993: Change ShutdownHookManger complete shutdown log from INFO to DEBUG.

  • HADOOP-12994: Specify PositionedReadable, add contract tests, fix problems.

  • HADOOP-13008: Add XFS Filter for UIs to Hadoop Common.

  • HADOOP-13011: Clearly Document the Password Details for Keystore-based Credential Providers

  • HADOOP-13018: Make Kdiag check whether hadoop.token.files points to existent and valid files.

  • HADOOP-13024: Distcp with -delete feature on raw data not implemented.

  • HADOOP-13026: Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator.

  • HADOOP-13028:add low level counter metrics for S3A; use in read performance tests.

  • HADOOP-13030: Handle special characters in passwords in KMS startup script.

  • HADOOP-13037: Refactor Azure Data Lake Store as an independent FileSystem.

  • HADOOP-13039: Add documentation for configuration property ipc.maximum.data.length

  • HADOOP-13042: Restore lost leveldbjni LICENSE and NOTICE changes.

  • HADOOP-13047:S3a Forward seek in stream length to be configurable.

  • HADOOP-13052: ChecksumFileSystem mishandles crc file permissions.

  • HADOOP-13058:S3A FS fails during init against a read-only FS if multipart purge.

  • HADOOP-13065: Add a new interface for retrieving FS and FC Statistics

  • HADOOP-13072: WindowsGetSpaceUsed constructor should be public

  • HADOOP-13079: Add -q option to Ls to print ? instead of non-printable characters.

  • HADOOP-13084: Fix ASF License warnings in branch-2.7.

  • HADOOP-13098: Dynamic LogLevel setting page should accept case-insensitive log level string.

  • HADOOP-13103: Group resolution from LDAP may fail on javax.naming.ServiceUnavailableException.

  • HADOOP-13105: Support timeouts in LDAP queries in LdapGroupsMapping.

  • HADOOP-13113:Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel.

  • HADOOP-13116:Jets3tNativeS3FileSystemContractTest does not run.

  • HADOOP-13122:Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.

  • HADOOP-13130: s3a failures can surface as RTEs, not IOEs.

  • HADOOP-13131: Add tests to verify that S3A supports SSE-S3 encryption.

  • HADOOP-13138: Unable to append to a SequenceFile with Compression.NONE.

  • HADOOP-13139:Branch-2: S3a to use thread pool that blocks clients.

  • HADOOP-13140: FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes.

  • HADOOP-13145:In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.

  • HADOOP-13146: Refactor RetryInvocationHandler.

  • HADOOP-13155: Implement TokenRenewer to renew and cancel delegation tokens in KMS.

  • HADOOP-13158:S3AFileSystem#toString might throw NullPointerException due to null cannedACL.

  • HADOOP-13159: Fix potential NPE in Metrics2 source for DecayRpcScheduler.

  • HADOOP-13162: Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs.

  • HADOOP-13163:Reuse pre-computed filestatus in Distcp-CopyMapper

  • HADOOP-13164:Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories.

  • HADOOP-13168: Support Future.get with timeout in ipc async calls

  • HADOOP-13169: Randomize file list in SimpleCopyListing.

  • HADOOP-13171: Add StorageStatistics to S3A; instrument some more operations.

  • HADOOP-13179: GenericOptionsParser is not thread-safe because commons-cli OptionBuilder is not thread-safe.

  • HADOOP-13183: S3A proxy tests fail after httpclient/httpcore upgrade.

  • HADOOP-13188:S3A file-create should throw error rather than overwrite directories.

  • HADOOP-13192: org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly.

  • HADOOP-13197: Add non-decayed call metrics for DecayRpcScheduler.

  • HADOOP-13202: Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes.

  • HADOOP-13203: S3a: Consider reducing the number of connection aborts by setting correct length in s3 request.

  • HADOOP-13207: Specify FileSystem listStatus, listFiles and RemoteIterator.

  • HADOOP-13208: S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories.

  • HADOOP-13212:Provide an option to set the socket buffers in S3AFileSystem

  • HADOOP-13213: Small Documentation bug with AuthenticatedURL in hadoop-auth.

  • HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials.

  • HADOOP-13239:Deprecate s3:// in branch-2.

  • HADOOP-13241: document s3a better.

  • HADOOP-13242: Authenticate to Azure Data Lake using client ID and keys.

  • HADOOP-13244: o.a.h.ipc.Server#Server should honor handlerCount when queueSizePerHandler is specified in consturctor.

  • HADOOP-13249: RetryInvocationHandler need wrap InterruptedException in IOException when call Thread.sleep.

  • HADOOP-13251: Authenticate with Kerberos credentials when renewing KMS delegation token.

  • HADOOP-13252: Tune S3A provider plugin mechanism.

  • HADOOP-13254: Create framework for configurable disk checkers

  • HADOOP-13255: KMSClientProvider should check and renew tgt when doing delegation token operations.

  • HADOOP-13257: Improve Azure Data Lake contract tests.

  • HADOOP-13263: Reload cached groups in background after expiry.

  • HADOOP-13270: BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks.

  • HADOOP-13280: FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps

  • HADOOP-13283: Support reset operation for new global storage statistics and per FS storage stats.

  • HADOOP-13285: DecayRpcScheduler MXBean should only report decayed CallVolumeSummary.

  • HADOOP-13287: TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +.

  • HADOOP-13288: Guard null stats key in FileSystemStorageStatistics

  • HADOOP-13290: Appropriate use of generics in FairCallQueue.

  • HADOOP-13291: Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented.

  • HADOOP-13299: JMXJsonServlet is vulnerable to TRACE.

  • HADOOP-13305: Define common statistics names across schemes.

  • HADOOP-13309: Document S3A known limitations in file ownership and permission model.

  • HADOOP-13315: FileContext#umask is not initialized properly.

  • HADOOP-13316: Enforce Kerberos authentication for required ops in DelegationTokenAuthenticator.

  • HADOOP-13320: Fix arguments check in documentation for WordCount v2.0.

  • HADOOP-13323: Downgrade stack trace on FS load from Warn to debug.

  • HADOOP-13324: s3a tests don't authenticate with S3 frankfurt

  • HADOOP-13336:S3A to support per-bucket configuration.

  • HADOOP-13350: Additional fix to LICENSE and NOTICE.

  • HADOOP-13351: TestDFSClientSocketSize buffer size tests are flaky

  • HADOOP-13352: Make X-FRAME-OPTIONS configurable in HttpServer2.

  • HADOOP-13353: LdapGroupsMapping getPassward shouldn't return null when IOException throws.

  • HADOOP-13354: Update WASB driver to use the latest version

  • HADOOP-13368: DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation.

  • HADOOP-13381: KMS clients should use KMS Delegation Tokens from current UGI.

  • HADOOP-13382: Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, HADOOP-openstack.

  • HADOOP-13387:users always get told off for using S3 —even when not using it.

  • HADOOP-13389:TestS3ATemporaryCredentials.testSTS error when using IAM credentials.

  • HADOOP-13392: [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.

  • HADOOP-13393: Omit unsupported fs.defaultFS setting in ADLS documentation.

  • HADOOP-13403: AzureNativeFileSystem rename/delete performance improvements.

  • HADOOP-13405:doc for fs.s3a.acl.default indicates incorrect values.

  • HADOOP-13406:S3AFileSystem: Consider reusing filestatus in delete() and mkdirs().

  • HADOOP-13422: ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process.

  • HADOOP-13433:Race in UGI.reloginFromKeytab.

  • HADOOP-13434: Add quoting to Shell class.

  • HADOOP-13443: KMS should check the type of underlying keyprovider of KeyProviderExtension before falling back to default.

  • HADOOP-13446: Support running isolated unit tests separate from AWS integration tests.

  • HADOOP-13447: Refactor S3AFileSystem to support introduction of separate metadata repository and tests.

  • HADOOP-13458: LoadBalancingKMSClientProvider#doOp should log IOException stacktrace.

  • HADOOP-13459: hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time.

  • HADOOP-13461: NPE in KeyProvider.rollNewVersion.

  • HADOOP-13466: Add an AutoCloseableLock class.

  • HADOOP-13467: Shell#getSignalKillCommand should use the bash builtin on Linux.

  • HADOOP-13470: GenericTestUtils$LogCapturer is flaky.

  • HADOOP-13494: ReconfigurableBase can log sensitive information.

  • HADOOP-13502: Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests.

  • HADOOP-13512: ReloadingX509TrustManager should keep reloading in case of exception.

  • HADOOP-13513: Java 1.7 support for org.apache.hadoop.fs.azure testcases.

  • HADOOP-13535: Add jetty6 acceptor startup issue workaround to branch-2.

  • HADOOP-13540:improve section on troubleshooting s3a auth problems.

  • HADOOP-13541:explicitly declare the Joda time version S3A depends on.

  • HADOOP-13546: Override equals and hashCode to avoid connection leakage

  • HADOOP-13552: RetryInvocationHandler logs all remote exceptions.

  • HADOOP-13552: RetryInvocationHandler logs all remote exceptions.

  • HADOOP-13558: UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket.

  • HADOOP-13560: S3ABlockOutputStream to support huge

  • HADOOP-13565: KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request.

  • HADOOP-13590: Retry until TGT expires even if the UGI renewal thread encountered exception.

  • HADOOP-13599: s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown.

  • HADOOP-13614: Purge some superfluous/obsolete S3 FS tests that are slowing test runs down.

  • HADOOP-13638: KMS should set UGI's Configuration object properly.

  • HADOOP-13674: S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.

  • HADOOP-13675: Bug in return value for delete() calls in WASB.

  • HADOOP-13680: fs.s3a.readahead.range to use getLongBytes.

  • HADOOP-13686: Adding additional unit test for Trash

  • HADOOP-13692: hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.

  • HADOOP-13707: If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed.

  • HADOOP-13710: Supress CachingGetSpaceUsed from logging interrupted exception stacktrace.

  • HADOOP-13716: Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup.

  • HADOOP-13727: S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.

  • HADOOP-13735:ITestS3AFileContextStatistics.testStatistics() failing.

  • HADOOP-13737: Cleanup DiskChecker interface.

  • HADOOP-13738: DiskChecker should perform some disk IO.

  • HADOOP-13742: Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy Battula.

  • HADOOP-13749: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.

  • HADOOP-13782: Make MutableRates metrics thread-local write, aggregate-on-read.

  • HADOOP-13792: Stackoverflow for schemeless defaultFS with trailing slash.

  • HADOOP-13801:regression: ITestS3AMiniYarnCluster failing.

  • HADOOP-13804: MutableStat mean loses accuracy if add(long, long) is used.

  • HADOOP-13822: Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(). Contribute by Brahma Reddy Battula .

  • HADOOP-13823: s3a rename: fail if dest file exists.

  • HADOOP-13824: FsShell can suppress the real error if no error message is present.

  • HADOOP-13826: S3A Deadlock in multipart copy due to thread pool limits.

  • HADOOP-13830:Intermittent failure of ITestS3NContractRootDir.testRecursiveRootListing "Can not create a Path from an empty string".

  • HADOOP-13831: Correct check for error code to detect Azure Storage Throttling and provide retries.

  • HADOOP-13838: KMSTokenRenewer should close providers

  • HADOOP-13847: KMSWebApp should close KeyProviderCryptoExtension.

  • HADOOP-13855: Fix a couple of the s3a statistic names to be consistent with the rest.

  • HADOOP-13857: S3AUtils.translateException to map

  • HADOOP-13863: Azure: Add a new SAS key mode for WASB.

  • HADOOP-13871: ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful.

  • HADOOP-13890: Maintain HTTP/host as SPNEGO SPN support and fix KerberosName parsing.

  • HADOOP-13930: Azure: Add Authorization support to WASB.

  • HADOOP-13945: Azure: Add Kerberos and Delegation token support to WASB client.

  • HADOOP-13956: Read ADLS credentials from Credential Provider.

  • HADOOP-13962: Update ADLS SDK to 2.1.4.

  • HADOOP-13988: KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser.

  • HADOOP-14017: User friendly name for ADLS user and group.

  • HADOOP-14019: Fix some typos in the s3a docs.

  • HADOOP-14028: S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.

  • HADOOP-14029: Fix KMSClientProvider for non-secure proxyuser use case.

  • HADOOP-14048: REDO operation of WASB#AtomicRename should create placeholder blob for destination folder.

  • HADOOP-14049: Honour AclBit flag associated to file/folder permission for Azure datalake account.

  • HADOOP-14055: SwiftRestClient includes pass length in exception if auth fails.

  • HADOOP-14081:S3A: Consider avoiding array copy in S3ABlockOutputStream

  • HADOOP-14091: AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.

  • HADOOP-14100: Upgrade Jsch jar to latest version to fix vulnerability in old versions.

  • HADOOP-14138:Remove S3A ref from META-INF service discovery, rely on existing core-default entry.

  • HADOOP-14150: Implement getHomeDirectory() method in NativeAzureFileSystem.

  • HADOOP-14214: DomainSocketWatcher::add()/delete() should not self interrupt while looping await().

  • HDFS-2043: TestHFlush failing intermittently.

  • HDFS-2580: NameNode#main(...) can make use of GenericOptionsParser.

  • HDFS-27: HDFS CLI with --config set to default config complains log file not found error.

  • HDFS-3677: dfs.namenode.edits.dir.required is missing from hdfs- default.xml.

  • HDFS-3918: EditLogTailer shouldn't log WARN when other node is in standby mode.

  • HDFS-4015: Safemode should count and report orphaned blocks.

  • HDFS-4176: EditLogTailer should call rollEdits with a timeout.

  • HDFS-4185: Add a metric for number of active leases

  • HDFS-4366: Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.

  • HDFS-4396: Add START_MSG/SHUTDOWN_MSG for ZKFC

  • HDFS-4937: ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom.

  • HDFS-4946: Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable.

  • HDFS-5640: Add snapshot methods to FileContext.

  • HDFS-6054: MiniQJMHACluster should not use static port to avoid binding failure in unit test.

  • HDFS-6101: TestReplaceDatanodeOnFailure fails occasionally.

  • HDFS-6184: Capture NN's thread dump when it fails over.

  • HDFS-6407: Add sorting and pagination in the datanode tab of the NN Web UI.

  • HDFS-6481: DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs.

  • HDFS-6533: TestBPOfferService#testBasicFunctionalitytest fails intermittently.

  • HDFS-6581: Support for writing to single replica in RAM. Applied HDFS-6581.merge.15.patch to Champlain. This is a consolidated patch containing all feature changes and was applied to trunk.

  • HDFS-6666: Abort NameNode and DataNode startup if security is enabled but block access token is not enabled.

  • HDFS-6763: Initialize file system-wide quota once on transitioning to active

  • HDFS-6832: Fix the usage of 'hdfs namenode' command.

  • HDFS-6860: BlockStateChange logs are too noisy.

  • HDFS-6945: BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed.

  • HDFS-7163: WebHdfsFileSystem should retry reads according to the configured retry policy.

  • HDFS-7166: SbNN Web UI shows #Under replicated blocks and #pending deletion blocks.

  • HDFS-7284: Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas.

  • HDFS-7314: When the DFSClient lease cannot be renewed, abort open-for- write files rather than the entire DFSClient.

  • HDFS-7390: Provide JMX metrics per storage type.

  • HDFS-7452: skip StandbyException log for getCorruptFiles()

  • HDFS-7483: Display information per tier on the Namenode UI.

  • HDFS-7597: DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping.

  • HDFS-7608: hdfs dfsclient newConnectedPeer has no write timeout

  • HDFS-7609: Avoid retry cache collision when Standby NameNode loading edits.

  • HDFS-7645: Rolling upgrade is restoring blocks from trash multiple times

  • HDFS-7701: Support reporting per storage type quota and usage with hadoop/hdfs shell.

  • HDFS-7725: Incorrect "nodes in service" metrics caused all writes to fail.

  • HDFS-7758: Retire FsDatasetSpi#getVolumes() and use FsDatasetSpi#getVolumeRefs() instead

  • HDFS-7890: Improve information on Top users for metrics in RollingWindowsManager and lower log level.

  • HDFS-7923: The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages

  • HDFS-7928: Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy.

  • HDFS-7933: fsck should also report decommissioning replicas.

  • HDFS-7964: Add support for async edit logging.

  • HDFS-7978: Add LOG.isDebugEnabled() guard for some LOG.debug(..).

  • HDFS-7990: IBR delete ack should not be delayed.

  • HDFS-8008: Support client-side back off when the datanodes are congested.

  • HDFS-8008: Support client-side back off when the datanodes are congested.

  • HDFS-8009: Signal congestion on the DataNode.

  • HDFS-8009: Signal congestion on the DataNode.

  • HDFS-8046: Allow better control of getContentSummary

  • HDFS-8055: NullPointerException when topology script is missing.

  • HDFS-8071:(Contributed by Tsz Wo Nicholas Sze).

  • HDFS-8099: Change "DFSInputStream has been closed already" message to debug log level.

  • HDFS-8101: DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS classes at runtime.

  • HDFS-8113: Add check for null BlockCollection pointers in BlockInfoContiguous structures.

  • HDFS-8131: Implement a space balanced block placement policy.

  • HDFS-8144: Split TestLazyPersistFiles into multiple tests.

  • HDFS-8152: Refactoring of lazy persist storage cases.

  • HDFS-8155: Support OAuth2 in WebHDFS.

  • HDFS-8180: AbstractFileSystem Implementation for WebHdfs.

  • HDFS-8205: CommandFormat#parse() should not parse option as value of option.

  • HDFS-8209: Support different number of datanode directories in MiniDFSCluster.

  • HDFS-8211: DataNode UUID is always null in the JMX counter.

  • HDFS-8219: setStoragePolicy with folder behavior is different after cluster restart.

  • HDFS-8224: Schedule a block for scanning if its metadata file is corrupt.

  • HDFS-8229: LAZY_PERSIST file gets deleted after NameNode restart.

  • HDFS-8232: Missing datanode counters when using Metrics2 sink interface.

  • HDFS-8247: TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing.

  • HDFS-8276: LazyPersistFileScrubber should be disabled if scrubber interval configured zero.

  • HDFS-8278: When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size.

  • HDFS-8309: Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows.

  • HDFS-8310: Fix TestCLI.testAll "help: help for find" on Windows.

  • HDFS-8311: DataStreamer.transfer() should timeout the socket InputStream.

  • HDFS-8337: Accessing httpfs via webhdfs doesn't work from a jar with kerberos.

  • HDFS-8345: Storage policy APIs must be exposed via the FileSystem interface.

  • HDFS-8384: Allow NN to startup if there are files having a lease but are not under construction.

  • HDFS-8431: hdfs crypto class not found in Windows.

  • HDFS-8432: Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases.

  • HDFS-8435: Support CreateFlag in WebHDFS.

  • HDFS-8492: DN should notify NN when client requests a missing block

  • HDFS-8539: Hdfs doesnt have class 'debug' in windows.

  • HDFS-8542: WebHDFS getHomeDirectory behavior does not match specification.

  • HDFS-8546: Use try with resources in DataStorage and Storage.

  • HDFS-8548: Minicluster throws NPE on shutdown.

  • HDFS-8549: Abort the balancer if an upgrade is in progress.

  • HDFS-8554: TestDatanodeLayoutUpgrade fails on Windows.

  • HDFS-8573: Move creation of restartMeta file logic from BlockReceiver to ReplicaInPipeline.

  • HDFS-8578: On upgrade, Datanode should process all storage/data dirs in parallel.

  • HDFS-8581: ContentSummary on / skips further counts on yielding lock

  • HDFS-8593: Calculation of effective layout version mishandles comparison to current layout version in storage.

  • HDFS-8615: Correct HTTP method in WebHDFS document.

  • HDFS-8633: Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml to match DFSConfigKeys.

  • HDFS-8642: Make TestFileTruncate more reliable.

  • HDFS-8647: Abstract BlockManager's rack policy into BlockPlacementPolicy.

  • HDFS-8656: Preserve compatibility of ClientProtocol#rollingUpgrade after finalization.

  • HDFS-8659: Block scanner INFO message is spamming logs.

  • HDFS-8674: Improve performance of postponed block scans.

  • HDFS-8676: Delayed rolling upgrade finalization can cause heartbeat expiration.

  • HDFS-8696: Make the lower and higher watermark in the DN Netty server configurable.

  • HDFS-8709: Clarify automatic sync in FSEditLog#logEdit.

  • HDFS-8722: Optimize datanode writes for small writes and flushes.

  • HDFS-8729: Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed.

  • HDFS-8767: RawLocalFileSystem.listStatus() returns null for UNIX pipefile.

  • HDFS-8772: Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails.

  • HDFS-8778: TestBlockReportRateLimiting#testLeaseExpiration can deadlock.

  • HDFS-8785: TestDistributedFileSystem is failing in trunk.

  • HDFS-8792: BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory.

  • HDFS-8797: WebHdfsFileSystem creates too many connections for pread.

  • HDFS-8806: Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared

  • HDFS-8809: HDFS fsck reports under construction blocks as CORRUPT.

  • HDFS-8815: DFS getStoragePolicy implementation using single RPC call

  • HDFS-8816: Improve visualization for the Datanode tab in the NN UI.

  • HDFS-8818: Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable.

  • HDFS-8824: Do not use small blocks for balancing the cluster.

  • HDFS-8826: In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes.

  • HDFS-8828: Utilize Snapshot diff report to build diff copy list in distcp.

  • HDFS-8829: Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning.

  • HDFS-8831: Trash Support for deletion in HDFS encryption zone.

  • HDFS-8844: TestHDFSCLI does not cleanup the test directory

  • HDFS-8845: DiskChecker should not traverse the entire tree

  • HDFS-8846: Add a unit test for INotify functionality across a layout version upgrade

  • HDFS-8850: VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks.

  • HDFS-8852: HDFS architecture documentation of version 2.x is outdated about append write support.

  • HDFS-8855: Webhdfs client leaks active NameNode connections.

  • HDFS-8856: Make LeaseManager#countPath O(1).

  • HDFS-8859: Improve DataNode ReplicaMap memory footprint to save about 45%.

  • HDFS-8860: Remove unused Replica copyOnWrite code

  • HDFS-8863: The remaining space check in BlockPlacementPolicyDefault is flawed.

  • HDFS-8867: Enable optimized block reports.

  • HDFS-8879: Quota by storage type usage incorrectly initialized upon namenode restart.

  • HDFS-8880: NameNode metrics logging.

  • HDFS-8883: NameNode Metrics : Add FSNameSystem lock Queue Length.

  • HDFS-8884: Fail-fast check in BlockPlacementPolicyDefault#chooseTarget.

  • HDFS-8885: ByteRangeInputStream used in webhdfs does not override available().

  • HDFS-8887: Expose storage type and storage ID in BlockLocation.

  • HDFS-8891: HDFS concat should keep srcs order.

  • HDFS-8896: DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager.

  • HDFS-8898: Create API and command-line argument to get quota and quota usage without detailed content summary.

  • HDFS-8911: NameNode Metric : Add Editlog counters as a JMX metric.

  • HDFS-8914: Document HA support in the HDFS HdfsDesign.md.

  • HDFS-8923: Add -source flag to balancer usage message.

  • HDFS-8930: Block report lease may leak if the 2nd full block report comes when NN is still in safemode

  • HDFS-8932: NPE thrown in NameNode when try to get TotalSyncCount metric before editLogStream initialization.

  • HDFS-8939: Test(S)WebHdfsFileContextMainOperations failing on branch-2.

  • HDFS-8946: Improve choosing datanode storage for block placement.

  • HDFS-8950: NameNode refresh doesn't remove DataNodes that are no longer in the allowed list.

  • HDFS-8964: When validating the edit log, do not read at or beyond the file offset that is being written

  • HDFS-8965: Harden edit log reading code against out of memory errors

  • HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.

  • HDFS-8969: Clean up findbugs warnings for HDFS-8823 and HDFS-8932.

  • HDFS-8983: NameNode support for protected directories.

  • HDFS-8983: NameNode support for protected directories.

  • HDFS-8995: Flaw in registration bookeeping can make DN die on reconnect.

  • HDFS-9008: Balancer#Parameters class could use a builder pattern.

  • HDFS-9008:was not backported correctly.

  • HDFS-9009: Send metrics logs to NullAppender by default.

  • HDFS-9019: Adding informative message to sticky bit permission denied exception.

  • HDFS-9033: dfsadmin -metasave prints "NaN" for cache used%.

  • HDFS-9038: DFS reserved space is erroneously counted towards non-DFS used.

  • HDFS-9042: Update document for the Storage policy name

  • HDFS-9043: Doc updation for commands in HDFS Federation

  • HDFS-9047: Retire libwebhdfs.

  • HDFS-9048: DistCp documentation is out-of-dated

  • HDFS-9063: Correctly handle snapshot path for getContentSummary.

  • HDFS-9072: Fix random failures in TestJMXGet.

  • HDFS-9082: Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG.

  • HDFS-9083: Replication violates block placement policy.

  • HDFS-9106: Transfer failure during pipeline recovery causes permanent write failures.

  • HDFS-9107: Prevent NNs unrecoverable death spiral after full GC

  • HDFS-9112: Improve error message for Haadmin when multiple name service IDs are configured.

  • HDFS-9128: TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows.

  • HDFS-9130: Use GenericTestUtils#setLogLevel to the logging level.

  • HFDS-9137:should be backported correctly.

  • HDFS-9142: Separating Configuration object for namenode(s) in MiniDFSCluster.

  • HDFS-9145: Tracking methods that hold FSNamesytemLock for too long.

  • HDFS-9148: Incorrect assert message in TestWriteToReplica#testWriteToTemporary

  • HDFS-9175: Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public.

  • HDFS-9178: Slow datanode I/O can cause a wrong node to be marked bad.

  • HDFS-9184: Logging HDFS operation's caller context into audit logs.

  • HDFS-9198: Coalesce IBR processing in the NN.

  • HDFS-9205: Do not schedule corrupt blocks for replication.

  • HDFS-9210: Fix some misuse of %n in VolumeScanner#printStats.

  • HDFS-9220: Reading small file

  • HDFS-9221: HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array.

  • HDFS-9226: Unit Tests failing due to mockito version issue.

  • HDFS-9239: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness.

  • HDFS-9251: Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly creating Files in the tests code.

  • HDFS-9259: Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario.

  • HDFS-9264: Minor cleanup of operations on FsVolumeList#volumes.

  • HDFS-9273: ACLs on root directory may be lost after NN restart.

  • HDFS-9276: Failed to Update HDFS Delegation Token for long running application in HA mode.

  • HDFS-9289: Make DataStreamer#block thread safe and verify genStamp in commitBlock.

  • HDFS-9290: DFSClient#callAppend() is not backward compatible for slightly older NameNodes.

  • HDFS-9294: DFSClient deadlock when close file and failed to renew lease.

  • HDFS-9305: Delayed heartbeat processing causes storm of subsequent heartbeats.

  • HDFS-9311: Support optional offload of NameNode HA service health checks to a separate RPC server.

  • HDFS-9313: Possible NullPointerException in BlockManager if no excess replica can be chosen.

  • HDFS-9314: Improve BlockPlacementPolicyDefault's picking of excess replicas.

  • HDFS-9317: Document fsck -blockId and -storagepolicy options in branch-2.7.

  • HDFS-9343: Empty caller context considered invalid.

  • HDFS-9347: Invariant assumption in TestQuorumJournalManager.shutdown() is wrong

  • HDFS-9350: Avoid creating temprorary strings in Block.toString() and getBlockName()

  • HDFS-9354: Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows.

  • HDFS-9358: TestNodeCount#testNodeCount timed out.

  • HDFS-9362: TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.

  • HDFS-9364: Unnecessary DNS resolution attempts when creating NameNodeProxies.

  • HDFS-9383: TestByteArrayManager#testByteArrayManager fails.

  • HDFS-9384: TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server.

  • HDFS-9395: Make HDFS audit logging consistant.

  • HDFS-9397: Fix typo for readChecksum() LOG.warn in BlockSender.java.

  • HDFS-9402: Switch DataNode.LOG to use slf4j.

  • HDFS-9406: FSImage may get corrupted after deleting snapshot.

  • HDFS-9412: getBlocks occupies FSLock and takes too long to complete.

  • HDFS-9413: getContentSummary() on standby should throw StandbyException.

  • HDFS-9415: Document dfs.cluster.administrators and dfs.permissions.superusergroup.

  • HDFS-9426: Rollingupgrade finalization is not backward compatible.

  • HDFS-9428: Fix intermittent failure of TestDNFencing.testQueueingWithAppend.

  • HDFS-9431: DistributedFileSystem#concat fails if the target path is relative.

  • HDFS-9434: Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messags.

  • HDFS-9445: Datanode may deadlock while handling a bad volume.

  • HDFS-9466: TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky

  • HDFS-9467: Fix data race accessing writeLockHeldTimeStamp in FSNamesystem.

  • HDFS-9470: Encryption zone on root not loaded from fsimage after NN restart. Xiao Chen via wang.

  • HDFS-9476: TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail

  • HDFS-9478: Reason for failing ipc.FairCallQueue contruction should be thrown.

  • HDFS-9484: NNThroughputBenchmark$BlockReportStats should not send empty block reports.

  • HDFS-9493: Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk.

  • HDFS-9500: Fix software version counts for DataNodes during rolling upgrade.

  • HDFS-9505: HDFS Architecture documentation needs to be refreshed.

  • HDFS-9516: Truncate file fails with data dirs on multiple disks.

  • HDFS-9517: Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes

  • HDFS-9521: TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk.

  • HDFS-9525: hadoop utilities need to support provided delegation tokens.

  • HDFS-9530: ReservedSpace is not cleared for abandoned Blocks

  • HDFS-9533: seen_txid in the shared edits directory is modified during bootstrapping.

  • HDFS-9534: Add CLI command to clear storage policy from a path.

  • HDFS-9549: TestCacheDirectives#testExceedsCapacity is flaky

  • HDFS-9555: LazyPersistFileScrubber should still sleep if there are errors in the clear progress.

  • HDFS-9557: Reduce object allocation in PB conversion.

  • HDFS-9566: Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method

  • HDFS-9569: Log the name of the fsimage being loaded for better supportability.

  • HDFS-9572: Prevent DataNode log spam if a client connects on the data transfer port but sends no data.

  • HDFS-9574: Reduce client failures during datanode restart.

  • HDFS-9584: NPE in distcp when ssl configuration file does not exist in class path.

  • HDFS-9589: Block files which have been hardlinked should be duplicated before the DataNode appends to the them

  • HDFS-9600: Do not check replication if the block is under construction.

  • HDFS-9605: Add links to failed volumes to explorer.html in HDFS Web UI.

  • HDFS-9608: Disk IO imbalance in HDFS with heterogeneous storages.

  • HDFS-9612: DistCp worker threads are not terminated after jobs are done.

  • HDFS-9618: Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks

  • HDFS-9619: SimulatedFSDataset sometimes can not find blockpool for the correct namenode

  • HDFS-9623: Update example configuration of block state change log in log4j.properties.

  • HDFS-9624: DataNode start slowly due to the initial DU command operations.

  • HDFS-9625: set replication for empty file failed when set storage policy

  • HDFS-9626: TestBlockReplacement#testBlockReplacement fails occasionally.

  • HDFS-9629: Update the footer of Web UI to show year 2016

  • HDFS-9630: DistCp minor refactoring and clean up.

  • HDFS-9634: webhdfs client side exceptions don't provide enough details.

  • HDFS-9638: to branch-2.7.

  • HDFS-9644: Update encryption documentation to reflect nested EZs.

  • HDFS-9655: NN should start JVM pause monitor before loading fsimage.

  • HDFS-9661: Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage moveBlockAcrossStorage and createRbw.

  • HDFS-9669: TcpPeerServer should respect ipc.server.listen.queue.size

  • HDFS-9670: DistCp throws NPE when source is root.

  • HDFS-9672: o.a.h.hdfs.TestLeaseRecovery2 fails intermittently

  • HDFS-9688: Test the effect of nested encryption zones in HDFS downgrade.

  • HDFS-9696: Garbage snapshot records linger forever.

  • HDFS-9701: DN may deadlock when hot-swapping under load.

  • HDFS-9706: Log more details in debug logs in BlockReceiver's constructor.

  • HDFS-9710: DN can be configured to send block receipt IBRs in batches.

  • HDFS-9711: Integrate CSRF prevention filter in WebHDFS.

  • HDFS-9713: DataXceiver#copyBlock should return if block is pinned.

  • HDFS-9715: Check storage ID uniqueness on datanode startup

  • HDFS-9721: Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference

  • HDFS-9724: Degraded performance in WebHDFS listing as it does not reuse ObjectMapper.

  • HDFS-9726: Refactor IBR code to a new class.

  • HDFS-9730: Storage ID update does not happen when there is a layout change.

  • HDFS-9739: DatanodeStorage.isValidStorageId() is broken.

  • HDFS-9740: Use a reasonable limit in DFSTestUtil.waitForMetric().

  • HDFS-9743: Fix TestLazyPersistFiles#testFallbackToDiskFull in branch-2.7.

  • HDFS-9748: Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice.

  • HDFS-9752: Permanent write failures may happen to slow writers during datanode rolling upgrades

  • HDFS-9760: WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler

  • HDFS-9764: DistCp doesn't print value for several arguments including -numListstatusThreads.

  • HDFS-9765: TestBlockScanner#testVolumeIteratorWithCaching fails intermittently.

  • HDFS-9766: TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently.

  • HDFS-9768: Reuse ObjectMapper instance in HDFS to improve the performance.

  • HDFS-9772: TestBlockReplacement#testThrottler doesn't work as expected.

  • HDFS-9777: Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh).

  • HDFS-9779: . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value.

  • HDFS-9781: FsDatasetImpl#getBlockReports can occasionally throw NullPointerException.

  • HDFS-9784: Example usage is not correct in Transparent Encryption document.

  • HDFS-9790: HDFS Balancer should exit with a proper message if upgrade is not finalized

  • HDFS-9797: Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level

  • HDFS-9799: Reimplement getCurrentTrashDir to remove incompatibility.

  • HDFS-9812: Streamer threads leak if failure happens when closing DFSOutputStream.

  • HDFS-9839: Reduce verbosity of processReport logging.

  • HDFS-9842: dfs.datanode.balance.bandwidthPerSec should accept friendly size units.

  • HDFS-9844: Correct path creation in getTrashRoot to handle root dir.

  • HDFS-9851: NameNode throws NPE when setPermission is called on a path that does not exist.

  • HDFS-9854: Log cipher suite negotiation more verbosely.

  • HDFS-9864: Correct reference for RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN in webhdfs doc.

  • HDFS-9865: TestBlockReplacement fails intermittently in trunk

  • HDFS-9871: "Bytes Being Moved" -ve(-1 B) when cluster was already balanced.

  • HDFS-9874: Long living DataXceiver threads cause volume shutdown to block.

  • HDFS-9880: TestDatanodeRegistration fails occasionally.

  • HDFS-9881: DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones.

  • HDFS-9882: Add heartbeatsTotal in Datanode metrics.

  • HDFS-9902: Support different values of dfs.datanode.du.reserved per storage type.

  • HDFS-9904: testCheckpointCancellationDuringUpload occasionally fails.

  • HDFS-9905: WebHdfsFileSystem#runWithRetry should display original stack trace on error.

  • HDFS-9906: Remove spammy log spew when a datanode is restarted.

  • HDFS-9917: IBR accumulate more objects when SNN was down for sometime.

  • HDFS-9926: MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils.

  • HDFS-9941: Do not log StandbyException on NN, other minor logging fixes.

  • HDFS-9947: Block#toString should not output information from derived classes

  • HDFS-9949: Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared

  • HDFS-9958: BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.

  • HDFS-10178: Permanent write failures can happen if pipeline recoveries occur for the first packet.

  • HDFS-10182: Hedged read might overwrite user's buf.

  • HDFS-10186: DirectoryScanner: Improve logs by adding full path of both actual and expected block directories.

  • HDFS-10189: PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE

  • HDFS-10199: Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing.

  • HDFS-10216: Distcp -diff throws exception when handling relative path.

  • HDFS-10217: show 'blockScheduled' tooltip in datanodes table.

  • HDFS-10223: peerFromSocketAndKey performs SASL exchange before setting connection timeouts.

  • HDFS-10225: DataNode hot swap drives should disallow storage type changes.

  • HDFS-10228: TestHDFSCLI fails.

  • HDFS-10235: Last contact for Live Nodes should be relative time.

  • HDFS-10239: Fsshell mv fails if port usage doesn't match in src and destination paths.

  • HDFS-10245: Fix the findbugs warnings in branch-2.7.

  • HDFS-10253: Fix TestRefreshCallQueue failure

  • HDFS-10261: TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports.

  • HDFS-10264: Logging improvements in FSImageFormatProtobuf.Saver.

  • HDFS-10270: TestJMXGet:testNameNode() fails.

  • HDFS-10271: Extra bytes are getting released from reservedSpace for append

  • HDFS-10275: TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly.

  • HDFS-10277: PositionedReadable test testReadFullyZeroByteFile failing in HDFS.

  • HDFS-10279: Improve validation of the configured number of tolerated failed volumes.

  • HDFS-10281: TestPendingCorruptDnMessages fails intermittently.

  • HDFS-10291: TestShortCircuitLocalRead failing

  • HDFS-10301: BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order .

  • HDFS-10301: Remove FBR tracking state to fix false zombie storage detection for interleaving block reports.

  • HDFS-10309: Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga).

  • HDFS-10312: Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.

  • HDFS-10313: Distcp need to enforce the order of snapshot names passed to -diff.

  • HDFS-10319: Balancer should not try to pair storages with different types.

  • HDFS-10320: Rack failures may result in NN terminate.

  • HDFS-10324: Trash directory in an encryption zone should be pre-created with correct permissions.

  • HDFS-10329: Bad initialisation of StringBuffer in RequestHedgingProxyProvider.

  • HDFS-10330: Add Corrupt Blocks Information in Metasave output.

  • HDFS-10335:Mover$Processor#chooseTarget() always chooses the first matching target storage group.

  • HDFS-10341: Add a metric to expose the timeout number of pending replication blocks

  • HDFS-10342: BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt.

  • HDFS-10343: BlockManager#createLocatedBlocks may return blocks on failed storages.

  • HDFS-10344: DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash.

  • HDFS-10347: Namenode report bad block method doesn't log the bad block or datanode.

  • HDFS-10360: DataNode may format directory and lose blocks if current/VERSION is missing.

  • HDFS-10367: TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception.

  • HDFS-10372: Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume.

  • HDFS-10377: CacheReplicationMonitor shutdown log message should use INFO level

  • HDFS-10390: Implement asynchronous setAcl/getAclStatus for DistributedFileSystem

  • HDFS-10396: Using -diff option with DistCp may get "Comparison method violates its general contract" exception.

  • HDFS-10397: Distcp should ignore -delete option if -diff option is provided instead of exiting.

  • HDFS-10410: RedundantEditLogInputStream.LOG is set to wrong class.

  • HDFS-10415: TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called

  • HDFS-10417: Improve error message from checkBlockLocalPathAccess.

  • HDFS-10423: Increase default value of httpfs maxHttpHeaderSize.

  • HDFS-10424: DatanodeLifelineProtocol not able to use under security cluster

  • HDFS-10431: Refactor and speedup TestAsyncDFSRename

  • HDFS-10437: ReconfigurationProtocol not covered by HDFSPolicyProvider.

  • HDFS-10438: When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.

  • HDFS-10440: Improve DataNode web UI

  • HDFS-10448: CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1

  • HDFS-10455: Logging the username when deny the setOwner operation.

  • HDFS-10457: DataNode should not auto-format block pool directory if VERSION is missing.

  • HDFS-10458: getFileEncryptionInfo should return quickly for non-encrypted cluster.

  • HDFS-10468: HDFS read ends up ignoring an interrupt.

  • HDFS-10469: Add number of active xceivers to datanode metrics.

  • HDFS-10471: DFSAdmin#SetQuotaCommand's help msg is not correct.

  • HDFS-10474: hftp copy fails when file name with Chinese+special char in branch-2

  • HDFS-10481: HTTPFS server should correctly impersonate as end user to open file.

  • HDFS-10485: Fix findbugs warning in FSEditLog.java

  • HDFS-10488: Update WebHDFS documentation regarding CREATE and MKDIR default permissions.

  • HDFS-10493: Add links to datanode web UI in namenode datanodes page.

  • HDFS-10499: TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently.

  • HDFS-10508: DFSInputStream should set thread's interrupt status after catching InterruptException from sleep.

  • HDFS-10512: VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks.

  • HDFS-10525: Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap

  • HDFS-10544: Balancer doesn't work with IPFailoverProxyProvider.

  • HDFS-10556: DistCpOptions should be validated automatically.

  • HDFS-10569: A bug causes OutOfIndex error in BlockListAsLongs.

  • HDFS-10579: HDFS web interfaces lack configs for X-FRAME-OPTIONS protection.

  • HDFS-10589: Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params.

  • HDFS-10609: Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications.

  • HDFS-10617: PendingReconstructionBlocks.size() should be synchronized.

  • HDFS-10623: Remove unused import of httpclient.HttpConnection from TestWebHdfsTokens.

  • HDFS-10625: VolumeScanner to report why a block is found bad.

  • HDFS-10627: Volume Scanner marks a block as "suspect" even if the exception is network-related.

  • HDFS-10642: TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently.

  • HDFS-10643: [Unit Test] Namenode should use loginUser(hdfs) to generateEncryptedKey.

  • HDFS-10643: HDFS namenode should always use service user

  • HDFS-10653: Optimize conversion from path string to components.

  • HDFS-10655: Fix path related byte array conversion bugs.

  • HDFS-10660: Expose storage policy apis via HDFSAdmin interface.

  • HDFS-10684: WebHDFS DataNode calls fail without parameter createparent.

  • HDFS-10688: BPServiceActor may run into a tight loop for sending block report when hitting IOException.

  • HDFS-10703: HA NameNode Web UI should show last checkpoint time.

  • HDFS-10710: In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui.

  • HDFS-10713: Throttle FsNameSystem lock warnings.

  • HDFS-10715: NPE when applying AvailableSpaceBlockPlacementPolicy.

  • HDFS-10716: In Balancer, the target task should be removed when its size < 0.

  • HDFS-10735:Distcp using webhdfs on secure HA clusters fails with StandbyException .

  • HDFS-10752: Several log refactoring/improvement suggestion in HDFS.

  • HDFS-10757: KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used.

  • HDFS-10763: Open files can leak permanently due to inconsistent lease update.

  • HDFS-10763: Open files can leak permanently due to inconsistent lease update TestTrashWithSecureEncryptionZones diverged from HDFS-10906.

  • HDFS-10764: Fix INodeFile#getBlocks to not return null.

  • HDFS-10793: Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. HDFS-9700. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol.

  • HDFS-10798: Make the threshold of reporting FSNamesystem lock contention configurable.

  • HDFS-10817: Add Logging for Long-held NN Read Locks.

  • HDFS-10822: Log DataNodes in the write pipeline. John Zhuge via Lei Xu.

  • HDFS-10913: Introduce fault injectors to simulate slow mirrors.

  • HDFS-10917: Collect peer performance statistics on DataNode.

  • HDFS-10922: Adding additional unit tests for Trash

  • HDFS-10930: Refactor: Wrap Datanode IO related operations.

  • HDFS-10934: TestDFSShell#testStat fails intermittently.

  • HDFS-10956: Remove rename/delete performance penalty when not using snapshots.

  • HDFS-10959: Update DataNode to use DatasetVolumeChecker.

  • HDFS-10962: TestRequestHedgingProxyProvider is flaky.

  • HDFS-10963: Reduce log level when network topology cannot find enough datanodes.

  • HDFS-10965: Add unit test for HDFS command 'dfsadmin -printTopology'.

  • HDFS-10966: Enhance Dispatcher logic on deciding when to give up a source DataNode.

  • HDFS-10984: Expose nntop output as metrics.

  • HDFS-10987: Make Decommission less expensive when lot of blocks present.

  • HDFS-11003: Expose XmitsInProgress through DataNodeMXBean.

  • HDFS-11008: Change unit test for testing parsing "-source" parameter in Balancer CLI.

  • HDFS-11011: Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'.

  • HDFS-11015: Enforce timeout in balancer.

  • HDFS-11018: Incorrect check and message in FsDatasetImpl#invalidate.

  • HDFS-11031: Add additional unit test for DataNode startup behavior when volumes fail.

  • HDFS-11047: Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode.

  • HDFS-11054: Suppress verbose log message in BlockPlacementPolicyDefault.

  • HDFS-11056: Addendum fix for self-healing branch.

  • HDFS-11056: Concurrent append and read operations lead to checksum error.

  • HDFS-11069: Tighten the authorization of datanode RPC.

  • HDFS-11076: Add unit test for extended Acls.

  • HDFS-11083: Add unit test for DFSAdmin -report command.

  • HDFS-11087: NamenodeFsck should check if the output writer is still writable.

  • HDFS-11112: Journal Nodes should refuse to format non-empty directories.

  • HDFS-11114: Support for running async disk checks in DataNode.

  • HDFS-11119: Support for parallel checking of StorageLocations on DataNode startup.

  • HDFS-11122: TestDFSAdmin#testReportCommand fails due to timed out.

  • HDFS-11128: CreateEditsLog throws NullPointerException.

  • HDFS-11148: Update DataNode to use StorageLocationChecker at startup.

  • HDFS-11149: Fix build break.

  • HDFS-11149: Support for parallel checking of FsVolumes.

  • HDFS-11160: VolumeScanner reports write-in-progress replicas as corrupt incorrectly.

  • HDFS-11180: Intermittent deadlock in NameNode when failover happens.

  • HDFS-11182: Addendum fix for Fenton to fix a unit test.

  • HDFS-11182: Update DataNode to use DatasetVolumeChecker.

  • HDFS-11194: Maintain aggregated peer performance metrics on NameNode.

  • HDFS-11195: Return error when appending files by webhdfs rest api fails.

  • HDFS-11197: Listing encryption zones fails when deleting a EZ that is on a snapshotted directory.

  • HDFS-11209: SNN can't checkpoint when rolling upgrade is not finalized.

  • HDFS-11224: Lifeline message should be ignored for dead nodes

  • HDFS-11229: HDFS-11056 failed to close meta file.

  • HDFS-11240: Remove snapshot version of SDK dependency from Azure Data Lake Store File System.

  • HDFS-11251: ConcurrentModificationException during DataNode#refreshVolumes .

  • HDFS-11253: FileInputStream leak on failure path in BlockSender.

  • HDFS-11258: File mtime change could not save to editlog.

  • HDFS-11263: ClassCastException when we use Bzipcodec for Fsimage compression.

  • HDFS-11267: Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage.

  • HDFS-11274: Datanode should only check the failed volume upon IO errors.

  • HDFS-11279: Cleanup unused DataNode#checkDiskErrorAsync().

  • HDFS-11280: Allow WebHDFS to reuse HTTP connections to NN.

  • HDFS-11282: Document the missing metrics of DataNode Volume IO operations.

  • HDFS-11292: log lastWrittenTxId etc info in logSyncAll.

  • HDFS-11299: Support multiple Datanode File IO hooks.

  • HDFS-11307: The rpc to portmap service for NFS has hardcoded timeout.

  • HDFS-11337:

  • HDFS-11339: Support File IO sampling for Datanode IO profiling hooks.

  • HDFS-11369: Change exception message in StorageLocationChecker.

  • HDFS-11379: DFSInputStream may infinite loop requesting block locations.

  • HDFS-11387: Socket reuse address option is not honored in PrivilegedNfsGatewayStarter.

  • HDFS-11391: Numeric usernames do no work with WebHDFS FS write access.

  • HDFS-11395: RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode.

  • HDFS-11403: Zookeper ACLs on NN HA enabled clusters should be handled consistently.

  • HDFS-11461: DataNode Disk Outlier Detection.

  • HDFS-11462: Fix occasional BindException in TestNameNodeMetricsLogger.

  • HDFS-11476: Fix NPE in FsDatasetImpl#checkAndUpdate

  • HDFS-11477: Simplify file IO profiling configuration.

  • HDFS-11479: Socket re-use address option should be used in SimpleUdpServer.

  • HDFS-11508: Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state.

  • HDFS-11517: Expose slow disks via DataNode JMX.

  • HDFS-11533: reuseAddress option should be used for child channels in Portmap and SimpleTcpServer.

  • MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API

  • MAPREDUCE-6197: Cache MapOutputLocations in ShuffleHandler.

  • MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.

  • MAPREDUCE-6213: NullPointerException caused by job history server addr not resolvable.

  • MAPREDUCE-6273: HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state.

  • MAPREDUCE-6302:

  • MAPREDUCE-6338: MR AppMaster does not honor ephemeral port range.

  • MAPREDUCE-6350: JobHistory doesn't support fully-functional search.

  • MAPREDUCE-6404: Allow AM to specify a port range for starting its webapp.

  • MAPREDUCE-6436: JobHistory cache issue.

  • MAPREDUCE-6436: JobHistory cache issue.

  • MAPREDUCE-6460: TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails.

  • MAPREDUCE-6485: Create a new task attempt with failed map task priority if in-progress attempts are unassigned.

  • MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.

  • MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask.

  • MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers.

  • MAPREDUCE-6543: Migrate MR Client test cases part 2.

  • MAPREDUCE-6549: multibyte delimiters with LineRecordReader cause duplicate records

  • MAPREDUCE-6566: Add retry support to mapreduce CLI tool.

  • MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.

  • MAPREDUCE-6577: MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.

  • MAPREDUCE-6579: TestNetworkedJob.testNetworkedJob failed due to YARN-3946.

  • MAPREDUCE-6593: TestJobHistoryEventHandler.testTimelineEventHandling fails because of NPE.

  • MAPREDUCE-6616: Fail to create jobhistory file if there are some multibyte characters in the job name.

  • MAPREDUCE-6618: YarnClientProtocolProvider leaking the YarnClient thread.(xgong).

  • MAPREDUCE-6621: Memory Leak in JobClient#submitJobInternal().

  • MAPREDUCE-6625: TestCLI#testGetJob fails occasionally

  • MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.

  • MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException.

  • MAPREDUCE-6639: Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.

  • MAPREDUCE-6672: TestTeraSort fails on Windows.

  • MAPREDUCE-6684: High contention on scanning of user directory under immediate_done in Job History Server.

  • MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests.

  • MAPREDUCE-6693: ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit.

  • MAPREDUCE-6701: application master log can not be available when clicking jobhistory's am logs link.

  • MAPREDUCE-6711: After enabling preemption on cluster, jobs failed without making any attempts made after being preempted.

  • MAPREDUCE-6724: Single shuffle to memory must not exceed Integer#MAX_VALUE.

  • MAPREDUCE-6762: ControlledJob#toString failed with NPE when job status is not successfully updated.

  • MAPREDUCE-6789: Fix TestAMWebApp failure.

  • MAPREDUCE-6792: Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir().

  • MAPREDUCE-6797: Job history server scans can become blocked on a single, slow entry.

  • MAPREDUCE-6852: Job#updateStatus() failed with NPE due to race condition.

  • YARN-857: Localization failures should be available in container diagnostics.

  • YARN-998: Keep NM resource updated through dynamic resource config for RM/NM restart.

  • YARN-1376: NM need to notify the log aggregation status to RM through heartbeat.

  • YARN-1402: Update related Web UI and CLI with exposing client API to check log aggregation status.

  • YARN-1462: AHS API and other AHS changes to handle tags for completed MR jobs.

  • YARN-1556: NPE getting application report with a null appId.

  • YARN-1880: Cleanup TestApplicationClientProtocolOnHA

  • YARN-1993: Cross-site scripting vulnerability in TextView.java.

  • YARN-2019: Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore.

  • YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.

  • YARN-2046: Out of band heartbeats are sent only on container kill and possibly too early.

  • YARN-2194: Fix bug causing CGroups functionality to fail on RHEL7.

  • YARN-2392: Add more diags about app retry limits on AM failures.

  • YARN-2495: Allow admin specify labels from each NM

  • YARN-2498: Respect labels in preemption policy of capacity scheduler for inter-queue preemption.

  • YARN-2571:RM to support YARN registry.

  • YARN-2619: Added NodeManager support for disk io isolation through cgroups.

  • YARN-2666: TestFairScheduler.testContinuousScheduling fails Intermittently.

  • YARN-2696: Queue sorting in CapacityScheduler should consider node label.

  • YARN-2740: Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled.

  • YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.

  • YARN-2821: Fixed a problem that DistributedShell AM may hang if restarted.

  • YARN-2868: FairScheduler: Metric for latency to allocate first container for an application.

  • YARN-2901: Add errors and warning metrics page to RM, NM web UI.

  • YARN-3021: YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp.

  • YARN-3026: Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp.

  • YARN-3102: Decommisioned Nodes not listed in Web UI.

  • YARN-3102: Decommisioned Nodes not listed in Web UI.

  • YARN-3110: Few issues in ApplicationHistory web ui.

  • YARN-3136: Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers.

  • YARN-3148: Allow CORS related headers to passthrough in WebAppProxyServlet.

  • YARN-3215: Respect labels in CapacityScheduler when computing headroom.

  • YARN-3216: Max-AM-Resource-Percentage should respect node labels.

  • YARN-3225: New parameter of CLI for decommissioning node gracefully in RMAdmin CLI.

  • YARN-3243: CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.

  • YARN-3248: Display count of nodes blacklisted by apps in the web UI.

  • YARN-3266: RMContext#inactiveNodes should have NodeId as map key.

  • YARN-3269: Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.

  • YARN-3293: Track and display capacity scheduler health metrics in web UI.

  • YARN-3294: Allow dumping of Capacity Scheduler debug logs via web UI for a fixed time period.

  • YARN-3305: Normalize AM resource request on app submission.

  • YARN-3318: Create Initial OrderingPolicy Framework and FifoOrderingPolicy.

  • YARN-3319: Implement a FairOrderingPolicy.

  • YARN-3326: Support RESTful API for getLabelsToNodes.

  • YARN-3343: Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate timeout.

  • YARN-3345: Add non-exclusive node label API.

  • YARN-3347: Improve YARN log command to get AMContainer logs as well as running containers logs.

  • YARN-3348: Add a 'yarn top' tool to help understand cluster usage.

  • YARN-3354: Add node label expression in ContainerTokenIdentifier to support RM recovery.

  • YARN-3356: Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.

  • YARN-3357: Move TestFifoScheduler to FIFO package.

  • YARN-3360: Add JMX metrics to TimelineDataManager .

  • YARN-3361: CapacityScheduler side changes to support non-exclusive node labels.

  • YARN-3362: Add node label usage in RM CapacityScheduler web UI.

  • YARN-3365: Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control.

  • YARN-3366: Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers.

  • YARN-3383: AdminService should use "warn" instead of "info" to log exception when operation fails.

  • YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.

  • YARN-3387: Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time.

  • YARN-3394: Enrich WebApplication proxy documentation.

  • YARN-3397: yarn rmadmin should skip -failover.

  • YARN-3404: Display queue name on application page.

  • YARN-3406: Display count of running containers in the RM's Web UI.

  • YARN-3410: YARN admin should be able to remove individual application records from RMStateStore.

  • YARN-3413: Changed Nodelabel attributes

  • YARN-3425: NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed.

  • YARN-3433: Jersey tests failing with Port in Use -again.

  • YARN-3435: AM container to be allocated Appattempt AM container shown as null.

  • YARN-3443: Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM.

  • YARN-3448: Added a rolling time-to-live LevelDB timeline store implementation.

  • YARN-3451: Display attempt start time and elapsed time on the web UI.

  • YARN-3453: Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing.

  • YARN-3459: Fix failiure of TestLog4jWarningErrorMetricsAppender.

  • YARN-3463: Integrate OrderingPolicy Framework with CapacityScheduler.

  • YARN-3467: Expose allocatedMB, allocatedVCores, and runningContainers metrics on running Applications in RM Web UI.

  • YARN-3480: Remove attempts that are beyond max-attempt limit from state store.

  • YARN-3494: Expose AM resource limit and usage in CS QueueMetrics.

  • YARN-3503: Expose disk utilization percentage and bad local and log dir counts in NM metrics.

  • YARN-3505: Node's Log Aggregation Report with SUCCEED should not cached in RMApps.

  • YARN-3511: Add errors and warnings page to ATS.

  • YARN-3517: RM web ui for dumping scheduler logs should be for admins only

  • YARN-3521: Support return structured NodeLabel objects in REST API.

  • YARN-3530: ATS throws exception on trying to filter results without otherinfo.

  • YARN-3541: Add version info on timeline service / generic history web UI and REST API.

  • YARN-3543: ApplicationReport should be able to tell whether the Application is AM managed or not.

  • YARN-3552: RM Web UI shows -1 running containers for completed apps

  • YARN-3565: NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String.

  • YARN-3579: CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings.

  • YARN-3580: [JDK8] TestClientRMService.testGetLabelsToNodes fails.

  • YARN-3581: Deprecate -directlyAccessNodeLabelStore in RMAdminCLI.

  • YARN-3583: Support of NodeLabel object instead of plain String in YarnClient side.

  • YARN-3589: RM and AH web UI display DOCTYPE wrongly.

  • YARN-3591: Resource localization on a bad disk causes subsequent containers failure.

  • YARN-3593: Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page.

  • YARN-3600: AM container link is broken

  • YARN-3604: removeApplication in ZKRMStateStore should also disable watch.

  • YARN-3632: Ordering policy should be allowed to reorder an application when demand changes.

  • YARN-3635: Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework.

  • YARN-3654: ContainerLogsPage web UI should not have meta-refresh.

  • YARN-3695: ServerProxy

  • YARN-3695: ServerProxy

  • YARN-3700: Made generic history service load a number of latest applications according to the parameter or the configuration.

  • YARN-3707: RM Web UI queue filter doesn't work.

  • YARN-3717: Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API.

  • YARN-3727: For better error recovery, check if the directory exists before using it for localization.

  • YARN-3740: Fixed the typo in the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS.

  • YARN-3751: Fixed AppInfo to check if used resources are null.

  • YARN-3766: Fixed the apps table column error of generic history web UI.

  • YARN-3769: Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler.

  • YARN-3785: Support for Resource as an argument during submitApp call in MockRM test class.

  • YARN-3846: RM Web UI queue filter is not working for sub queue.

  • YARN-3849:(Too much of preemption activity causing continuos killing of containers across queues) to 2.3-maint.

  • YARN-3873: PendingApplications in LeafQueue should also use OrderingPolicy.

  • YARN-3885: ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level.

  • YARN-3894: RM startup should fail for wrong CS xml NodeLabel capacity configuration.

  • YARN-3896: RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously.

  • YARN-3920: FairScheduler container reservation on a node should be configurable to limit it to large containers

  • YARN-3932: SchedulerApplicationAttempt#getResourceUsageReport and UserInfo should based on total-used-resources.

  • YARN-3941: Proportional Preemption policy should try to avoid sending duplicate PREEMPT_CONTAINER event to scheduler.

  • YARN-3946: Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message.

  • YARN-3948: Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks).

  • YARN-3961: Expose pending, running and reserved containers of a queue in REST api and yarn top

  • YARN-3966: Fix excessive loggings in CapacityScheduler.

  • YARN-3971: Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery.

  • YARN-3978: Configurably turn off the saving of container info in Generic AHS

  • YARN-3982: container-executor parsing of container-executor.cfg broken in trunk and branch-2.

  • YARN-3983: Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic.

  • YARN-3986: getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface.

  • YARN-3987: am container complete msg ack to NM once RM receive it.

  • YARN-4000: RM crashes with NPE if leaf queue becomes parent queue during restart.

  • YARN-4023: Publish Application Priority to TimelineServer.

  • YARN-4026: Refactored ContainerAllocator to accept a list of priorites rather than a single priority.

  • YARN-4082: Container shouldn't be killed when node's label updated.

  • YARN-4087: Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs.

  • YARN-4092: Fixed UI redirection to print useful messages when both RMs are in standby mode.

  • YARN-4101: RM should print alert messages if ZooKeeper and Resourcemanager gets connection issue.

  • YARN-4108: CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request.

  • YARN-4113: RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER.

  • YARN-4140: RM container allocation delayed incase of app submitted to Nodelabel partition.

  • YARN-4149: yarn logs -am should provide an option to fetch all the log files.

  • YARN-4155: TestLogAggregationService.testLogAggregationServiceWithInterval failing.

  • YARN-4162: CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API.

  • YARN-4215: RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request.

  • YARN-4225: Add preemption status to yarn queue -status for capacity scheduler.

  • YARN-4233: YARN Timeline Service plugin: ATS v1.5.

  • YARN-4245: Generalize config file handling in container-executor.

  • YARN-4270: Limit application resource reservation on nodes for non-node/rack specific requests

  • YARN-4285: Display resource usage as percentage of queue and cluster in the RM UI

  • YARN-4287: Capacity Scheduler: Rack Locality improvement

  • YARN-4288: Fixed RMProxy to retry on IOException from local host.

  • YARN-4290: Add -showDetails option to YARN Nodes CLI to print all nodes reports information.

  • YARN-4304: AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics.

  • YARN-4309: Add container launch related debug information to container logs when a container fails.

  • YARN-4313: Race condition in MiniMRYarnCluster when getting history server address.

  • YARN-4315: NaN in Queue percentage for cluster apps page.

  • YARN-4329: [YARN-5437] Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler

  • YARN-4345: yarn rmadmin -updateNodeResource doesn't work

  • YARN-4347: Resource manager fails with Null pointer exception.

  • YARN-4349:YARN_APPLICATION call to ATS does not have YARN_APPLICATION_CALLER_CONTEXT.

  • YARN-4365: FileSystemNodeLabelStore should check for root dir existence on startup.

  • YARN-4384: updateNodeResource CLI should not accept negative values for resource.

  • YARN-4390: Do surgical preemption based on reserved container in CapacityScheduler.

  • YARN-4392: ApplicationCreatedEvent event time resets after RM restart/failover.

  • YARN-4393: TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently.

  • YARN-4403:

  • YARN-4405: Support node label store in non-appendable file system.

  • YARN-4414: Nodemanager connection errors are retried at multiple levels.

  • YARN-4414: Nodemanager connection errors are retried at multiple levels.

  • YARN-4416: Deadlock due to synchronised get Methods in AbstractCSQueue.

  • YARN-4417: Make RM and Timeline-server REST APIs more consistent.

  • YARN-4418: AM Resource Limit per partition can be updated to ResourceUsage as well.

  • YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.

  • YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page.

  • YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.

  • YARN-4428: Redirect RM page to AHS page when AHS turned on and RM page is not available.

  • YARN-4439: Clarify NMContainerStatus#toString method.

  • YARN-4454: NM to nodelabel mapping going wrong after RM restart.

  • YARN-4477: FairScheduler: Handle condition which can result in an infinite loop in attemptScheduling.

  • YARN-4479: Change CS LeafQueue pendingOrderingPolicy to hornor recovered apps.

  • YARN-4497: RM might fail to restart when recovering apps whose attempts are missing.

  • YARN-4502: Fix two AM containers get allocated when AM restart.

  • YARN-4522: Queue acl can be checked at app submission.

  • YARN-4535: Fix checkstyle error in CapacityScheduler.java

  • YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.

  • YARN-4546: YARN-4546. ResourceManager crash due to scheduling opportunity overflow.

  • YARN-4556: TestFifoScheduler.testResourceOverCommit fails.

  • YARN-4557: Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=*.

  • YARN-4565:

  • YARN-4573: Fix test failure in TestRMAppTransitions#testAppRunningKill and testAppKilledKilled.

  • YARN-4584: RM startup failure when AM attempts greater than max-attempts.

  • YARN-4591: YARN Web UIs should provide a robots.txt. Junping Du.

  • YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.

  • YARN-4598: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL.

  • YARN-4607: Pagination support for AppAttempt page TotalOutstandingResource Requests table.

  • YARN-4610: Reservations continue looking for one app causes other apps to starve.

  • YARN-4617: LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy.

  • YARN-4623: TestSystemMetricsPublisher#testPublishAppAttemptMetricsForUnmanagedAM fails with NPE on branch-2.7.

  • YARN-4625: Make ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent.

  • YARN-4633: TestRMRestart.testRMRestartAfterPreemption fails intermittently.

  • YARN-4634: Scheduler UI/Metrics need to consider cases like non-queue label mappings.

  • YARN-4667: RM Admin CLI for refreshNodesResources throws NPE when nothing is configured.

  • YARN-4680: Fix TimerTasks leak in ATS V1.5 Writer.

  • YARN-4696: TimelineClient to add flush operation for deterministic writes

  • YARN-4697: NM aggregation thread pool is not bound by limits

  • YARN-4699: Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node.

  • YARN-4709: NMWebServices produces incorrect JSON for containers.

  • YARN-4709: NMWebServices produces incorrect JSON for containers.

  • YARN-4723: NodesListManager$UnknownNodeId ClassCastException.

  • YARN-4737: Add CSRF filter support in YARN

  • YARN-4737: Add CSRF filter support in YARN

  • YARN-4746: yarn web services should convert parse failures of appId, appAttemptId and containerId to 400.

  • YARN-4767: Network issues can cause persistent RM UI outage.

  • YARN-4769: Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI.

  • YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.

  • YARN-4785: Inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API.

  • YARN-4811: Generate histograms in ContainerMetrics for actual container resource usage.

  • YARN-4814: ATS 1.5 timelineclient impl call flush after every event write.

  • YARN-4815: ATS 1.5 timelineclinet impl try to create attempt directory for every event call.

  • YARN-4817: TimelineClient ATSv1.5 logging is very noisy.

  • YARN-4822: Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates.

  • YARN-4832: NM side resource value should get updated if change applied in RM side.

  • YARN-4842: Fixed "yarn logs" command to guess

  • YARN-4844: Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource.

  • YARN-4846: Fix random failures for TestCapacitySchedulerPreemption #testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers.

  • YARN-4865: Track Reserved resources in ResourceUsage and QueueCapacities.

  • YARN-4890: Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate.

  • YARN-4893: Fix intermittent failures in TestRMAdminService.

  • YARN-4905: Improved "yarn logs" command-line to optionally show log metadata also.

  • YARN-4913: Yarn logs should take a -out option to write to a directory.

  • YARN-4916: TestNMProxy.tesNMProxyRPCRetry fails.

  • YARN-4920: ATS/NM should support a link to dowload/get the logs in text format.

  • YARN-4934: Reserved Resource for QueueMetrics needs to be handled correctly in few cases.

  • YARN-4955: Add retry for SocketTimeoutException in TimelineClient.

  • YARN-4966: Improve yarn logs to fetch container logs without specifying nodeId.

  • YARN-4984: LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak.

  • YARN-4990: Re-direction of a particular log file within in a container in NM UI does not redirect properly.

  • YARN-5023: TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry random failure.

  • YARN-5048: DelegationTokenRenewer#skipTokenRenewal may throw NPE.

  • YARN-5069: TestFifoScheduler.testResourceOverCommit race condition.

  • YARN-5076: YARN web interfaces lack XFS

  • YARN-5080: Cannot obtain logs using YARN CLI -am for either KILLED or RUNNING AM.

  • YARN-5083: YARN CLI for AM logs does not give any error message if entered invalid am value.

  • YARN-5088: Improve "yarn log" command-line to read the last K bytes for the log files.

  • YARN-5089: Improve "yarn log" command-line "logFiles" option to support regex.

  • YARN-5112: Excessive log warnings for directory permission issue on NM recovery.

  • YARN-5115: Avoid setting CONTENT-DISPOSITION header in the container-logs web-service.

  • YARN-5121: fix some container-executor portability issues.

  • YARN-5122: "yarn logs" for running containers should print an explicit footer saying that the log may be incomplete.

  • YARN-5131: Distributed shell AM fails when extra container arrives during finishing. Wangda Tan via junping_du.

  • YARN-5141: Get Container logs for the Running application from Yarn Logs CommandLine.

  • YARN-5190: Registering/unregistering container metrics triggered by ContainerEvent and ContainersMonitorEvent are conflict which cause uncaught exception in ContainerMonitorImpl.

  • YARN-5191: Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option.

  • YARN-5199: Close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs.

  • YARN-5200: Enhanced "yarn logs" to be able to get a list of containers whose logs are aggregated via a "show_container_log_info" option.

  • YARN-5206: RegistrySecurity includes id:pass in exception text if considered invalid.

  • YARN-5208: Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled.

  • YARN-5213: Fix a bug in LogCLIHelpers which cause TestLogsCLI#testFetchApplictionLogs fails intermittently.

  • YARN-5214: Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks.

  • YARN-5223: Container line in yarn logs output for a live application should include the hostname for the container.

  • YARN-5224: Added new web-services /containers/{containerid}/logs & /containers/{containerid}/logs/{filename} and using them in "yarn logs" CLI to get logs of finished containers of a running application.

  • YARN-5226: Remove AHS enable check from LogsCLI#fetchAMContainerLogs.

  • YARN-5227: Yarn logs command: no need to specify applicationId when specifying containerId.

  • YARN-5251: Yarn CLI to obtain App logs for last 'n' bytes fails.

  • YARN-5266: Wrong exit code while trying to get app logs using regex via CLI.

  • YARN-5270: Solve miscellaneous issues caused by YARN-4844.

  • YARN-5294: Pass remote ip address down to YarnAuthorizationProvider.

  • YARN-5296: NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl.

  • YARN-5297: Avoid printing a stack trace when recovering an app after the RM restarts.

  • YARN-5337: Fix Dshell OOM issues.

  • YARN-5339: Passing file to -out for YARN log CLI doesnt give warning or error code.

  • YARN-5353: ResourceManager can leak delegation tokens when they are shared across apps.

  • YARN-5363: For AM containers, or for containers of running-apps, "yarn logs" incorrectly only

  • YARN-5456: container-executor support for FreeBSD, NetBSD, and others if conf path is absolute.

  • YARN-5470: Differentiate exactly match with regex in yarn log CLI.

  • YARN-5476: Not existed application reported as ACCEPTED state by YarnClientImpl.

  • YARN-5523: Yarn running container log fetching causes OutOfMemoryError

  • YARN-5524: Yarn live log aggregation does not throw if command line arg is wrong.

  • YARN-5539: TimelineClient failed to retry on java.net.SocketTimeoutException: Read timed out.

  • YARN-5549: AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately.

  • YARN-5551: Ignore file backed pages from memory computation when smaps is enabled.

  • YARN-5659: getPathFromYarnURL should use standard methods.

  • YARN-5677: RM should transition to standby when connection is lost for an extended period.

  • YARN-5704: Provide config knobs to control enabling/disabling new/work in progress features in container-executor.

  • YARN-5711: Propogate exceptions back to client when using hedging RM failover provider.

  • YARN-5718: TimelineClient

  • YARN-5773: RM recovery too slow due to LeafQueue#activateApplications.

  • YARN-5864: Capacity Scheduler queue priority.

  • YARN-5873: RM crashes with NPE if generic application history is enabled.

  • YARN-5915: ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write.

  • YARN-5923: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled.

  • YARN-5999: AMRMClientAsync will stop if any exceptions thrown on allocate call.

  • YARN-6011: Add a new web service to list the files on a container in AHSWebService.

  • YARN-6029: CapacityScheduler deadlock when ParentQueue#getQueueUserAclInfo is called by one thread and LeafQueue#assignContainers is releasing excessive reserved container by another thread.

  • YARN-6068: Log aggregation get failed when NM restart even with recovery.

  • YARN-6081:LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container.

  • YARN-6099: Improve webservice to list aggregated log files.

  • YARN-6100: Improve YARN webservice to output aggregated container logs.

  • YARN-6108: Improve AHS webservice to accept NM address as a parameter to get container logs.

  • YARN-6113: Re-direct NM Web Service to get container logs for finished applications.

  • YARN-6123: Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed.

  • YARN-6126: Obtaining app logs for Running application fails with json parse error.

  • YARN-6137: Yarn client implicitly invoke ATS client which accesses HDFS.

  • YARN-6145: Improve log message on fail over.

  • YARN-6174: Log files pattern should be same for both running and finished container.

  • YARN-6189: ApplicationNotFoundException due to RM failed over should throw more user friendly exception.

  • YARN-6228: EntityGroupFSTimelineStore should allow configurable cache stores.

  • YARN-6270: WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting.

  • YARN-6313: yarn logs cli should provide logs for a completed container even when application is still running.

  • YARN-6314: Potential infinite redirection on YARN log redirection web service.

  • YARN-6332: Make RegistrySecurity use short user names for ZK ACLs.

  • YARN-6367: YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService.