Hortonworks Data Platform
Also available as:
PDF

Hadoop

This release provides Hadoop Common 3.1.1 and the listed Apache patches.

  • HADOOP-15571: Multiple FileContexts created with the same configuration object should be allowed to have different umask. Contributed by Vinod Kumar Vavilapalli.

  • HDFS-13712: BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.

  • HDFS-13715:diskbalancer does not work if one of the blockpools are empty on a Federated cluster. Contributed by Bharat Viswanadham

  • HDFS-13715: Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak.

  • HDFS-13635: Incorrect message when block is not found. Contributed by Gabor Bota.

  • HADOOP-15554: Improve JIT performance for Configuration parsing. Contributed by Todd Lipcon.

  • HDFS-13702: Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.

  • HDFS-13703: Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.

  • HADOOP-15574: Suppress build error if there are no docs after excluding private annotations. Contributed by Takanobu Asanuma.

  • HADOOP-15548: Randomize local dirs. Contributed by Jim Brennan.

  • HDFS-13705:The native ISA-L library loading failure should be made warning rather than an error message. Contributed by Shashikant Banerjee.

  • HADOOP-15458: TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows. Contributed by Xiao Liang.

  • HDFS-13692: StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham.

  • HDFS-13682: Cannot create encryption zone after KMS auth token expires.

  • HADOOP-15527: Improve delay check for stopping processes. Fixed script location. Contributed by Vinod Kumar Vavilapalli

  • HADOOP-15527: Improve delay check for stopping processes. Contributed by Vinod Kumar Vavilapalli

  • HADOOP-15533: Make WASB listStatus messages consistent. Contributed by Esfandiar Manii

  • HADOOP-15523: Shell command timeout given is in seconds whereas it is taken as millisec while scheduling. Contributed by Bilwa S T.

  • HDFS-13681: Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang.

  • HDFS-13686: Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak.

  • HDFS-13174: hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth.

  • HDFS-13676: TestEditLogRace fails on Windows. Contributed by Zuoming Zhang.

  • HDFS-13673: TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang.

  • HDFS-13675 Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak.

  • HDFS-13563: TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak.

  • HDFS-13641: Add metrics for edit log tailing. Contributed by Chao Sun.

  • BUG-103209:Dependency upgrades related to blackduck scan.

  • HADOOP-15527: Improve delay check for stopping processes. Contributed by Vinod Kumar Vavilapalli

  • HADOOP-15532: TestBasicDiskValidator fails with NoSuchFileException. Contributed by Giovanni Matteo Fumarola.

  • HDFS-13511: Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota.

  • HADOOP-15529: ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows. Contributed by Giovanni Matteo Fumarola.

  • HADOOP-15506: Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. Contributed by Esfandiar Manii.

  • HDFS-13653: Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.

  • HADOOP-15499: Performance severe drops when running RawErasureCoderBenchmark with NativeRSRawErasureCoder. Contributed by Sammi Chen.

  • HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar

  • HDFS-12670: can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal.

  • HDFS-13664: Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.

  • HADOOP-15516: Add test cases to cover FileUtil#readLink. Contributed by Giovanni Matteo Fumarola.

  • HDFS-13659: Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang.

  • HDFS-12950: [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang.

  • HADOOP-15217: FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.

  • HDFS-13545: "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia.

  • HDFS-13650: Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu.

  • HDFS-13649: Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu.

  • HDFS-13652: Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu.

  • HADOOP-15137: ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster. Contributed by Bharat Viswanadham

  • HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah

  • HDFS-13155: BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel.

  • HDFS-13339: Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel.

  • HDFS-13648: Fix TestGetConf#testGetJournalNodes on Windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola.

  • HDFS-13651: TestReencryptionHandler fails on Windows. Contributed by Anbang Hu.

  • MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user(addendum). Contributed by Sunil Govindan.

  • HDFS-13637: RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE. Contributed by CR Hota.

  • HDFS-13636: Cross-Site Scripting vulnerability in HttpServer2 (Contributed by Haibo Yan via Daniel Templeton)

  • HDFS-13582: Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.

  • HDFS-13602: Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun.

  • HDFS-12978: Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. (cherry picked from commit 442dd87dcd3e9913a1d97793fde879ce51d701e7)

  • HDFS-13631: TestDFSAdmin#testCheckNumOfBlocksInReportCommand should use a separate MiniDFSCluster path. Contributed by Anbang Hu.

  • BUG-104243:Dependency jackson-databind dependency related to blackduck scan.

  • HADOOP-15406: hadoop-nfs dependencies for mockito and junit are not test scope

  • HDFS-13611: Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.

  • HDFS-13629: Some tests in TestDiskBalancerCommand fail on Windows due to MiniDFSCluster path conflict and improper path usage. Contributed by Anbang Hu.

  • HDFS-13632: Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu.

  • Additional check when unpacking archives. Contributed by Wilfred Spiegelenburg.

  • HADOOP-15497: TestTrash should use proper test path to avoid failing on Windows. Contributed by Anbang Hu.

  • HADOOP-15498: TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu.

  • HDFS-13591: TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu.

  • HDFS-13627: TestErasureCodingExerciseAPIs fails on Windows. Contributed by Anbang Hu.

  • HADOOP-15449: Increase default timeout of ZK session to avoid frequent NameNode failover

  • HDFS-13628: Update Archival Storage doc for Provided Storage

  • HDFS-13620: Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu.

  • HDFS-13619: TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu.

  • HADOOP-15473: Configure serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997. Contributed by Gabor Bota.

  • HDFS-13618: Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang.

  • HADOOP-15486: Make NetworkTopology#netLock fair. Contributed by Nanda kumar.

  • Skip the proxy user check if the ugi has not been initialized. Contributed by Daryn Sharp

  • HDFS-13540: DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.

  • HADOOP-15478: WASB: hflush() and hsync() regression. Contributed by Thomas Marquardt.

  • HDFS-13493: Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen.

  • Additional check when unpacking archives. Contributed by Jason Lowe and Akira Ajisaka.

  • HDFS-13587: TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu.

  • HDFS-13588: Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang.

  • HADOOP-15450: Avoid fsync storm triggered by DiskChecker and handle disk full situation. Contributed by Arpit Agarwal.

  • HDFS-13601: Optimize ByteString conversions in PBHelper.

  • HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru

  • HADOOP-15299: Bump Jackson 2 version to Jackson 2.9.x.

  • HDFS-13593: TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu.

  • HDFS-13592: TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu.

  • HDFS-13586: Fsync fails on directories on Windows. Contributed by Lukas Majercack.

  • HDFS-13560: Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu.

  • HDFS-13556: TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu.

  • HDFS-13554: TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu.

  • HDFS-13558: TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu.

  • BUG-103397. HDFS-13589. Add dfsAdmin command to query if upgrade is finalized.

  • HADOOP-15469: S3A directory committer commit job fails if _temporary directory created under dest. Contributed by Steve Loughran.

  • HADOOP-15250: Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong Contributed by Ajay Kumar

  • HDFS-13570: TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu.

  • HDFS-13581: DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee.

  • HDFS-13559: TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu.

  • HDFS-13512: WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.

  • HDFS-13550:TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu.

  • HDFS-13557: TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu.

  • BUG-103055 Remove aiyun, openstack from hadoop-cloud-project

  • HDFS-13567: TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu.

  • HDFS-13548: TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu.

  • HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu.

  • HDFS-13551: TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu.

  • HADOOP-15442: ITestS3AMetrics.testMetricsRegister can't know metrics source's name. Contributed by Sean Mackrory.

  • HDFS-13544: Improve logging for JournalNode in federated cluster.

  • HDFS-13539 DFSStripedInputStream NPE when reportCheckSumFailure.

  • Add 2.9.1 release notes and changes documents

  • BUG-102169 Include Google Connector in Hadoop

  • HADOOP-15441: Log kms url and token service at debug level. Contributed by Gabor Bota

  • HDFS-13542: TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu.

  • HDFS-13346: RBF: Fix synchronization of router quota and nameservice quota.

  • HADOOP-15454: TestRollingFileSystemSinkWithLocal fails on Windows. Contributed by Xiao Liang.

  • HDFS-13537: TestHdfsHelper does not generate jceks path properly for relative path in Windows. Contributed by Xiao Liang.

  • HDFS-12981: renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi.

  • HADOOP-15446: WASB: PageBlobInputStream.skip breaks HBASE replication. Contributed by Thomas Marquardt

  • HADOOP-14841 Kms client should disconnect if unable to get output stream from connection. Contributed by Rushabh S Shah

  • HDFS-13525: RBF: Add unit test TestStateStoreDisabledNameservice. Contributed by Yiqun Lin.

  • HDFS-13481: TestRollingFileSystemSinkWithHdfs#testFlushThread: test failed intermittently (Contributed by Gabor Bota via Daniel Templeton)

  • HDFS-13512 . WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.

  • HDFS-13488: RBF: Reject requests when a Router is overloaded. Contributed by Inigo Goiri.

  • HDFS-13503: Fix TestFsck test failures on Windows. Contributed by Xiao Liang.

  • HADOOP-15390: Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter)

  • HDFS-13283: Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak.

  • HDFS-13509: Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows. Contributed by Xiao Liang.

  • HDFS-13508: RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries. Contributed by Ekanth S.

  • HDFS-13484: RBF: Disable Nameservices from the federation. Contributed by Inigo Goiri.

  • HDFS-13434: RBF: Fix dead links in RBF document. Contributed by Chetna Chaudhari.

  • HDFS-13490: RBF: Fix setSafeMode in the Router. Contributed by Inigo Goiri.

  • HDFS-13499: RBF: Show disabled name services in the UI. Contributed by Inigo Goiri.

  • HDFS-13326: RBF: Improve the interfaces to modify and view mount tables. Contributed by Gang Li.

  • HADOOP-15411: AuthenticationFilter should use Configuration.getPropsWithPrefix instead of iterator. (Suma Shivaprasad via wangda)

  • HDFS-13336: Test cases of TestWriteToReplica failed in windows. Contributed by Xiao Liang.

  • HDFS-13356: Balancer:Set default value of minBlockSize to 10mb. Contributed by Bharat Viswanadham

  • BUG-101422. Disable HDFS and Hadoop unit tests that are flaky in Mawo.

  • HDFS-13408: MiniDFSCluster to support being built on randomized base directory. Contributed by Xiao Liang

  • HDFS-13433: webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal.

  • HDFS-13478: RBF: Disabled Nameservice store API. Contributed by Inigo Goiri.

  • HDFS-10183: Prevent race condition during class initialization. Contributed by Pavel Avgustinov.

  • HDFS-13453: RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table. Contributed by Dibyendu Karmakar.

  • HADOOP-15396: Some java source files are executable

  • HDFS-13464: Fix javadoc in FsVolumeList#handleVolumeFailures. Contributed by Shashikant Banerjee

  • HADOOP-15366: Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown. Contributed by Shashikant Banerjee.

  • HDFS-13426: Fix javadoc in FsDatasetAsyncDiskService#removeVolume. Contributed by Shashikant Banerjee.

  • HDFS-13463: Fix javadoc in FsDatasetImpl#checkAndUpdate. Contributed by Shashikant Banerjee.

  • HADOOP-15391: Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules.

  • HDFS-12828: OIV ReverseXML Processor fails with escaped characters

  • BUG-100895. Ignore UTs failing due to environmental issues.

  • HDFS-13439: Add test case for read block operation when it is moved. Contributed by Ajay Kumar.

  • HDFS-13435: RBF: Improve the error loggings for printing the stack trace.

  • HDFS-13330: ShortCircuitCache#fetchOrCreate never retries. Contributed by Gabor Bota.

  • HDFS-13311: RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows. Contributed by Inigo Goiri

  • HDFS-13056: Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo.

  • HDFS-13466: RBF: Add more router-related information to the UI.

  • HDFS-13462: Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers. Contributed by Lukas Majercak.

  • HDFS-13129: Add a test for DfsAdmin refreshSuperUserGroupsConfiguration. Contributed by Mukul Kumar Singh

  • HDFS-13433: webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal.

  • HDFS-11900: Hedged reads thread pool creation not synchronized. Contributed by John Zhuge.

  • HDFS-11043: TestWebHdfsTimeouts fails. Contributed by Xiaoyu Yao and Chao Sun.

  • HDFS-13315: Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997. Contributed by Yongjun Zhang.

  • HADOOP-15332: Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.

  • HADOOP-15331: Fix a race condition causing parsing error of java.io.BufferedInputStream in class org.apache.hadoop.conf.Configuration. Contributed by Miklos Szegedi.

  • HADOOP-15330 Remove jdk1.7 profile from hadoop-annotations module

  • HADOOP-15062: TestCryptoStreamsWithOpensslAesCtrCryptoCodec fails on Debian 9. Contributed by Miklos Szegedi.

  • HADOOP-14667: Flexible Visual Studio support. Contributed by Allen Wittenauer

  • HDFS-13438: Fix javadoc in FsVolumeList#removeVolume. Contributed by Shashikant Banerjee.

  • HDFS-13418: NetworkTopology should be configurable when enable DFSNetworkTopology. Contributed by Tao Jie.

  • HADOOP-14970: MiniHadoopClusterManager doesn't respect lack of format option. Contributed by Erik Krogen

  • HADOOP-15379: Make IrqHandler.bind() public. Contributed by Ajay Kumar

  • HDFS-13436: Fix javadoc of package-info.java

  • HDFS-7101: Potential null dereference in DFSck#doWork(). Contributed by skrho and Ted Yu.

  • HDFS-13427: Fix the section titles of transparent encryption document.

  • HDFS-13386: RBF: Wrong date information in list file(-ls) result. Contributed by Dibyendu Karmakar.

  • HDFS-13428: RBF: Remove LinkedList From StateStoreFileImpl.java. Contributed by BELUGA BEHR.

  • HDFS-13045: RBF: Improve error message returned from subcluster. Contributed by Inigo Goiri.

  • HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran.

  • HADOOP-15313: TestKMS should close providers.

  • HADOOP-15357: Configuration.getPropsWithPrefix no longer does variable substitution. Contributed by Jim Brennan

  • HDFS-13328: Abstract ReencryptionHandler recursive logic in separate class. Contributed by Surendra Singh Lilhore.

  • HADOOP-15376: Remove double semi colons on imports that make Clover fall over.

  • HDFS-13420: License header is displayed in ArchivalStorage/MemoryStorage html pages. Contributed by Akira Ajisaka.

  • HADOOP-15374: Add links of the new features of 3.1.0 to the top page

  • HDFS-13384: RBF: Improve timeout RPC call mechanism. Contributed by Inigo Goiri.

  • HDFS-13410: RBF: Support federation with no subclusters. Contributed by Inigo Goiri.

  • HDFS-13380: RBF: mv/rm fail after the directory exceeded the quota limit. Contributed by Yiqun Lin.

  • HDFS-13388: RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun.

  • HDFS-13402: RBF: Fix java doc for StateStoreFileSystemImpl. Contributed by Yiran Wu.

  • Added CHANGES/RELEASES/Jdiff for 3.1.0 release

  • HDFS-13353: RBF: TestRouterWebHDFSContractCreate failed. Contributed by Takanobu Asanuma.

  • HDFS-13350: Negative legacy block ID will confuse Erasure Coding to be considered as striped block. (Contributed by Lei (Eddy) Xu).

  • HDFS-13364: RBF: Support NamenodeProtocol in the Router. Contributed by Inigo Goiri.

  • HADOOP-15355: TestCommonConfigurationFields is broken by HADOOP-15312. Contributed by LiXin Ge.

  • HADOOP-15317: Improve NetworkTopology chooseRandom's loop.

  • HDFS-13289: RBF: TestConnectionManager#testCleanup() test case need correction. Contributed by Dibyendu Karmakar.

  • HADOOP-15253: Should update maxQueueSize when refresh call queue. Contributed by Tao Jie.

  • HADOOP-14999: AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu.

  • HADOOP-12862: LDAP Group Mapping over SSL can not specify trust store. Contributed by Wei-Chiu Chuang and Konstantin Shvachko.

  • HDFS-13087: Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge.

  • HADOOP-15342: Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria.