Hortonworks Data Platform
Also available as:
PDF

Fixed Issues

Fixed issues represents selected issues that were previously logged via Hortonworks Support, but are now addressed in the current release. These issues may have been reported in previous versions within the Known Issues section; meaning they were reported by customers or identified by Hortonworks Quality Engineering team.

Incorrect Results

Hortonworks Bug ID Apache JIRA Summary
BUG-108283 RANGER-2169 Create unique index on service and name column of x_policy table
BUG-108346 KNOX-1398 WebHDFS and WebHBase APIs listed incorrectly as services that need whitelist enforcement
BUG-108348 KNOX-1392, KNOX-1393, KNOX-1395 Remove use of InetAddress in Default Whitelist Creation
BUG-108351 KNOX-1386 Fix NiFi service discovery
BUG-108352 KNOX-1385 Whitelist test inconsistencies
BUG-108802 KNOX-1412 Knox fails while executing a `curl` command using Content-Type: text/xml" "Content-Encoding: gzip"
BUG-108882 RANGER-2187 External Group search fails on Ranger UI when installed with postgres
BUG-109034 RANGER-2114 Backport RANGER-2114 on Atlantic-M05
BUG-109385 HIVE-20406 Nested Coalesce giving incorrect results
BUG-109887 HIVE-20187 Backport HIVE-20187
BUG-95462 HDFS-13205, RANGER-2183 Incorrect path is passed to checkPermission during authorization of file under a snapshot (specifically under a subdir) after original subdir is deleted

Other

Hortonworks Bug ID Apache JIRA Summary
BUG-102276 HIVE-19739 Use the checkpoint on table / partition to create relevant repl load tasks.
BUG-102279 HIVE-19739 Repl load should not fail if Target Db already exists.
BUG-103052 HIVE-19564 Vectorization: Backport HIVE-19564 to Atlantic-M05 --> Fix NULL / Wrong Results issues in Arithmetic Functions
BUG-105695 HIVE-20090 TPCDS query64 has regressed in Atlantic by 5x times.
BUG-105963 HIVE-19408 Show tables statement includes views and materialized views
BUG-106249 HIVE-20011 Move away from append mode in proto logging hook
BUG-106294 HIVE-20028 hive streaming fails with unable to abort invalid txns
BUG-106359 HIVE-4367 enhance TRUNCATE syntex to drop data of external table
BUG-107015 N/A TestReplicationScenariosAcidTables & TestReplicationOnHDFSEncryptedZones failing
BUG-107018 HIVE-19104 TestHS2ImpersonationWithRemoteMS.testImpersonation failing
BUG-107020 HIVE-20299 TestLlapSignerImpl.testSigning failing
BUG-107023 N/A TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1 failing with time out
BUG-107024 N/A TestCustomPartitionVertex.testGetBytePayload failing due to timeout
BUG-107025 HIVE-19800 TestUpgradeTool.testPostUpgrade failing
BUG-107026 N/A TestWarehouseExternalDir.testManagedPaths failing
BUG-107062 N/A TestSchemaToolForMetastore failing
BUG-107135 HIVE-20353 Follow redirects when hive connects to a passive druid overlord/coordinator
BUG-107180 N/A CLONE - UT error in storm: org.apache.storm.sql.hdfs.TestHdfsDataSourcesProvider
BUG-107264 N/A TestMiniDruidCliDriver failing
BUG-107300 KNOX-1386 Autodiscovery not using the correct NiFi properties
BUG-107398 HIVE-20151 External table: exception while storing stats
BUG-107516 HIVE-19800 PreUpgrade Tool: Support upgrade from >= 2.6.0
BUG-107547 HIVE-20135 Backport HIVE-20135: Fix incompatible change in TimestampColumnVector to default to UTC
BUG-107548 HIVE-20090 Backport HIVE-20090: Extend creation of semijoin reduction filters to be able to discover new opportunities
BUG-107553 N/A Vectorization: Backport HIVE-19992 to Atlantic-M05: Follow-on to HIVE-19951 --> add call to SchemaEvolution.isOnlyImplicitConversion to disable encoded LLAP I/O for ORC only when data type conversion is not implicit
BUG-107572 N/A Vectorization: Backport HIVE-20174 to Atlantic-M05: Fix NULL / Wrong Results issues in GROUP BY Aggregation Functions
BUG-107694 HIVE-20098 Backport HIVE-20098: Statistics: NPE when getting Date column partition statistics
BUG-107695 HIVE-20152 Backport HIVE-20152: reset db state, when repl dump fails, so rename table can be done
BUG-107713 N/A Vectorization: Backport HIVE-20197 to Atlantic-M05: Add DECIMAL_64 testing, add Date/Interval/Timestamp arithmetic, and add more GROUP BY Aggregation tests
BUG-107768 HIVE-19360 backport: CBO: Add an "optimizedSQL" to QueryPlan object
BUG-107866 ATLAS-2786 Atlas HA fails to use all the zookeeper in the environment
BUG-108021 HIVE-20383 provide the correct queue name to proto logger
BUG-108080 HIVE-20172 Backport HIVE-20172: StatsUpdater failed with GSS Exception while trying to connect to remote metastore
BUG-108109 N/A Disable flaky spark Tests
BUG-108201 HIVE-20207 Vectorization: Backport HIVE-20207 to Atlantic-M05: Fix NULL / Wrong Results issues in Filter / Compare
BUG-108216 HIVE-20192 HS2 with embedded metastore is leaking JDOPersistenceManager objects.
BUG-108263 N/A Fix TestJdbcWithDBTokenStore.testTokenAuth, TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth failures
BUG-108265 HIVE-20212 Backport HIVE-20212: Hiveserver2 in http mode emitting metric default.General.open_connections incorrectly
BUG-108287 HIVE-20383 Missing pre-hook and duplicate post hook events from hive proto events hook.
BUG-108345 HIVE-20245 Vectorization: Backport HIVE-20245 to Atlantic-M05: Fix NULL / Wrong Results issues in BETWEEN / IN
BUG-108357 HIVE-20082, HIVE-20204 Decimal truncation for trailing zeros - port to 3.0-maint
BUG-108424 HIVE-20033 PowerBI unable to connect to LLAP cluster with large Databases/Tables
BUG-108439 N/A TestJdbcWithMiniLlapRow is failing on ptest
BUG-108440 N/A TestHiveDruidQueryBasedInputFormat is failing on ptest
BUG-108444 HIVE-18201 Subquery joins result in cross product instead of broadcast
BUG-108511 HIVE-20263 Backport HIVE-20263: Typo in HiveReduceExpressionsWithStatsRule variable
BUG-108512 HIVE-20241 Backport HIVE-20241: Support partitioning spec in CTAS statements
BUG-108513 HIVE-14493 Backport HIVE-14493: Partitioning support for materialized views
BUG-108558 HIVE-20281 Backport HIVE-20281: SharedWorkOptimizer fails with 'operator cache contents and actual plan differ'
BUG-108566 N/A TPCDS query23 runs longer on hdp3.0 with varying runtimes.
BUG-108573 KNOX-1415 Upgrade Jetty to 9.2.25
BUG-108586 N/A Vectorization: Backport HIVE-20294 to Atlantic-M05: Fix NULL / Wrong Results issues in COALESCE / ELT
BUG-108642 N/A Update golden files for TezDriver
BUG-108643 HIVE-20302 Backport HIVE-20302: LLAP: non-vectorized execution in IO ignores virtual columns, including ROW__ID
BUG-108644 HIVE-20314 Backport HIVE-20314: Include partition pruning in materialized view rewriting
BUG-108645 HIVE-20301 Backport HIVE-20301: Enable vectorization for materialized view rewriting tests
BUG-108719 HIVE-20322 FlakyTest: TestMiniDruidCliDriver
BUG-108720 N/A Add missing Apache Hadoop 3.1.1 commits to Atlantic M05 (YARN/MR)
BUG-108816 HIVE-20383 Hive: ProtoLoggingHook NPE on queue name field
BUG-108894 HIVE-20336 Backport HIVE-20336: Masking and filtering policies for materialized views
BUG-108910 N/A Tez History API fails with ClassCastException
BUG-108917 N/A Backport Druid perf enhacements
BUG-108999 HIVE-20347 Backport HIVE-20347: hive.optimize.sort.dynamic.partition should work with partitioned CTAS and MV
BUG-109060 HIVE-20364 Update default for hive.map.aggr.hash.min.reduction
BUG-109168 N/A TestHiveCli timing out (need HIVE-20149 backport)
BUG-109173 N/A Backport CALCITE-2465: Enable use of materialized views for any planner
BUG-109245 HIVE-20379 Backport HIVE-20379: Rewriting with partitioned materialized views may reference wrong column
BUG-109247 HIVE-20382 Backport HIVE-20382: Materialized views: Introduce heuristic to favour incremental rebuild
BUG-109272 HIVE-20345 Backport HIVE-20345: Drop database may hang if the tables get deleted from a different call
BUG-109311 HIVE-20391 Backport HIVE-20391: HiveAggregateReduceFunctionsRule may infer wrong return type when decomposing aggregate function
BUG-109313 N/A Backport HIVE-17979 : Tez: Improve ReduceRecordSource passDownKey copying
BUG-109319 N/A Backport HIVE-20177: Vectorization: Reduce KeyWrapper allocation in GroupBy Streaming mode
BUG-109321 N/A Backport HIVE-20153: Count and Sum UDF consume more memory in Hive 2+
BUG-109703 RANGER-2167 Upgrade to Apache parent pom version 20
BUG-110109 N/A Remove unidecode from superset
BUG-110124 YARN-8726 YARN UI2 is not accessible
BUG-110428 HIVE-20450 Add replication test for LOAD command on ACID table.
BUG-89982 HIVE-20357 Forward port azure db scripts to atlantic
BUG-92690 HIVE-19267 Generate new event for ACID Write operations
BUG-92700 HIVE-19340 Disable timeout of transactions opened by replication task at target cluster
BUG-95678 HIVE-17683 Add explain locks <sql> command
BUG-98176 HIVE-20337 TPCDS query16 compilation fails on cn105 when CachedStore is used. ObjectStore works fine.
BUG-99461 HIVE-20337 Queries returning incorrect results using CachedStore. ObjectStore works fine.
BUG-110970 N/A

For long-running SparkSQL jobs on a Kerberized cluster, some JDBC clients may randomly fail with a "no token found in cache" error after the delegation token expiry period.

Performance

Hortonworks Bug ID Apache JIRA Summary
BUG-105965 N/A Adding contraints on TPCDS schema, changes query plan for query93 and degrades its perf by 4x times
BUG-107255 HIVE-20209 Metastore connection fail 1st attempt in repl dump
BUG-107475 RANGER-2162 Rangeradmin stuck with db connection issue.
BUG-107932 RANGER-2158 [Atlantic-M05] - RANGER-2158 Performance improvement for REST API calls
BUG-107990 HIVE-20300 VectorFileSinkArrowOperator
BUG-108108 HADOOP-15598 DataChecksum calculate checksum is contented on hashtable synchronization
BUG-108281 RANGER-2165 Address JPA Cache issue when policies Create, Update and Delete are done via REST API in Apache Ranger admin
BUG-108282 RANGER-2168 Add service admin user through service config
BUG-108401 RANGER-2173 RANGER-2173: Optimize Trie construction and Policy lookup
BUG-108422 HADOOP-15547 WASB: improve listStatus performance
BUG-108515 HIVE-20290 Lazy initialize ArrowColumnarBatchSerDe so it doesn't allocate buffers during GetSplits
BUG-108638 HIVE-20316 Skip external table file listing for create table event
BUG-108742 HIVE-18624 Hive parser is very slow in parsing nested expressions
BUG-109693 RANGER-2188 Support multiple threads to build Trie and on-lookup post-setup for Trie nodes
BUG-109694 RANGER-2191 Update ranger-tool with new options to control Trie
BUG-109702 RANGER-2186 Increment service-specific policy and tag versions after…
BUG-109720 RANGER-2196 Ensure that any explicit threads used by Ranger are marked as daemon threads

Potential Data Loss

Hortonworks Bug ID Apache JIRA Summary
BUG-104681 HIVE-19891 insert from select overwrites the partition location path

Security

Hortonworks Bug ID Apache JIRA Summary
BUG-106840 N/A CVE-2018-8009: Zip Slip Vulnerability - Apache Hadoop distributed cache archive vulnerability
BUG-107549 N/A druid bolt fails with NoSuchMethodError
BUG-107610 ATLAS-2788 Add policy permissions for tagsync user
BUG-107863 STORM-3052 Backport STORM-3052: Let blobs un archive (HDP 3.0-maint)
BUG-108451 KNOX-1416 Upgrade Spring Framework to 4.3.17
BUG-108574 N/A Upgrade Bouncy Castle Dependency from Pac4J
BUG-108749 ZEPPELIN-3686 Update org.eclipse.jetty version to 9.4.11.v20180605 to avoid CVE issues
BUG-108874 N/A Upgrade of Jackson Databind to 2.9.5 in Oozie
BUG-108997 N/A BlackDuck for HWC
BUG-109155 N/A Address Storm black-duck scan issues
BUG-109209 ATLAS-2821 Update old atlas-simple-authz-policy.json (m0) file with relationship permission attributes

Stability

Hortonworks Bug ID Apache JIRA Summary
BUG-101700 SPARK-23007 Add read schema suite for file-based data sources
BUG-102162 YARN-8595 [UI2] Container diagnostic missing from container page
BUG-102349 YARN-8593, YARN-8594 [UI2] Show the current logged in user in UI2
BUG-102389 YARN-8620 [UI2] YARN Services UI new submission failures are not debuggable
BUG-102493 YARN-8608 [UI2] No information available per application appAttempt about 'Total Outstanding Resource Requests'
BUG-103056 N/A [YARN-8301] Add documentation for yarn service upgrade
BUG-104220 RANGER-2150 Unit test coverage for org.apache.ranger.biz.XUserMgr and UserMgr
BUG-104727 N/A [YARN-8403] NM log should print Exception at ERROR log level
BUG-104844 YARN-8407 [YARN-8407] Container launch exception in AM log should be printed in ERROR level
BUG-105493 YARN-8429 [YARN-8429] Improve diagnostic message when artifact is not set properly
BUG-106203 YARN-8603 [UI2] Latest run application should be listed first in the RM UI
BUG-106504 N/A Update `spark2:atlantic` with Spark 2.3.2 RC5
BUG-106619 N/A Zengine UT failure: IllegalStateException: Queue configuration missing child queue names for root
BUG-107236 ATLAS-2793,, ATLAS-2794, RANGER-2166 Tagsync start fails with CNF error for org/codehaus/jackson/xc/JaxbAnnotationIntrospector when using Atlas-REST as source
BUG-107241 HIVE-19860 HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
BUG-107347 N/A Zeppelin, set right URL for jdbc for different transport modes
BUG-107413 SPARK-23679 Spark Web UI in master=yarn mode does not proxy/redirect with YARN HA (secured)
BUG-107477 YARN-8418 [YARN-8418] App local logs could leaked if log aggregation fails to initialize for the app
BUG-107576 HIVE-20069 Backport HIVE-20069: Fix reoptimization in case of DPP and Semijoin optimization
BUG-107632 N/A [YARN-8545] YARN native service should return container if launch failed
BUG-107700 HIVE-20203 Arrow SerDe leaks a DirectByteBuffer
BUG-107725 N/A Hive warehouse directory creation issue at blueprint cluster creation
BUG-107871 N/A [Service Discovery] Add Support for discovering beacon service
BUG-107922 HADOOP-15593 Backport HADOOP-15593 to Atlantic-M05
BUG-107945 RANGER-2160 'Email Address' search is not working in user listing page,userRoles filters also needs to be improved.
BUG-107955 YARN-8571 op/cli commands to a running service is failing in a secure cluster
BUG-107965 RANGER-2164 Ranger to add default altlas policy for rangertagsync user.
BUG-108112 N/A backport HIVE-19935 (race in WM resulting in a session dying)
BUG-108146 RANGER-2182 Handle upgrade scenario since atlas-service def is added with new resources for relationship
BUG-108209 N/A [ATSv2] [YARN-8591] NPE error while checking for entity acl in non-secure cluster
BUG-108210 YARN-8592 [UI2] Resource Manager Web UI V2 shows a blank page in windows OS And Chrome browser
BUG-108283 RANGER-2169 Create unique index on service and name column of x_policy table
BUG-108349 KNOX-1391 Fix IllegalStateException when accessing Admin UI
BUG-108350 KNOX-1387, KNOX-1390 Fix NPE in AclsAuthorizationFilter
BUG-108354 HIVE-20244, HIVE-20247, HIVE-20249 backport LLAP error handling changes
BUG-108407 RANGER-2171 RANGER-2171: Unit Test cases to cover policy operations from service admin user
BUG-108583 N/A Issue with tutorial notebooks
BUG-108624 N/A ATSv2 HBase Application does not appear when HBase app is stopped [System Service Mode]
BUG-108632 HIVE-20312 HWC: Assign Arrow ChildAllocator per-task to fail-fast on memory leak
BUG-108762 SPARK-24948 Spark History Server filters some applications on ADL
BUG-108833 N/A Upgrade bouncy castle version to 1.6.0 to fix CVE issue
BUG-108915 N/A Upgrade Jetty version to fix CVE issue in Spark
BUG-109034 RANGER-2114 Backport RANGER-2114 on Atlantic-M05
BUG-109183 N/A [YARN-8667] Container Relaunch fails with "find: File system loop detected;"
BUG-109184 N/A Add yarn.service.container-failure.retry.max for ats-hbase app.
BUG-109195 SPARK-25051 Correctness issue with outer joins for Spark 2.3
BUG-109198 N/A Provide work around handle BUG-108844 impact on YARN
BUG-109312 HIVE-20393 Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
BUG-109384 N/A [YARN-8679] Gridmix of HDP-3.0 is stuck
BUG-109581 ZEPPELIN-3741 Do not clear "Authorization" header if Z-server is running behind proxy
BUG-109697 RANGER-2180 Handle token replacement correctly when token is not defined in the request context
BUG-109720 RANGER-2196 Ensure that any explicit threads used by Ranger are marked as daemon threads
BUG-109777 RANGER-2197 Delegate Admin is not able to create policy

Supportability

Hortonworks Bug ID Apache JIRA Summary
BUG-107437 ZOOKEEPER-2662, ZOOKEEPER-3019 Port JMX metrics for Zookeeper I/O to Atlantic-Next
BUG-109698 RANGER-2152 Incorrect debugging information in RangerPluginClassLoader.java
BUG-109700 RANGER-2177 Handle validations for duplicate configuration item during service create/edit
BUG-109701 RANGER-2153 Supply the function of reverting policy history version.
BUG-30983 HIVE-14898 HiveServer2 error message from knox in logs - benign error confuses users

Upgrade

Hortonworks Bug ID Apache JIRA Summary
BUG-108146 RANGER-2182 Handle upgrade scenario since atlas-service def is added with new resources for relationship
BUG-109209 ATLAS-2821 Update old atlas-simple-authz-policy.json (m0) file with relationship permission attributes
BUG-109696 RANGER-2184 Update RangerAtlas authorization to authorize add/update/remove of relationships
BUG-109728 N/A phoenix-sqlline launch fails on upgraded cluster

Usability

Hortonworks Bug ID Apache JIRA Summary
BUG-103643 RANGER-2161 it will be good to mark exclude from deny and allow condition with some background color
BUG-104455 HIVE-19694 create materialized view should validate if view already exists
BUG-106469 HIVE-20044 [SPARK_LLAP] fixed char type output via spark + HDWC is inconsistent with that via HS2 beeline
BUG-106681 OOZIE-3328 Reenable hive action in Oozie
BUG-106688 HIVE-20341 Druid Needs Explicit CASTs from Timestamp to STRING when the output of timestamp function is used as String
BUG-106818 ATLAS-2785 import_hive.sh with "-t" option imports all metadata instead of just that table.
BUG-107236 ATLAS-2793,, ATLAS-2794, RANGER-2166 Tagsync start fails with CNF error for org/codehaus/jackson/xc/JaxbAnnotationIntrospector when using Atlas-REST as source
BUG-107241 HIVE-19860 HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
BUG-107742 NA Update some Ambari defaults based on 3.0-EA
BUG-107945 RANGER-2160 'Email Address' search is not working in user listing page,userRoles filters also needs to be improved.
BUG-107965 RANGER-2164 Ranger to add default altlas policy for rangertagsync user.
BUG-108114 N/A Backport HIVE-20240
BUG-108282 RANGER-2168 Add service admin user through service config
BUG-108312 N/A Oozie doesn't work out of the box with Spark2
BUG-108347 KNOX-1383, KNOX-1396, KNOX-1397 Admin UI Fixes for Topology List Refresh and URL Validation
BUG-108630 HIVE-20277 Vectorization: Backport HIVE-20277 to Atlantic-M05: Vectorization: Case expressions that return BOOLEAN are not supported for FILTER
BUG-108738 N/A HWC: Python API is missing "executeUpdate"
BUG-108882 RANGER-2187 External Group search fails on Ranger UI when installed with postgres
BUG-109034 RANGER-2114 Backport RANGER-2114 on Atlantic-M05
BUG-109290 N/A HWC: HiveWarehouseSession.table(<tableName>) doesn't accept unqualified table name
BUG-109309 N/A HWC: HiveWarehouseSession.dropTable API is broken
BUG-109698 RANGER-2152 Incorrect debugging information in RangerPluginClassLoader.java
BUG-109700 RANGER-2177 Handle validations for duplicate configuration item during service create/edit
BUG-109701 RANGER-2153 Supply the function of reverting policy history version.
BUG-109777 RANGER-2197 Delegate Admin is not able to create policy
BUG-109784 N/A Misleading documentation to enable SSL on Spark2 History Server
BUG-111169 N/A Oozie hive spark jobs failing because spark still use hcat type delegation tokens
BUG-95269 YARN-8318 [UI2] Few Missing information on Component Instance Information page
BUG-95462 HDFS-13205, RANGER-2183 Incorrect path is passed to checkPermission during authorization of file under a snapshot (specifically under a subdir) after original subdir is deleted
BUG-95678 HIVE-17683 Add explain locks <sql> command