Release Notes
Also available as:
PDF

Known Issues

Hortonworks Bug IDApache JIRAApache ComponentSummary
BUG-50023 PHOENIX-3916Phoenix

Description of Problem: The hbck repair tool sometimes generates local indexes that are inconsistent with table data when overlapping regions are encountered.

Workaround: If you know the database schema, fix this issue by dropping and recreating all local indexes of the table after the hbck tool completes operation. Alternatively, rebuild the local indexes using the following ALTER query:

ALTER INDEX IF EXISTS index_name ON data_table_name REBUILD

BUG-60904 KNOX-823 Knox

Description of Problem: When Apache Knox uses Ambari as a proxy, the QuickLinks are not rewritten to go back through the gateway.

Workaround: If all access to Ambari is through Knox in the deployment the new Ambari quicklink profile may be used to hide and/or change URLs to go through Knox permanently. A future release will make these reflect the gateway appropriately.

BUG-65977 SPARK-14922 Spark

Description of Problem: Since Spark 2.0.0, `DROP PARTITION BY RANGE` is not supported grammatically. In other words, only '=' is supported while `<', '>', '<=', '>=' aren't.

Associated Error Message:

scala> sql("alter table t drop partition (b<1) ").show
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '<' expecting {')', ','}(line 1, pos 31)

== SQL ==
alter table t drop partition (b<1)
-------------------------------^^^ 

Workaround: To drop partition, use the exact match with '='. scala> sql("alter table t drop partition (b=0) ").show

BUG-68628SPARK-16605, SPARK-16628Spark

Description of Problem: Column names are not getting created for Spark DataFrame

Workaround: There are two workaround options for this issue:

  • Use Spark to create the tables instead of Hive.

  • Set: sqlContext.setConf("spark.sql.hive.convertMetastoreOrc", "false")

BUG-68632 SPARK-18355 Spark

Description of Problem: The property spark.sql.hive.convertMetastoreOrc is set to "true" by default. This may impact performance.

Workaround: You can set this property as "false" to gain some performance improvement. However, you cannot use this property for ORC tables with new columns made by `ALTER TABLE`. The default is false for safety.

BUG-70956N/AZeppelin

Description of Problem: A Hive query submitted to the %jdbc interpreter returns a proxy validation error.

Associated error messages:

  • HiveSQLException: Failed to validate proxy privilege of zeppelin for <user>

  • The hiveserver2.log file lists a permission denied exception: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=<user>, access=WRITE, inode="/user/<user>":hdfs:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319 … org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)

Workaround:

  1. Create the user account on all worker nodes. For example, if the account is user3, issue the following command: $ adduser user3

  2. Restart the %jdbc interpreter.

BUG-70956N/AZeppelin

Description of Problem: When used with Hive, the %jdbc interpreter might require Hadoop common jar files that need to be added manually.

Workaround:

  1. On the Interpreters page, add a new repository.

  2. Set the repository ID to “hortonworks”.

  3. Set the URL to “http://nexus-private.hortonworks.com/nexus/content/groups/public/”.

  4. Identify the version of HDP running on the cluster. You can find this in the Zeppelin UI by clicking on your user ID and choosing "About Zeppelin." The number after the Zeppelin version is the HDP version number.

  5. In the Dependencies section on the Interpreters page, remove existing jar files and add the following three jar files, with the correct HDP version for your cluster. The following example uses version 2.6.0.0-484:

    • org.apache.hive:hive-jdbc::standalone:1.2.1000.2.6.0.0-484 Xerces:xerces:2.4.0

    • org.apache.hadoop:hadoop-common:2.7.3.2.6.0.0-484

BUG-72905FLUME-2620Flume

Description of Problem: When Flume receives an event with null value in its header then File Channel will not accept it however the source will keep retrying to send it. Therefore that channel will stop working.

Error messages in log depending on the type of source:

'Error appending event to channel.' / 'Unable to put batch on required channel'
...
Caused by: java.lang.NullPointerException
at org.apache.flume.channel.file.proto.ProtosFactory$FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)

Workarounds:

  • Make sure that event headers have no null values or

  • Use custom interceptor to remove null values or replace them with empty string or

  • Use Memory Channel

BUG-74152 PHOENIX-3688 Phoenix

Description of Problem: Rebuild(ALTER INDEX IDX ON TABLE REBUILD) of indexes created on the table having row_timestamp column will result in no data visible to the User for that Index.

Workaround: Drop the index and recreate the same index. There will not be any extra overhead of recreating index when compared with rebuild Index.

BUG-75179 ZEPPELIN-2170 Zeppelin

Description of Problem: Zeppelin does not show all WARN messages thrown by spark-shell. At the Zeppelin notebook level, we cannot change the log level that comes as an output.

Associated Error Message: There is no error message. This error is only related to increasing or decreasing logging in notebook output.

Workaround: Currently, there is no workaround for this.

BUG-76996N/ASpark 2 (Livy)

Description of Problem: When upgrading from HDP-2.5.x to HDP-2.6.0 and using Spark2, the Livy interpreter is configured with a scope of 'global', and should be changed to 'scoped'.

Workaround: After upgrading from HDP 2.5 to HDP 2.6, set the interpreter mode for %livy (Spark 2) to "scoped" using the pulldown menu in the %livy section of the Interpreters page.

BUG-78035 ZEPPELIN-1263 Spark

Component Affected: Zeppelin

Description of Problem: spark.driver.memory will not take effect, the driver memory is always 1G.

Workaround: To change the driver memory, specify it in the SPARK_DRIVER_MEMORY property on the interpreter setting page for your spark interpreter.

BUG-80256N/AZeppelin

Description of Problem: Upgrading to a newer version of HDP version will not persist any Zeppelin notebooks from the previous version.

Workaround: Copy the Zeppelin notebooks from you previous HDP installation into /usr/hdp/current/zeppelin-server/notebook and restart zeppelin service. Zeppelin notebooks are located in /usr/hdp/<PREV_HDP_VERSION>/zeppelin/notebook

BUG-80656 Zeppelin

Description of Problem: Zeppelin fails to start during the upgrade process from HDP 2.5 to HDP 2.6. The error starts with

Exception in thread "main" org.apache.shiro.config.ConfigurationException: Unable to instantiate class org.apache.zeppelin.server.ActiveDirectoryGroupRealm for object named 'activeDirectoryRealm'. Please ensure you've specified the fully qualified class name correctly.

Workaround: This error is due to a change in configuration class for Active Directory.

In HDP 2.5:

org.apache.zeppelin.server.ActiveDirectoryGroupRealm

In HDP 2.6:

org.apache.zeppelin.realm.ActiveDirectoryGroupRealm

To resolve this issue, choose one of the following two alternatives:

  • Proceed with the upgrade, and change the configuration in the shiro.ini file after the upgrade is complete (when Ambari allows configuration change).

  • At time of failure, change the class name in /usr/hdp/current/zeppelin-server/conf/shiro.ini, and then start Zeppelin manually.

BUG-80808OOZIE-2777Oozie

Description of Problem: If config-default.xml is larger than 64k, it will produce an error.

Error Message: java.lang.RuntimeException: java.io.UTFDataFormatException: encoded string too long: 70771 bytes

Workaround: Currently, there is no known workaround.

BUG-80901N/AZeppelin

Component Affected: Zeppelin/Livy

Description of Problem: This occurs when running applications through Zeppelin/Livy that requires 3rd-party libraries. These libraries are not installed on all nodes in the cluster but they are installed on their edge nodes. Running in yarn-client mode, this all works as the job is submitted and runs on the edge node where the libraries are installed. In yarn-cluster mode,this fails because the libraries are missing.

Workaround: Set the location inspark.jars in spark-defaults.conf. For Livy, you will need to set livy.spark.jars (the HDFS location) in livy interpreters conf. Both are globally applicable. The jars need to be present on the livy machine in both cases. Updating livy conf is preferable since it affects only the Zeppelin users.

BUG-81637N/ASpark

Description of Problem: Executing concurrent queries over Spark via Spark1-llap package spawns multiple threads. This may cause multiple queries to fail. However, this will not break the spark thrift server. Spark 1.6 is built using Scala 2.10, which is where this issue manifests (i.e. " synchronize reflection code as scala 2.10 reflection is not threadsafe self"). This issue was subsequently fixed in Scala 2.11 based on this patch https://issues.scala-lang.org/browse/SI-6240.

Associated error messages:

  • scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving class LlapContext
  • SparkExecuteStatementOperation: Error running hive query:
    org.apache.hive.service.cli.HiveSQLException: scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving class LlapContext

Workaround: Isolate the broken queries and re-run them one by one. This will limit the query to one spawned thread.

BUG-82251N/ARanger

Description of Problem: When Enable Group Search First is true and Enable User Search is true, users are not synced to Ranger where CN of the user is different from sAMAccountName.

Workaround: Currently, there is no known workaround.

BUG-83171HIVE-16820Hive

Description of Problem: Canceling a Hive query from some third-party tools may not work (Squirel, Alteryx).

Workaround: Currently, there is no known workaround.

BUG-84163N/AZeppelin

Description of Problem: Missing HTTP Headers and Cookie flags in Zeppelin

Workaround: If using or upgrading to HDP 2.6.2, you will need to add the following properties in the Zeppelin configuration in shiro.ini:

cookie = org.apache.shiro.web.servlet.SimpleCookie
cookie.name = JSESSIONID
cookie.secure = true
cookie.httpOnly = true
sessionManager.sessionIdCookie = $cookie
BUG-85309HDFS-12130HDFS

Description of Problem: When getting content summary is called (such as when du command is called) on a directory with a large number of subdirectories and files, other NameNode operations (for example, writes to the directory) may be slow. This is because get content summary will recursively check the permissions of all the sub-directories while holding an expensive lock.

Workaround: Currently, there is no known workaround.

BUG-85436KNOX-986Knox

Component Affected: Knox metrics collection

Description of Problem: Knox metrics collection was previously enabled by default. After extensive testing, test findings indicate that the level of detail provided at the URL level can adversely lead to an increase in memory footprint.

Workaround: Knox metrics collection is now disabled by default and you should only enable it with an understanding of the number of unique URLs that Knox metrics collection will access. This includes but is not limited to the number of unique files Knox can access via WebHDFS.

BUG-85734ATLAS-2028Atlas

Description of Problem: Basic-search API ignores unknown attributes specified in entity or tag filters; and returns results after applying other valid filters.

Workaround: Currently, there is no known workaround. The API implementation will be updated in the next release to treat presence of unknown attributes as an error.

BUG-86394N/AZeppelin

Description of Problem: Spark2, jdbc(spark), jdbc(spark2) and jdbc(hive_interactive) interpreters are not available on upgraded clusters from Ambari 2.4.2.0 and HDP 2.5.5.0.

Workaround: Delete interpreter.json (from /etc/zeppelin/conf) and restart Zeppelin Server from Ambari.

BUG-86418N/AZeppelin

Description of Problem: After upgrading from Ambari 2.4.2 to Ambari 2.5.2 and subsequent HDP stack upgrade from 2.5 to 2.6, jdbc(hive) interpreter fails to work correctly in Zeppelin.

Error Message: You might see one of the following errors in the Zeppelin stacktrace after running jdbc(hive):

  • Error in doAs

  • Failed to validate proxy privilege of zeppelin

Workaround:

  1. Make sure hadoop.proxyuser.zeppelin.groups=* and hadoop.proxyuser.zeppelin.hosts=* are set in HDFS core-site.xml. If not, then configure these properties and restart all stale services. (AMBARI-21772 is currently tracking this item).

  2. Make user hive.url is configured correctly in Zeppelin's JDBC hive interpreter.

    [Note]Note

    The URL configured might be wrong, especially on secured and/or wire-encrypted clusters, due to a known issue that we will address in a future release.

  3. Restart HS2.

BUG-86549HBASE-18646HBase

Description of Problem: HBase backup and restore related operations report failure after a 60 second timeout.

Error Message: You may see something similar to the following:

2017-05-31 20:28:24,527 ERROR [(zk1-e4a1b4.hdinsight.net,16000,1496260101005)-proc-coordinator-pool2-thread-1] procedure.Procedure: Procedure 'rolllog' execution failed!
org.apache.hadoop.hbase.errorhandling.TimeoutException via timer-java.util.Timer@7d071b9:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed!
Source:Timeout caused Foreign Exception Start:1496262444394, End:1496262504394, diff:60000, max:60000 ms
        at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
        at org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:397)
        at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:216)
        at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:68)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed!
Source:Timeout caused Foreign Exception Start:1496262444394, End:1496262504394, diff:60000, max:60000 ms
        at org.apache.hadoop.hbase.errorhandling.TimeoutExceptionInjector$1.run(TimeoutExceptionInjector.java:66)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

Workaround: Under normal circumstances, 60 seconds is a sufficient amount of time for this operation to succeed but results are not consistent. There is no workaround at the present time. HBASE-18646 tracks an improvement that will allow you to adjust the 60 second timeout value.

BUG-86794N/AAtlas

Description of Problem: When using basic search, the atlas logs are filled with INFO messages from the Titan layer. The log statements show the conversion of the entity/tag attribute names to the ones that are mapped under solr/elasticsearch (indexing backend), basically it shows the query transformation.

Workaround: To avoid excessive INFO level logs from index-search, you suppress the logs by setting the following property in Ambari for atlas configuration:

com.thinkaurelius.titan=WARN

BUG-86970N/ARanger

Description of Problem: A newly introduced REPLADMIN permission appears in Hive permission list. However, this permission should not be included in policies. If added to policies, the service/public/api/policy API will return a 404 response.

Workaround: Remove REPLADMIN permission from affected policies.

BUG-87128N/AMahout

Since Mahout is deprecated in favor of Spark ML, and every code change carries the risk of creating additional incompatibilities, we will document these difficulties rather than change these established behaviors in Mahout. These issues affect only Mahout.

  • BUG-80945: Potential DoS vulnerability due to ReadLine usage in 9 files

  • BUG-80944: Potential JavaScript vulnerability in RecommenderServlet.java

  • BUG-80942: Potential Path Manipulation vulnerability due to path input usage in 8 files

  • BUG-80941: Potential DoS vulnerability due to StringBuilder usage in ConfusionMatrix.java

  • BUG-80940: Potential DoS vulnerability due to ReadLine usage in 8 additional files

  • BUG-80938: Potential invalid use of Object.equals in FastMap.java and FastByIDMap.java

BUG-90362N/AZeppelin

Description of Problem: A new property has been added in HDP-2.6.2 to ensure that the Zeppelin JDBC Interpreter works correctly. However, this property is not added automatically when upgrading from lower versions to HDP-2.6.2. This can result in an Error in doAs when running the Zeppelin JDBC Interpreter for Hive in HDP-2.6.2.

Workaround: Perform the following steps to add the property manually:

  1. In the Zeppelin UI, navigate to the %jdbc section of the Interpreter page.

  2. Click edit, then add a hive.proxy.user.property property and set its value to hive.server2.proxy.user.

  3. Click Save, then click restart to restart the JDBC interpreter.

N/A N/AN/A

Description of problem: Open JDK 8u242 is not supported as it causes Kerberos failure.

Workaround: Use a different version of Open JDK.

Technical Service BulletinApache JIRAApache ComponentSummary
TSB-327HDFS-5698HDFS

CVE-2018-11768: HDFS FSImage Corruption (potential DoS, file/dir takeover)

In very large clusters, the in-memory format to store the user, group, acl, and extended attributes may exceed the size of the on disk format, causing corruption of fsImage.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-327:CVE-2018-11768: HDFS FSImage Corruption (potential DoS, file/dir takeover)

TSB-405N/AN/A

Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

Microsoft has introduced changes in LDAP Signing and LDAP Channel Binding to increase the security for communications between LDAP clients and Active Directory domain controllers. These optional changes will have an impact on how 3rd party products integrate with Active Directory using the LDAP protocol.

Workaround

Disable LDAP Signing and LDAP Channel Binding features in Microsoft Active Directory if they are enabled

For more information on this issue, see the corresponding Knowledge article: TSB-2021 405: Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

TSB-406N/AHDFS

CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

WebHDFS clients might send SPNEGO authorization header to remote URL without proper verification. A maliciously crafted request can trigger services to send server credentials to a webhdfs path (ie: webhdfs://…) for capturing the service principal

For more information on this issue, see the corresponding Knowledge article: TSB-2021 406: CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

TSB-434HADOOP-17208, HADOOP-17304Hadoop

KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

For more information on this issue, see the corresponding Knowledge article: TSB 2020-434: KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

TSB-465N/AHBase

Corruption of HBase data stored with MOB feature

For more information on this issue, see the corresponding Knowledge article: TSB 2021-465: Corruption of HBase data stored with MOB feature on upgrade from CDH 5 and HDP 2

TSB-497N/ASolr

CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

The Apache Solr ReplicationHandler (normally registered at "/replication" under a Solr core) has a "masterUrl" (also "leaderUrl" alias) parameter. The “masterUrl” parameter is used to designate another ReplicationHandler on another Solr core to replicate index data into the local core. To help prevent the CVE-2021-27905 SSRF vulnerability, Solr should check these parameters against a similar configuration used for the "shards" parameter.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-497: CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

TSB-512N/AHBase

HBase MOB data loss

HBase tables with the MOB feature enabled may encounter problems which result in data loss.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-512: HBase MOB data loss