7. Known Issues

Moved application from HDP 2.2 to HDP 2.3 and now ACLs don't appear to be functioning the same

Workaround: Set hbase.security.access.early_out=false, as in the following example:

<property>
    <name>hbase.security.access.early_out</name>
    <value>false</value>
  </property>

The HBase bulk load process is a MapReduce job that typically runs under the user ID who owns the source data. HBase data files created as a result of the job are then bulk-loaded into HBase RegionServers. During this process, HBase RegionServers move the bulk-loaded files from the user's directory, and moves (renames) the files under the HBase root.dir (/apps/hbase/data). When HDFS data encryption is used, HDFS cannot rename across encryption zones with different keys.

Workaround: Run the MapReduce job as the hbase user, and specify an output directory in the same encryption zone as the HBase root directory.

When rolling upgrade is performed for HDFS, sometimes the HBase Master might run out of datanodes on which to keep its write-pipeline active. When this occurs, the HBase Master Aborts after a few attempts to keep the pipeline going. To avoid this situation:

Workaround:

Note: There is a window of time during the rolling upgrade of HDFS when the HBase Master might be working with just one node and if that node fails, the WAL data might be lost. In practice, this is an extremely rare situation.

Alternatively, the HBase Master can be turned off during the rolling upgrade of HDFS to avoid the above procedure. If this strategy is taken, client DDL operations and RegionServer failures cannot be handled during this time.

A final alternative if the HBase Master fails during rolling upgrade of HDFS, a manual start can be performed.

HDP 2.3 HBase install needs MapReduce class path modified for HBase functions to work

Cluster that have Phoenix enabled placed the following config in hbase-site.xml:

Property: hbase.rpc.controllerfactory.class
Value:org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory

This property points to a class found only in phoenix-server jar. To resolve this class at run time for the above listed Mapreduce Jobs, it needs to be part of the MapReduce classpath.

Workaround: Update mapreduce.application.classpath property in mapred-site.xml file to point to /usr/hdp/current/phoenix-client/phoenix-server.jar file.

Hive Hybrid Grace MapJoin can cause OutOfMemory Issues

Hive Hybrid Grace Mapjoin is a new feature in HDP 2.3 (Hive 1.2). Mapjoin joins two tables, holding the smaller one in memory. Grace Hybrid Mapjoin spills parts of the small table to disk when the Map Join does not fit in memory at runtime. Right now there is a bug in the code that can cause this implementation to use too much memory, causing an OutOfMemory error. This applies to the Tez execution engine only.

Workaround: Turn off hybrid grace map join by setting this property in hive-site.xml:

Users should not use datanucleus.identifierFactory = datanucleus2 in hive config.

Setting datanucleus.identifierFactory to datanucleus2 can potentially lead to data corruption if directSql is enabled. Avoid using this setting if you are setting up a new metastore. If you are migrating an old metastore with this configuration parameter already set, contact Support for a few steps to address the issue.

When HDFS is encrypted (data at rest encryption is enabled) and the Hadoop Trash feature is enabled, DROP TABLE and DROP PARTITION have unexpected behavior.

(The Hadoop Trash feature is enabled by setting fs.trash.interval > 0 in core-site.xml.)

When Trash is enabled, the data file for the table should be "moved" to the Trash bin, but if the table is inside an Encryption Zone, this "move" operation is not allowed.

Workaround: Here are two ways to work around this issue:

1. Use PURGE, as in DROP TABLE ... PURGE. This skips the Trash bin even if Trash is enabled.

2. set fs.trash.interval = 0. Caution: this configuration change must be done in core-site.xml. Setting it in hive-site.xml may lead to data corruption if a table with the same name is created later.

With RHEL7, the cpu and cpuacct controllers are managed together by default. The default directory is /sys/fs/cgroup/cpu,cpuacct. The presence of the comma leads to failures when initializing the NodeManager (when using the LinuxContainerExecutor).

Workaround: Create your own directory(such as /sys/fs/cgroup/hadoop/cpu) and set yarn.nodemanager.linux-container-executor.cgroups.mount to true. This will allow the NodeManager to mount the cpu controller, and YARN will be able to enforce CPU limits for you.

If you wish to mount the cgroups yourself (or provide a mount point), please set yarn.nodemanager.linux-container-executor.cgroups.mount to false and ensure that the hierarchy specified in yarn.nodemanager.linux-container-executor.cgroups.hierarchy exists in the mount location. Make sure there are no commas in your pathnames.

Limitations while using timestamp.formats serde parameter.

Two issues involving the timestamp.formats SerDe parameter:

Spark ATS is missing Kill event

If a running Spark application is killed in the YARN ATS (yarn application -kill <appid>), the log will not list the outcome of the kill operation.

When accessing an HDFS file from pyspark, the HADOOP_CONF_DIR environment must be set. For example:

export HADOOP_CONF_DIR=/etc/hadoop/conf
[hrt_qa@ip-172-31-42-188 spark]$ pyspark
[hrt_qa@ip-172-31-42-188 spark]$ >>>lines = sc.textFile("hdfs://ip-172-31-42-188.ec2.internal:8020/tmp/PySparkTest/file-01")
.......

If HADOOP_CONF_DIR is not set properly, you might receive the following error:

Py4JJavaError: An error occurred while calling z:org.apache.spark.api. python.PythonRDD.collectAndServe.
org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  
Available:[TOKEN, KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

Tez UI View/Download link fails if URL does not match cookie.

Workaround: Tez UI View/Download link will work if a browser accesses a URL that matches the cookie.

Example: MapReduce JHS cookie is set with an external IP address. If a user clicks on the link from their internal cluster, the URL will differ and the request will fail with a dr.who error.

Users must manually configure ZooKeeper security with ResourceManager High Availability.

Right now, the default value of yarn.resourcemanager.zk-acl is world:any:rwcda. That means anyone can read/write/create/delete/setPermission for the znode which is not secure and not acceptable.

To make it more secure, we can rely on Kerberos to do the authentication for us. We could configure sasl authentication and only Kerberos authenticated user can access to zkrmstatestore.

ZooKeeper Configuration

Note: This step of securing ZooKeeper is to be done once for the HDP cluster. If this has been done to secure HBase, for example, then you do not need to repeat these ZooKeeper steps if Apache YARN ResourceManager High Availability is to use the same ZooKeeper.

Apache YARN Configuration

The following applies to HDP 2.2 and HDP 2.3.

Note: All nodes which launched the ResourceManager (active / standby) should make these changes.

HDFS Configuration

Note: This applies to HDP 2.1, 2.2, and 2.3.

HDP 2.3: Cannot set non HDFS FS as default. This prevents S3, WASB, and GCC from working.

HDP cannot be configured to use an external file system as the default file system - such as Azure WASB, Amazon S3, Google Cloud Storage. The default file system is configured in core-site.xml using the fs.defaultFS property. Only HDFS can be configured as the default file system.

These external file systems can be configured for access as an optional file system, just not as the default file system.

Upgrade to block ID-based DN storage layout delays DN registration.

When upgrading from a pre-HDP-2.2 release, a DataNode with a lot of disks, or with blocks that have random block IDs, can take a long time (potentially hours). The DataNode will not register to the NameNode until it finishes upgrading the storage directory.