Known Issues

Problem: Cluster status is not consistent between the DLM App and the DPS Platform pages.

Description: The status for NodeManager and for the DataNode on a cluster can take up to 10 minutes to update in DPS Platform.

Workaround: If the information does not update, wait a few minutes and refresh the page to view the status.

Problem: DLM: Hive cloud replication succeeds, but doesn't copy the data

Workaround: When upgrading HDP to 2.6.5 with DLM 1.1, you must add the following parameters in Ambari, in HDFS core-site:

fs.s3a.fast.upload = true

fs.s3a.fast.upload.buffer = disk

fs.s3a.multipart.size = 67108864

Problem: HDFS replication with both TDE and plain directory fails

Description: HDFS Replication fails when some files are encrypted and some are unencrypted. If the source directory is unencrypted, but contains both encrypted and unencrypted subfolders, then replication jobs fail with checksum mismatch error.

Workaround: Ensure that all folders in a source root directory have the same encryption setting (enabled/not enabled or same key)

Problem: HS2 failover requires knox restart if cookie use is enabled for HS2

Description: When HiveServer2 is accessed via Knox Gateway and HiveServer2 has cookie-based authentication enabled, a HiveServer2 restart requires that Knox also be restarted to get Knox-HiveServer2 interaction working again.

Workaround: Set hive.server2.thrift.http.cookie.auth.enabled=false in hive-site.xml in Ambari.

Problem: Hive Replication [ Onprem to Cloud ] not working when using Credential Providers

Description: Hive commands do not work on a cloud cluster when AWS S3 credentials are set up using hadoop.security.credential.provider.path configuration.

Workaround: Set credentials using fs.s3a.security.credential.provider.path configuration

Problem: beacon user is not added to kms policy by default

User:beacon not allowed to do 'DECRYPT_EEK' on '<keyname>’

Workaround: Encryption key management and policies are not handled by DLM. Set up the keys and key policies manually.

Problem: HDFS: Onprem to Onprem: Parent dataset identification failed.

Source dataset already in replication

Description: Creating a replication policy fails when a directory name is the prefix of another directory which is already part of the replication job. For example, if there is a replication policy created for data in /data/reporting, you cannot create another replication policy on /data/reporting, as “report” is a prefix of “reporting”.

This conflict only arises when the directories are at the same level. For example, replication for /data/2018/report would execute correctly.

Workaround: Rename the directories, if possible, to ensure directory names do not have a prefix match with other folders being replicated.