Policy download optimizations.

These optimizations improve performance of Ranger while handling a large number of authorization policies – like tens of thousands of policies. The optimizations include:

Block ID-based DN storage layout can be very slow for datanode on ext4.

HDFS-8791 introduces a new datanode layout format. This layout is identical to the previous block ID based layout except it has a smaller 32x32 sub-directory structure in each data storage. On startup, the datanode will automatically upgrade its storages to this new layout. Currently, datanode layout changes support rolling upgrades, but downgrading is not supported between datanode layout changes and a rollback would be required.

Mirror maker command is failing

When running the Kafka MirrorMaker tool:

IssueIf you specify --white-list=".*", MirrorMaker tries to fetch data from the system-level topic __consumer-offsets and produce the data to the target cluster. This can result in the following error:

Producer cannot send requests to __consumer-offsets

Workaround: add --whitelist="topic1,topic2" instead of .*

#1

Issue: Given one table name, creating, pre-splitting, deleting and then recreating with the same name and split point in quick succession can result in the HBase region locator cache returning invalid region locations. This has been observed in tests around the HBase replication feature.

Error Message: On the source HBase cluster: 2016-04-19 13:07:28,355 WARN [main-EventThread.replicationSource,TestPeer] regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of an error on the remote cluster: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException):

Workaround: When re-creating a table which has already been configured for replication, restart the sink (destination) cluster after the table was recreated to ensure that invalid region locations are not cached.

#2

Issue: It has been observed that, occasionally, when configuring HBase to use an HDFS encryption zone (aka TDE - transparent data encryption), some WAL updates were lost in the HBase replication process. This results in not all updates to be propagated from the source cluster to the sink cluster.

Error Message: No error message, but fewer entries in the sink cluster's table than the source cluster's table when TDE is enabled would be the sign that there was a problem.

Workaround: None.

Description of Problem: Ranger audit to HDFS fails with TGT errors

Workaround: Currently, there is no known workaround for this issue.

Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

Microsoft has introduced changes in LDAP Signing and LDAP Channel Binding to increase the security for communications between LDAP clients and Active Directory domain controllers. These optional changes will have an impact on how 3rd party products integrate with Active Directory using the LDAP protocol.

Workaround

Disable LDAP Signing and LDAP Channel Binding features in Microsoft Active Directory if they are enabled

For more information on this issue, see the corresponding Knowledge article: TSB-2021 405: Impact of LDAP Channel Binding and LDAP signing changes in Microsoft Active Directory

CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

WebHDFS clients might send SPNEGO authorization header to remote URL without proper verification. A maliciously crafted request can trigger services to send server credentials to a webhdfs path (ie: webhdfs://…) for capturing the service principal

For more information on this issue, see the corresponding Knowledge article: TSB-2021 406: CVE-2020-9492 Hadoop filesystem bindings (ie: webhdfs) allows credential stealing

KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

For more information on this issue, see the corresponding Knowledge article: TSB 2020-434: KMS Load Balancing Provider Fails to invalidate Cache on Key Delete

Corruption of HBase data stored with MOB feature

For more information on this issue, see the corresponding Knowledge article: TSB 2021-465: Corruption of HBase data stored with MOB feature on upgrade from CDH 5 and HDP 2

CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

The Apache Solr ReplicationHandler (normally registered at "/replication" under a Solr core) has a "masterUrl" (also "leaderUrl" alias) parameter. The “masterUrl” parameter is used to designate another ReplicationHandler on another Solr core to replicate index data into the local core. To help prevent the CVE-2021-27905 SSRF vulnerability, Solr should check these parameters against a similar configuration used for the "shards" parameter.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-497: CVE-2021-27905: Apache Solr SSRF vulnerability with the Replication handler

HBase MOB data loss

HBase tables with the MOB feature enabled may encounter problems which result in data loss.

For more information on this issue, see the corresponding Knowledge article: TSB 2021-512: HBase MOB data loss