Fixed issues in 7.1.9 CHF 6

Know more about the cumulative hotfix 6 for 7.1.9. This cumulative hotfix was released on April 26, 2024.

Following is the list of fixes that were shipped for CDP Private Cloud Base version 7.1.9-1.cdh7.1.9.p9.52289703

COMPX-16221: CLONE [7.1.9 | 7.2.19]- Queue Manager - Upgrade protobuf-java to 3.16.3/3.19.6/3.20.3/3.21.7 due to CVE-2022-3171: Upgraded protobuf-java to 3.16.3/3.19.6/3.20.3/3.21.7 due to CVE-2022-3171.
CDPD-68597: Need to include HDDS-10430 in CHF6: A race condition resulted in failure of serialization of Pipeline information at Ozone Manager when using topology-aware read. This issue is now fixed.
CDPD-68541: SnakeYaml upgrade due to CVE in schema_registry_ranger_plugin: Upgraded the SnakeYaml version referenced in the schema_registry_ranger_plugin component, due CVE-2022-1471.
CDPD-68466, CDPD-68129: Change snapshot purge request from batch to single snapshot at a time: In Ozone Snapshot, the previous snapshot was retained for each SnapshotInfo entry in the SnapshotInfoTable. On the snapshot create code path, the latest snapshot was added as the previous snapshot. During snapshot purging, there was race condition on the snapshot purge path, where two consecutive snapshots pointed to the same previous snapshot causing snapshot chain corruption. This issue is now fixed.
CDPD-68339: [AUTOSYNC] Ozone Recon - Filter EMPTY MISSING Containers in UnHealthy State Containers API: Recon UnHealthy State Containers API was not filtering the empty containers and the containers were reported as MISSING. This issue is now fixed.
CDPD-67945: [AUTOSYNC] Link rocksdb lib to Ozone rocksdb tools lib relative path instead of absolute path: The native library load failed because the rocksdb library was linked to the absolute path. This issue is now fixed by linking the rocksdb library to the Ozone rocksdb tools library relative path instead of theabsolute path.
CDPD-67944: Make Rocksdb tools native lib compatible with all chipsets within the arch: While building rocksdb tools library, the makefile of rocksdb added the native optmization gcc flags when compiling the rocksdb code.This made the build code incompatible with other chipsets.This issue is now fixed and the rocksdb tools native library is now compatible with all chipsets within the arch.
CDPD-67864: Ranger - Upgrade Spring Security to 5.7.12/5.8.11/6.1.8/6.2.3 due to CVE-2024-22257: Upgraded Spring Security to 5.7.12 due to CVE-2024-22257.
CDPD-67828: Hadoop - Upgrade Nimbus-JOSE-JWT to 9.37.3 due to CVE-2023-52428: Upgraded Nimbus-JOSE-JWT to 9.37.3 due to CVE-2023-52428
CDPD-67821: Zookeeper - Information disclosure in persistent watcher handling(CVE-2024-23944): There was information disclosure in persistent watchers handling in Apache ZooKeeper due to CVE-2024-23944. This issue is now fixed.
CDPD-67820: Backport HIVE-26955 to CDH-7.1.9.x: Select query failed when the decimal column data type was changed to string, char or varchar in Parquet format. Thi sissue is now fixed.
CDPD-67819: Backport HIVE-26320 to CDH-7.1.9.x: A query involving case statement with two or more conditions lead to incorrect result for tables with parquet format. This issue is now fixed.
CDPD-67750: Ranger - Upgrade telemetry to 1.36.0: Upgraded Telemetry version to 1.36.0.
CDPD-67749: Ranger - Upgrade protobuf-java to 3.21.7 due to CVE-2022-3171: Upgraded protobuf-java to 3.21.7 due to CVE-2022-3171.
CDPD-67746: Ranger - Upgrade Nimbus-JOSE-JWT to 9.37.3 due to CVE-2023-52428: Upgrade Nimbus-JOSE-JWT to 9.31
CDPD-67744: Ranger - Exclude Apache Derby from ranger-rms module due to CVE-2022-46337: Exclude apache derby from ranger-rms module
CDPD-67440, CDPD-67727: [AUTOSYNC] Snapshot chain corruption: In Ozone Snapshot, we keep snapshot's previous snapshot for each SnapshotInfo entry in the SnapshotInfoTable. On the snapshot create code path, we use the latest snapshot added as the previous snapshot. And on snapshot purging, we update the next snapshot's previous to the snapshotToPurge’s previous to maintain the chain. (Similar to deleting a node from a LinkedList/DoublyLinkedList). There is a race condition on the snapshot purge path because of which sometimes, two consecutive snapshots point to the same previous snapshot and cause snapshot chain corruption.
CDPD-67593: [Spark Compaction] Count function in impala giving incorrect results for V2 Iceberg tables: The count function in Impala gave incorrect results for V2 Iceberg tables. This issue is now fixed by turning off the optimized count(*) feature.
CDPD-67070: Backport HBASE-27230 to 7.1.9 CHF5: As per the design, write-ahead-log synchronisation (WAL Sync) should always succeed or abort. There is no failure state. Hence, if the WAL sync runs into timeout exception, the region-server should abort. The timeout for the sync is generally very large (5 minutes). Hence, if the operation does not complete by that time and runs into timeout, then, the region server should abort.
CDPD-66926: OM startup failure : OMLeaderNotReadyException leader is not getting ready: Ozone health was in a bad state and all the requests failed with errors as OM leaders did process any request. This issue is now fixed.
CDPD-66893: Hadoop - Upgrade moment.js to 2.29.4 due to CVE-2022-24785, CVE-2022-31129: Upgraded moment.js to version 2.29.4 due to CVE-2022-24785 and CVE-2022-31129.
CDPD-66890: TestMapReduceIndexerTool localfs test error: On FIPS environments the localfs functionality of both MRIT and HBase MRIT was not working because the underlying JSCH library did not support SHA2. This issue is now fixed by replacing JSCH with native implementation.
CDPD-66743: [AUTOSYNC] Recon - Handle the pre-existing missing empty containers in clusters: There were EMPTY MISSING containers (zero keys), and those were reported as MISSING by Recon. This fixes and handles the pre-existing empty missing containers.
CDPD-66668: Kafka_connect_ext - Upgrade commons-compress to 1.26.0 due to CVE-2023-42503, CVE-2024-25710 and CVE-2024-26308: Upgrade Commons-Compress to version 1.26.0 due to CVE-2023-42503, CVE-2024-25710 and CVE-2024-26308.
CDPD-65188: [AUTOSYNC] Remove applyTransactionMap and ratisTransactionMap from OzoneManagerStateMachine: The applyTransactionMap and the ratisTransactionMap in OzoneManagerStateMachine is now replaced by two TermIndex values:

lastApplied (implemented by BaseStateMachine) : keeps track of the last flushed TermIndex from OzoneManagerDoubleBuffer

lastNotified(a new field in OzoneManagerStateMachine): keeps track the last term, index passed from notifyTermIndexUpdated(..)
CDPD-65178: [AUTOSYNC] OzoneManagerRatisServer.getServer() should return Division
CDPD-65158: [AUTOSYNC] NumKeys metric not decremented on FSO directory delete: The NumKeys metric is now decremented for all the keys within a directory when the directory is deleted. This is done in the background as the directory is cleaned up after the initial deletion.
CDPD-64334: Ranger - Upgrade Bouncy Castle to 1.78 due to CVE-2023-33202, CVE-2023-33201, CVE-2024-29857, CVE-2024-30171 and CVE-2024-30172: Upgraded Bouncy Castle library version to 1.77 due to CVE-2023-33202, CVE-2023-33201, CVE-2024-29857, CVE-2024-30171 and CVE-2024-30172.
CDPD-63605: Verbose log messages regarding pipeline creation failure: New pipelines were not created because logs were flooded with messages regarding SCM and when the cluster had reached maximum pipelines per datanode configuration limit. This issue is now fixed by limiting such verbose messages.
CDPD-50862: Extend usage of fire_listener_event API to HS2/Spark to generate events on DML queries: The data written from Spark now generates an Insert event in the HMS notification log table. This is useful for external services (listeners) such as Impala to know the current condition of an external table.
CDPD-42610: Make RetryPolicy Configurable in Read Key path: Ozone client retries read operation three times when Datanodes are unavailable temporarily. The retry policy is now configurable using the client configurations ozone.client.read.max.retries(3 by default) and ozone.client.read.retry.interval(1 by default).

Common Vulnerabilities and Exposures (CVE) that is fixed in this CHF:

CVE-2023-44483