What's New in HDF 3.4.0

Apache Kafka Updates

Kafka has been upgraded from 2.0.0 to 2.1.0.

	Warning
We strongly discourage downgrading from Kafka 2.1.0 to earlier versions. Though Ambari allows you to downgrade to earlier versions of Kafka after you upgrade to 2.1.0, consider the following before you do so: If you upgrade both the broker and consumer to 2.1.0, a downgrade might cause data loss. If you upgrade the broker to 2.1.0 and the consumer remains at 2.0.0, then you can downgrade the broker to 2.0.0 without any issue.

Warning

We strongly discourage downgrading from Kafka 2.1.0 to earlier versions. Though Ambari allows you to downgrade to earlier versions of Kafka after you upgrade to 2.1.0, consider the following before you do so:

If you upgrade both the broker and consumer to 2.1.0, a downgrade might cause data loss.
If you upgrade the broker to 2.1.0 and the consumer remains at 2.0.0, then you can downgrade the broker to 2.0.0 without any issue.

New features and improvements in Apache Kafka include:

KAFKA-4514 Add Codec for ZStandard Compression (KIP-110).
KAFKA-4682 Committed offsets should not be deleted if a consumer is still active (KIP-211).
KAFKA-5886 Introduce delivery.timeout.ms producer config (KIP-91).
KAFKA-7264 Initial Kafka support for Java 11.
KAFKA-7333 Protocol changes to allow fetchers to detect and handle log truncation (KIP-320).
KAFKA-7394 Allow OffsetsForLeaderEpoch requests with topic describe ACL (KIP-320).
KAFKA-7395 Add fencing to replication protocol (KIP-320).
Streams API improvements (KIP-319, KIP-321, KIP-330, KIP-353, KIP-356).
Admin script and admin client API improvements to simplify admin operation (KIP-231, KIP-308, KIP-322, KIP-324, KIP-338, KIP-340).
DNS handling improvements (KIP-235, KIP-302).

For a full list of features and bug fixes, see the following: https://dist.apache.org/repos/dist/release/kafka/2.1.0/RELEASE_NOTES.html.

Apache NiFi Updates

NiFi has been upgraded from 1.8.0 to 1.9.0.

New features and improvements in Apache NiFi include:

New processors and controller services can be loaded live while the nodes are running.
Database connection pooling now supports Kerberos secured Databases.
Penalty duration remaining now visible in the UINodes can be offloaded to prepare for decommissioning.
Several standard security headers now supported (HSTS, X-XSS-Protection, Content-Security-Policy).
Record Readers and Writers now easier to use with automatic schema inference.
New or Improved Processors and Controller Services
- Kudu Processors updated, validated, and now support Kerberos.
- Various updates to Amazon Web Services, Big Query, and Google Cloud Storage Processors.
- MongoDB GridFS processors now included.
- HiveQL for Hive 1.1 now supported.

For a full list of features and bug fixes, see the following: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344357.

Apache NiFi Registry Updates

NiFi Registry has been upgraded from 0.2.0 to 0.3.0.

New features and improvements in Apache NiFi Registry include:

New Apache Ranger Authorizer allows authorization rules to be defined in Apache Ranger.
Expansion and refinement of the Versioned Flow data model to facilitate easier use by consumers.
Security enhancements: upgrades dependencies that have been patched for security issues.
Event Hook API improvements
- New REGISTRY_START event type
- Event Type Whitelisting feature for Event Hook Providers
Docker image improvements.

For a full list of features and bug fixes, see the following: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343483&styleName=Html&projectId=12320920

Schema Registry Updates

Schema Registry has been upgraded from 0.5.4 to 0.7.0.

New features and improvements in Schema Registry include:

ISSUE-509: Provide a delete API for schema metadata (#510).
ISSUE-496: Kafka Streams and Schema Registry Integration Support.
ISSUE-500: Changes related to enabling JWT based SSO along with SPNEGO/Kerberos.
EAR-9794: SchemaRegistry UI is not getting rendered properly.
ISSUE-513: Fix storage manager to correctly handle the types in queries.
ISSUE-517 Fix concurrent schema version adds.
ISSUE-502: Field props stripped in deserialization.
Update documentation: schema-registry.rst.
ISSUE-543 Fix broken json format in Schema.Field when it's used as nested - introduced in Jackson 2.9added test which doesn't make sense for Streamline to have.
ISSUE-538 Move transaction package from common to storage-core. This helps us to isolate storage-core with registry-common, which brings huge number of transitive dependencies especially from hadoop-client.
ISSUE-540 Exclude Avro dependency from registry-common moduleThis would save lots of modules which has registry-common as a transitive dependency.
Upgrade jackson-databind to v2.9.8 to fix the following CVEs: CVE-2018-14719, CVE-2018-14720, CVE-2018-14721, CVE-2018-1000873, CVE-2018-7489, CVE-2018-19362, CVE-2017-15095, CVE-2018-19361, CVE-2017-7525, CVE-2018-19360, CVE-2017-17485, CVE-2018-5968.
Remove deprecated API usages.
- added logback.xml file to adjust logger levels.
- fixed the typographical errors and broken links in documents.
ISSUE-506: Resolve reported CVEs (CVE-2014-0114, CVE-2015-6420, CVE-2015-7501, CVE-2017-15708, CVE-2012-5783, CVE-2014-3577, CVE-2012-6153, CVE-2015-5262, CVE-2018-1288, CVE-2009-2625, CVE-2012-0881, CVE-2009-1523, CVE-2011-4461, CVE-2017-9735, CVE-2017-7656, CVE-2017-7657, CVE-2017-7658, CVE-2018-12536, CVE-2014-0193, CVE-2015-2156, CVE-2016-4970).

Apache Ranger Updates

Ranger plugin properties in Ambari and scripting requirements for NiFi Registry are enabled.