Release Notes
Also available as:

What's New in HDF 3.2.0

HDF 3.2.0 is a minor release that includes the following new features and bug fixes. In addition to the updates listed below, HDF 3.2.0 also includes Knox 1.0.0 in the public repository.

Apache Kafka Updates

Kafka has been upgraded from 1.0.0 to 1.1.0 with some critical bug fixes including:

  • KAFKA-6172 - Cache lastEntry in TimeIndex to avoid unnecessary disk access

  • KAFKA-6175 - AbstractIndex should cache index file to avoid unnecessary disk access during resize()

  • KAFKA-6258 - SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data

In addition to these bug fixes, an important new feature was added to capture producer and topic partition level metrics without instrumenting or configuring interceptors on the clients.

This provides an non-invasive approach to capture important metrics for producers without refactoring/modifying your existing Kafka clients.

Apache Storm Updates

Storm has been upgraded from 1.1.0 to 1.2.1. Storm 1.2.1 now supports all HDP 3.0 components including Hadoop/HDFS 3.0, HBase 2.0 and Hive 3.

Key Bug Fixes include:

  • STORM-2993 - Storm HDFS bolt throws ClosedChannelException when Time rotation policy is used

  • BUG-104149 - Prevent supervisor crash when topology conf file doesn't exist in updateBlobsForTopology

  • BUG-94588 - Skip the user while checking isProcessAlive

For a full list of bug fixes, see the following:

SAM Updates

New Features and Improvements in SAM include:

  • Rule, Projection and Aggregate processor supports writing complex conditions

  • Atlas integration – Ability to import ‘Atlas’ as a service and push SAM app information to Atlas during deployment

  • Compatibility with HDP 3.0 (Hive 3.0, Hbase 2.0, Hadoop/HDFS 3.0)

  • Test mode enhancement to support test mode without a cluster

  • Support for updating a SAM app environment

Key Bug Fixes include:

  • SAM view mode metrics related fixes

  • Custom processor handling null values

Schema Registry Updates

Schema Registry has been upgraded from 0.5.0 to 0.5.2 with some important improvements including:

  • ISSUE-441 Let addOrUpdate be compatible with PostgreSQL under 9.5

  • ISSUE-418 Add JWT based SSO as alternative authentication to common-auth module.

  • ISSUE-420 Add migration to add extents for created tables to fix serializable as an isolation level for Oracle database

  • ISSUE-415 Client side schema validation - Disable or Customize

Apache NiFi Updates

NiFi has been upgraded from 1.5.0 to 1.7.0. NiFi 1.7.0 now supports all HDP 3.0 components including Hadoop/HDFS 3.0, HBase 2.0 and Hive 3.

New Features:

  • NIFI-5214 - Add a REST lookup service

  • NIFI-5133 - Create a processor to Publish and Subscribe to Google Pub/Sub

  • NIFI-4982 - Add a DistributedMapCacheLookupService

  • NIFI-4963 - Add support for Hive 3.0 processors

  • NIFI-4917 - Create a new Controller Service for specifying Keytabs

  • NIFI-4906 - Add GetHdfsFileInfo Processor

  • NIFI-4872 - NIFI component high resource usage annotation

  • NIFI-4833 - Add ScanHBase processor

  • NIFI-4809 - Implement a SiteToSiteMetricsReportingTask

  • NIFI-4227 - Create a ForkRecord processor

  • NIFI-4185 - Add XML record reader

  • NIFI-5186 - Update UI to account for asynchronous validation

Key Bugs Fixed:

  • NIFI-5200 - Nested resulting in outer stream being closed.

  • NIFI-5033 - Cannot update variable referenced in restricted components

  • NIFI-4837 - Thread leak on HandleHTTPRequest processor

Apache NiFi Registry Updates

NiFi Registry has been upgrade from 0.1.0 to 0.2.0.

New Features:

  • NIFIREG-162 Add Git backed persistence provider

  • NIFIREG-173 Allow metadata DB to use other DBs besides H2

  • NIFIREG-172 Add self-hosted Swagger UI to nifi-registry-web-api WAR

  • NIFIREG-160 Implement a hook provider allowing users to create their own hook implementations that execute when a new flow is committed to the registry

  • NIFIREG-134 Enable Spring Boot Actuator REST API endpoints

  • NIFIREG-120 Docker image for NiFi Registry

  • NIFIREG-109 LdapUserGroupProvider: Allow admin to configure group membership attribute

  • NIFIREG-39 Leverage Apache NiFi Flow Design System

Key Bugs Fixed:

  • NIFIREG-174 - Parameter conf.dir in bootstrap.conf not working

  • NIFIREG-153 - The angular router module not properly injected.

  • NIFIREG-144 - NPE when accessing for metadata of 'latest' version and no versions exists

  • NIFIREG-140 - Nifi Registry not able to start - NoClassDefFoundError org/apache/nifi/registry/util/FileUtils

  • NIFIREG-126 - Entering an invalid bucket id in a deep link causes JS error

Apache MiNiFi Java Updates

MiNiFi Java has been upgraded from 0.4.0 to 0.5.0. Improvements include integrations with the MiNiFi C2 server.

New Features:

  • MINIFI-449 - Transform NiFi Registry's VersionedFlowSnapshot to YAML

  • MINIFI-441 - Improved Docker image

  • MINIFI-434 - Preserve security properties when pulling a new config

Key Bugs Fixed:

  • MINIFI-460 - SiteToSiteProvenanceReporting task fails to start

  • MINIFI-457 - Getting flow status for a processor returns "no processor"

  • MINIFI-456 - Exception when hitting MiNiFi C2

  • MINIFI-450 - PullHttpChangeIngestor needs to properly close OkHttp response

  • MINIFI-435 - MiNiFi - Cannot perform secured S2S with NiFi 1.5.0

  • MINIFI-431 - C2 - Exception while agent tries to pull the config

Apache MiNiFi C++ Updates

MiNiFi C++ has been upgraded from 0.4.0 to 0.5.0. Improvements around stability, extendible processors, and interactive ease of use have been made.

New Features:

Key Bugs Fixed:

  • MINIFICPP-495 - will exit with error in default install Debian systems

  • MINIFICPP-430 - GetFile incorrectly reports 0 file size for non-regular files

  • MINIFICPP-396 - RPG does not pass SSLContextService to site to site client in a consistent way

  • MINIFICPP-389 - FlexLexer.h missing, causing broken Alpine docker build

  • MINIFICPP-385 - RPG destruction can lead to EOFException in NiFi when sockets are not closed.