Release Notes
Also available as:
PDF

Chapter 1. Hortonworks DataFlow 3.0.1.1 Release Notes

This document provides you with the latest information about the HDF 3.0.1.1 release and its product documentation.

Component Support

HDF 3.0.1.1 includes the following components:

  • Apache Ambari 2.5.1

  • Apache Kafka 0.10.2.1

  • Apache NiFi 1.2.0

  • Apache Ranger 0.7.0

  • Apache Storm 1.1.0

  • Apache ZooKeeper 3.4.6

  • Apache MiNiFi Java Agent 0.2.0

  • Apache MiNiFi C++ Technical Preview

  • Hortonworks Schema Registry 0.3.0

  • Hortonworks Streaming Analytics Manager 0.5.0

Component Availability in HDF

Previous HDF releases shipped with the following components versions.

 NiFiStormKafkaZooKeeperAmbariRangerMiNiFi Java AgentMiNiFi C++Streaming Analytics ManagerSchema Registry
HDF 3.0.1.11.2.01.1.00.10.2.1 3.4.62.5.10.7.00.2.0TP0.5.00.3.0
HDF 3.0.11.2.01.1.00.10.2.1 3.4.62.5.10.7.00.2.0TP0.5.00.3.0
HDF 3.0.01.2.01.1.00.10.2.1 3.4.62.5.10.7.00.2.0TP0.5.00.3.0
HDF 2.1.41.1.01.0.21.10.13.4.62.4.2.00.6.20.1.0TPN/AN/A
HDF 2.1.21.1.01.0.20.10.13.4.62.4.2.00.6.20.1.0TPN/AN/A
HDF 2.1.11.1.01.0.20.10.13.4.62.4.2.00.6.20.1.0TPN/AN/A
HDF 2.1.01.1.01.0.20.10.13.4.62.4.2.00.6.20.1.0TPN/AN/A
HDF 2.0.21.0.01.0.10.10.0.13.4.62.4.1.00.6.00.0.1TPN/AN/A
HDF 2.0.11.0.01.0.10.10.0.13.4.62.4.1.00.6.00.0.1TPN/AN/A
HDF 2.0.01.0.01.0.10.10.0.13.4.62.4.0.10.6.00.0.1TPN/AN/A
HDF 1.2.10.6.10.10.00.9.0.13.4.6N/AN/AN/AN/AN/AN/A
HDF 1.2.0.10.6.10.10.00.9.0.13.4.6N/AN/AN/AN/AN/AN/A
HDF 1.2.00.6.00.10.00.9.0.13.4.6N/AN/AN/AN/AN/AN/A
HDF 1.1.00.4.0N/AN/AN/AN/AN/AN/AN/AN/AN/A
HDF 1.00.3.0N/AN/AN/AN/AN/AN/AN/AN/AN/A

What's New in HDF 3.0.1.1

HDF 3.0.1.1 is a hotfix release that includes the following bug fix:

  • BUG-85312 – Substitute registry.version for NiFi builds.

For complete information about the HDF 3.x release, see:

Unsupported Features

Some features exist within HDF 3.0.1.1, but Hortonworks does not currently support these capabilities.

Technical Preview Features

The following features are available within HDF 3.0.1.1 but are not ready for production deployment. Hortonworks encourages you to explore these technical preview features in non-production environments and provide feedback on your experiences through the Hortonworks Community Forums.

Table 1.1. Technical Previews

ComponentFeature

MiNiFi

MiNiFi C++
Streaming Analytics Manager
  • SAM Stream Insights module (Druid and Apache Superset)

  • Sinks

    • Cassandra

    • OpenTSDB

    • Solr

  • Sources

    • HDFS


Community Driven Features

The following features are developed and tested by the Hortonworks community but are not officially supported by Hortonworks. These features are excluded for a variety of reasons, including insufficient reliability or incomplete test case coverage, declaration of non-production readiness by the community at large, and feature deviation from Hortonworks best practices. Do not use these features in your production environments.

Community Driven Kafka features

  • Kafka Connect

  • Kafka Streams

Community Driven NiFi Tools and Services

  • Embedded ZooKeeper

  • Sensitive key migration toolkit

  • Docker image for Apache NiFi

Community Driven NiFi Processors

  • AttributeRollingWindow

  • AWSCredentialsProviderControllerService

  • CompareFuzzyHash

  • ConsumeEWS

  • ConsumeIMAP

  • ConsumePOP3

  • ConvertExcelToCSVProcessor

  • DebugFlow

  • DeleteDynamoDB

  • DeleteGCSObject

  • DeleteHDFS

  • ExtractCCDAAttributes

  • ExtractEmailAttachments

  • ExtractEmailHeaders

  • ExtractMediaMetadata

  • ExtractTNEFAttachments

  • FetchAzureBlobStorage

  • FetchGCSObject

  • FuzzyHashContent

  • GetDynamoDB

  • GetHDFSEvents

  • GetSNMP

  • ISPEnrichIP

  • InferAvroSchema

  • ListenBeats

  • ListenLumberjack

  • ListenSMTP

  • ListAzureBlobStorage

  • ListGCSBucket

  • ListS3

  • ModifyBytes

  • OrcFormatConversion

  • PutAzureBlobStorage

  • PutDynamoDB

  • PutGCSObject

  • PutIgniteCache

  • PutKinesisFirehose

  • PutKinesisStream

  • PutLambda

  • PutSlack

  • PutTCP

  • PutUDP

  • QueryDNS

  • SetSNMP

  • SpringContextProcessor

  • StoreInKiteDataset

Community Driven NiFi Controller Services

  • AWSCredentialsProviderControllerService

  • GCPCredentialsControllerService

Community Driven NiFi Reporting Tasks

  • SiteToSiteBulletinReportingTask

  • SiteToSiteStatusReportingTask

  • DataDogReportingTask

Unsupported Customizations

Hortonworks cannot guarantee that default NiFi processors are compatible with proprietary protocol implementations or proprietary interface extensions. For example, we support interfaces like JMS and JDBC that are built around standards, specifications, or open protocols. But we do not support customizations of those interfaces, or proprietary extensions built on top of those interfaces.

Deprecated Technologies

This section points out any technology from previous releases that has been deprecated or removed from this release (operating systems, Java versions, databases, product features). Use this section as a guide for your implementation plans.

Deprecated

Technology that Hortonworks is removing in a future release. Deprecated items are supported until they are removed; deprecation gives you time to plan for removal.

Removed

Technology that Hortonworks has removed from production and is no longer supported.

Table 1.2. Deprecated Operating Systems

Operating System

Release DeprecatedRelease Removed
Debian 6HDF 2.1.2HDF 3.0.0
Ubuntu 12HDF 3.0.0HDF 3.0.0

Table 1.3. Deprecated NiFi Processors

Processor

Release Deprecated
ConvertCSVToAvroHDF 3.0.0
ConvertJSONToAvroHDF 3.0.0
GetKafka HDF 2.0.0
PutKafkaHDF 2.0.0
EvaluateRegularExpressionHDF 1.0.0

HDF Repository Locations

Use the following table to identify the HDF 3.0.1.1 repository location for your operating system and operational objectives. HDF 3.0.1.1 supports the following operating systems:

Table 1.4. RHEL/Oracle Linux/CentOS 6 HDF repository & additional download locations

OS FormatDownload location
RHEL/Oracle Linux/CentOS 6 (64-bit):HDF Base URL http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.1
HDF Repohttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.1/hdf.repo
RPM tarballhttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.1/HDF-3.0.1.1-centos6-rpm.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.1/HDF-3.0.1.1-centos6-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.1-5.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.1.0/ambari.repo
HDP http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.1.0/hdp.repo
OS Agnostic Downloads
NiFi only
NiFi Toolkit
MiNiFi Java Agent
MiNiFi Toolkit

Table 1.5. RHEL/Oracle Linux/CentOS 7 HDF repository & additional download locations

OS FormatDownload location
RHEL/Oracle Linux/CentOS 7 (64-bit):HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.1.1
HDF Repohttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.1.1/hdf.repo
RPM tarballhttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.1.1/HDF-3.0.1.1-centos7-rpm.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.1.1/HDF-3.0.1.1-centos7-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.0.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.1-5.tar.gz
HDP and Ambari Repositories
Ambari http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.1.0/ambari.repo
HDP http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/hdp.repo
OS Agnostic Downloads
NiFi only
NiFi Toolkit
MiNiFi Java Agent
MiNiFi Toolkit

Table 1.6. SLES 11 SP3/SP4 HDF repository & additional download locations

OS FormatDownload location
SUSE Enterprise Linux 11 SP3, SP4 HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.1
Repohttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.1/hdf.repo
RPM tarball http://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.1/HDF-3.0.1.1-suse11sp3-rpm.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.1/HDF-3.0.1.1-suse11sp3-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.0.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.1-5.tar.gz
HDP and Ambari Repositories
Ambari http://public-repo-1.hortonworks.com/ambari/suse11/2.x/updates/2.5.1.0/ambari.repo
HDP http://public-repo-1.hortonworks.com/HDP/suse11sp3/2.x/updates/2.6.1.0/hdp.repo
OS Agnostic Downloads
NiFi only
NiFi Toolkit
MiNiFi Java Agent
MiNiFi Toolkit

Table 1.7. SLES 12 HDF repository & additional download locations

OS FormatDownload location
SUSE Linux Enterprise Server (SLES) v12 SP1HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.1
Repohttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.1/hdf.repo
RPM tarballhttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.1/HDF-3.0.1.1-sles12-rpm.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.1/HDF-3.0.1.1-sles12-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.0.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.1-5.tar.gz
HDP and Ambari Repositories
Ambari http://public-repo-1.hortonworks.com/ambari/sles12/2.x/updates/2.5.1.0/ambari.repo
HDP http://public-repo-1.hortonworks.com/HDP/sles12/2.x/updates/2.6.1.0/hdp.repo
OS Agnostic Downloads
NiFi only
NiFi Toolkit
MiNiFi Java Agent
MiNiFi Toolkit

Table 1.8. Ubuntu 14 HDF repository & additional download locations

OS FormatDownload location
Ubuntu Trusty (14.04) (64-bit)HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.1.1
Repohttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.1.1/hdf.list
Deb tarballhttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.1.1/HDF-3.0.1.1-ubuntu14-deb.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.1.1/HDF-3.0.1.1-ubuntu14-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.1-5.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.5.1.0/ambari.list
HDP http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.6.1.0/hdp.list
OS Agnostic Downloads
NiFi only
NiFi Toolkit
MiNiFi Java Agent
MiNiFi Toolkit

Table 1.9. Debian 7 HDF repository & additional download locations

OS FormatDownload location
Debian 7HDF Base URL http://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.1
Repohttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.1/hdf.list
Deb tarballhttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.1/HDF-3.0.1.1-debian7-deb.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.1/HDF-3.0.1.1-debian7-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.0.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.1-5.tar.gz
HDP and Ambari Respositories
Ambari http://public-repo-1.hortonworks.com/ambari/debian7/2.x/updates/2.5.1.0/ambari.list
HDPhttp://public-repo-1.hortonworks.com/HDP/debian7/2.x/updates/2.6.1.0/hdp.list
OS Agnostic Downloads
NiFi only
NiFi Toolkit
MiNiFi Java Agent
MiNiFi Toolkit

Common Vulnerabilities and Exposures

The following CVEs have been fixed in HDF 3.0.1.1.

CVE-2016-8746

Summary: Apache Ranger path matching issue in policy evaluation
Severity: Normal
Vendor: Hortonworks
Versions Affected: All HDF versions 2.0.x and 2.1.x using Apache Ranger versions 0.6.0/0.6.1/0.6.2.
Users affected: All users of the ranger policy admin tool.
Impact: Ranger policy engine incorrectly matches paths in certain conditions when a policy contains wildcards and recursive flags.
Fix detail: Fixed policy evaluation logic
Recommended Action: Users should upgrade to HDF 3.0.0 (with Apache Ranger 0.7.0+)

CVE-2016-8751

Summary: Apache Ranger stored cross site scripting issue
Severity: Normal
Vendor: Hortonworks
Versions Affected: HDF versions 2.0.x, 2.1.x using Apache Ranger versions 0.6.0/0.6.1/0.6.2.
Users affected: All users of the ranger policy admin tool.
Impact: Apache Ranger is vulnerable to a Stored Cross-Site Scripting when entering custom policy conditions. Admin users can store some arbitrary javascript code execute when normal users login and access policies.
Fix detail: Added logic to sanitize the user input.
Recommended Action: Users should upgrade to HDF 3.0.0 (with Apache Ranger 0.7.0+)

CVE-2017-7676

Summary: Apache Ranger policy evaluation ignores characters after ‘*’ wildcard character
Severity: Critical
Vendor: Hortonworks
Versions Affected: HDF versions 2.0.x, 2.1.x.
Users affected: Environments that use Ranger policies with characters after ‘*’ wildcard character – like my*test, test*.txt
Impact: Policy resource matcher ignores characters after ‘*’ wildcard character, which can result in unintended behavior.
Fix detail: Ranger policy resource matcher was updated to correctly handle wildcard matches.
Recommended Action: Upgrade to HDF 3.0.0.

CVE-2017-7677

Summary: Apache Ranger Hive Authorizer should check for RWX permission when external location is specified
Severity: Critical
Vendor: Hortonworks
Versions Affected: HDF versions 2.0.x, 2.1.x.
Users affected: Environments that use external location for hive tables
Impact: In environments that use external location for hive tables, Apache Ranger Hive Authorizer should check for RWX permission for the external location specified for create table.
Fix detail: Ranger Hive Authorizer was updated to correctly handle permission check with external location.
Recommended Action: Upgrade to HDF 3.0.0.

Known Issues

Hortonworks Bug ID

Apache JIRA

Component

Summary

BUG-82327 Streaming Analytics Manager

Issue: In HA installations of SAM, you may run into a situation where bootstrap has not been executed.

Result: When bootstrap has not been started, the UI displays the following message:

1. No component definitions found.
2. Please run ./bin/streamline bootstrap to 
initialize the component definitions.

Workaround for a secured cluster:

kinit -kt /etc/security/keytabs/streamline.service.keytab 
streamline-{cluster_name_in_lower-case}@domain

Workaround for an unsecured cluster:

cd /usr/hdf/current/streamline
./bootstrap/bootstrap.sh
BUG-82681AMBARI-21261Ambari/ NiFi

Issue: When NiFi is installed on an Ambari-managed HDP cluster, NiFi information is not available in Ambari Metrics.

Workaround: There is no workaround for this issue.

BUG-82311AMBARI-21263Ambari/Solr/NiFi

Issue: The Ambari Server upgrade fails when the HDP Search (Solr) or HDF management pack is installed.

Error Message:

File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 896, in install_mpack
    (mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode)
  File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 792, in _install_mpack
    process_service_definitions_artifact(artifact, artifact_source_dir, options)
  File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 515, in process_service_definitions_artifact
    create_symlink(src_service_definitions_dir, dest_service_definitions_dir, file, options.force)
  File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 235, in create_symlink
    create_symlink_using_path(src_path, dest_link, force)
  File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 247, in create_symlink_using_path
    sudo.symlink(src_path, dest_link)
  File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 123, in symlink
    os.symlink(source, link_name)
OSError: [Errno 17] File exists

Workaround for HDP Search mpack: To work around this issue, enter the following commands:

rm -rf /var/lib/ambari-server/resources/common-services_05_06_17_19_49.old/SOLR
rm -rf /var/lib/ambari-server/resources/common-services/SOLR

Workaround for HDF mpack: To work around this issue, enter the following commands:

rm -rf /var/lib/ambari-server/resources/common-services/NIFI/ 
rm -rf /var/lib/ambari-server/resources/common-services_17_02_17_18_20.old/NIFI/ 
rm -rf /var/lib/ambari-server/resources/common-services_20_06_17_15_47.old/NIFI/ 
[Note]Note

In each case, the date-time pattern is a timestamp of the management pack file and differs in every case.

BUG-82561AMBARI-21291Ambari/HDF

Issue: You may experience a schema error when upgrading to Ambari 2.5.1.

Error Message:

WARN [main] DBAccessorImpl:814 - Error executing query: UPDATE request AS a SET cluster_host_info = b.cluster_host_info FROM stage AS b WHERE a.request_id = b.request_id, errorCode = 0, message = ERROR: column b.cluster_host_info does not exist 
Position: 45 
20 Jun 2017 19:45:23,466 ERROR [main] SchemaUpgradeHelper:209 - Upgrade failed. 
org.postgresql.util.PSQLException: ERROR: column b.cluster_host_info does not exist 
Position: 45

Workaround: To work around this issue, re-add the column to the stage table before issuing the 'ambari-server upgrade' command.

Example for updating the Ambari Postgres database:

  1. psql -U ambari (default password is 'bigdata')

  2. alter table stage add column cluster_host_info bytea not null default '{}';

Third-Party Licenses

HDF 3.0.1.1 deploys numerous third-party licenses and dependencies, all of which are compatible with the Apache software license. For complete third-party license information, see the licenses and notice files contained within the distribution.