# Copyright 2012, Hortonworks Inc. All rights reserved.# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # RELEASE NOTES: Hortonworks Data Platform with Hortonworks Management Console powered by Apache Hadoop Product Version: HDP-2.0.0.1 (Alpha) ============================================ This release of Hortonworks Data Platform (HDP) deploys the following Hadoop-related components: * Apache Hadoop 2.0.2 (Alpha) * Apache HBase 0.94.2 * Apache Pig 0.9.2 * Apache ZooKeeper 3.4.3 * Apache HCatalog 0.4.0 * Apache Hive 0.9.0 * Apache Oozie 3.2.0 * Hortonworks Management Center (HMC) 2.0.0.1 Third party components: * Ganglia 3.2.0 * Nagios 3.2.3 Patch Information ================================ Apache Hadoop 2.0.2 (Alpha) --------------------------- * No patches are added for Apache Hadoop 2.0.2 (Alpha). Apache HBase 0.94.2 --------------------------- * No patches are added for Apache HBase 0.94.2. Apache Pig 0.9.2 --------------------------- Pig is patched to include the following: PIG-2766: Improved Pig-HCatalog usability. PIG-2791: Fixed issue with using ViewFileSystem for Pig. Apache ZooKeeper 3.4.3 --------------------------- * No patches are added for Apache ZooKeeper 3.4.3. Apache HCatalog 0.4.0 --------------------------- * HCatalog is patched to include the following: HCATALOG-412: HCatalog now publishes artifacts to the local M2 cache. HCATALOG-410: Added support for proxy user in HCatalog client. HCATALOG-420: Backport HCATALOG-363 for 0.4 branch. HCATALOG-485: Added documentation for storage-based security which ignores GRANT/REVOKE statements. For details, see: http://docs.hortonworks.com/HCatalog/CURRENT/authorization.html HCATALOG-431: Added documentation for mapping HCatalog type to Java class/Pig type. HCATALOG-492: Added documentation for CTAS workaround for Hive with JSON SerDe. For details, see: http://docs.hortonworks.com/HCatalog/CURRENT/supportedformats.html HCATALOG-442: Updated instructions for using HCatalog with Pig. HCATALOG-482: Added instructions on using -libjars from HDFS for HCatalog with MapReduce. HCATALOG-481: Fixed command line interface (CLI) usage syntax in documentation and revised HCatalog documentation. For details, see: http://docs.hortonworks.com/HCatalog/CURRENT/cli.html HCATALOG-444: Added documenatation for Reader and Writer Interfaces. For details, see: http://docs.hortonworks.com/HCatalog/CURRENT/readerwriter.html HCATALOG-427: Added documenatation for using storage-based authorization. For details, see: http://docs.hortonworks.com/HCatalog/CURRENT/authorization.html HCATALOG-448: Added changes to datanucleus to build with version 3.x instead of version 2.x to fix the performance related issue with HCatStorer. HCATALOG-350: Fixed dependecy for HCatRecord. Writing BINARY data to HCatRecord now does not depend on a Hive class. HCATALOG-436: Fixed incorrect naming for JSON SerDe column on CTAS. HCATALOG-471: Fixed Test HCat_ShowDes_1[1-3] failures. HCATALOG-375: Added support to make HCatalog interoperable with Hadoop 0.23. Apache Hive 0.9.0 --------------------------- * Hive is patched to include the following: HIVE-2084: Upgraded DataNucleus from v2.0.3 to v3.0.1. HIVE-2918: Fixed issue with Hive Dynamic Partition Insert. The move task will now consider 'hive.exec.max.dynamic.partitions'. HIVE-3008: Fixed memory leak in TUGIContainingTransport. HIVE-3063: Fixed failures when using drop partition for non-string columns. HIVE-3076: Fixed failures when using drop partition for non-partition columns HIVE-3168: Fixed copy issue for LazyBinaryObjectInspector.getPrimitiveJavaObject. HIVE-3246: Fixed binary datatype for Java primitive type. HIVE-3153: Added improvements to reduce memory consumption per a RCFile.Writer. HIVE-3291: Fixed issues with fs resolvers. HIVE-3098: Fixed memory leak issue caused by large number of FileSystem instances in FileSystem.CACHE. HIVE-2928: Added support for Oracle-backed Hive-Metastore ("longvarchar" to "clob" in package.jdo). HIVE-3082: Added support for Oracle Metastore schema script to include DDL for DataNucleus' internal tables. Apache Oozie 3.2.0 --------------------------- * Oozie is patched to include the following: OOZIE-698: Added support to configure version dependency for the sharelib components. OOZIE-810: Updated Oozie POM file to use Doxia 9.2y from the available repository. OOZIE-863: Fixed failures when invoking the oozie-env.sh script file. The JAVA_HOME does not need to be set explicitly at client. OOZIE-968: Added support to source Oozie environment from conf in Oozie db setup script file. OOZIE-947: Forward porting OOZIE-733 to 3.2 and trunk. OOZIE-1006: Added support to make Oozie interoperable with Hadoop 2.0.2. Hortonworks Management Center (HMC) 2.0.0.1 --------------------------- * HMC is patched to include the following: AMBARI-664: Fixed issue with mapred io sort mb and heap size for map/reduce. AMBARI-641: Added support to change the nagios status.data file location according to the underlying platform. AMBARI-628: Fixed configuration and permission issues with hdp-nagios and hdp-monitoring files. AMBARI-633: Fixed invalid HTML markup for the Monitoring Dashboard. AMBARI-597: Removed /usr/bin/php dependency from the RPMs. AMBARI-701: Added support to handle the pre-setup user-supplied Hive Metastore. Minimum system requirements ============================================== Hardware Recommendations --------------------------- Although there is no single hardware requirement for installing HDP, there are some basic guide­lines. You can see sample setups here: http://docs.hortonworks.com/HDP2Alpha/About_Hortonworks_Data_Platform/Hardware_Recommendations_For_Apache_Hadoop.htm Operating Systems Requirements ------------------------------------ The following operating systems are supported: * 64-bit Red Hat Enterprise Linux (RHEL) v5.*, v6.* * 64-bit CentOS v5.*, v6.* IMPORTANT: All hosts in the cluster must run the same OS, version and patch sets. Graphics Requirements ------------------------ The HMC deployment wizard runs as a browser-based Web app. You must have a machine capable of running a graphical browser to use this tool. Software Requirements ----------------------- On each of your hosts: * yum * rpm * scp * curl * wget * pdsh On the machine from which you will run HMC: * Firefox v.12+ Database Requirements ----------------------- Hive or HCatalog requires a MySQL database for its use. You can choose to use a current instance or let the HMC deployment wizard create one for you. Optional: Configure the local repositories ------------------------------------------ If your cluster does not have access to the Internet, or you are creating a large cluster and you want to conserve bandwidth, you need to provide access to the HDP installation packages using an alternative method. For more information, see http://docs.hortonworks.com/HDP2Alpha/Appendix/Deploying_HDP_In_Production_Data_Centers_with_Firewalls/Deploying_HDP_In_Production_Data_Centers.htm IMPORTANT: The installer pulls many packages from the base OS repos. If you do not have a complete base OS available to all your machines at the time of installation, you may run into issues. For example, if you are using RHEL 6 your hosts must be able to access the “Red Hat Enterprise Linux Server 6 Optional (RPMs)” repo. If this repo is disabled, the installation is unable to access the rubygems package, which is necessary for HMC to operate. If you encounter problems with base OS repos being unavailable, please contact your system administrator to arrange for these additional repos to be proxied or mirrored. Improvements ============================================= * Hadoop updated to upstream version 2.0.2 (Alpha). * HBase updated to upstream version 0.94.2. * ZooKeeper updated to upstream version 3.4.3. * Oozie updated to upstream version 3.2.0. * HMC updated to upstream version 2.0.0.1. Known issues ============================================= * The ALTER INDEX command in an automated script that also contains CREATE INDEX command will fail for Hive. The workaround is to either use the ALTER INDEX command in an interactive shell or add this command in a separate script file. * Hive and HCatalog authorizations are based on permissions in the underlying storage system and so are not affected by account-management DDL statements such as GRANT and REVOKE. For details, see: http://docs.hortonworks.com/HCatalog/CURRENT/authorization.html * Preview of the mount point directories will display the Oozie and ZooKeeper directories even when the corresponding services are not enabled. For details, see: https://issues.apache.org/jira/browse/AMBARI-572 * While finalizing the bootstrap nodes for HMC, in some cases the update might show incorrect message. * HMC installation currently does not support Hadoop security. * It is not recommended to use init.d scripts for starting or stopping Hadoop services. * Pig or MapReduce jobs get incorrect data when reading binary data type from the HCatalog table. For more details, see: https://issues.apache.org/jira/browse/HCATALOG-430. Hortonworks offers technical support subscriptions for Hortonworks Data Platform. For more information, please visit http://hortonworks.com/support/ or contact us directly at info@hortonworks.com.