Installing HDP Manually
Copyright © 2012-2015 Hortonworks, Inc.
 | Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 4.0 License |
2015-10-13
Abstract
The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Hadoop projects including MapReduce, Hadoop Distributed File System (HDFS), HCatalog, Pig, Hive, HBase, ZooKeeper and Ambari. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included.
Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain, free and open source.
Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. Feel free to contact us directly to discuss your specific needs.
Contents
- 1. Getting Ready to Install
- Meet Minimum System Requirements
- Virtualization and Cloud Platforms
- Configure the Remote Repositories
- Decide on Deployment Type
- Collect Information
- Prepare the Environment
- Download Companion Files
- Define Environment Parameters
- [Optional] Create System Users and Groups
- Determine HDP Memory Configuration Settings
- Configuring NameNode Heap Size
- Allocate Adequate Log Space for HDP
- 2. Installing HDFS and YARN
- 3. Installing Apache ZooKeeper
- 4. Setting Up the Hadoop Configuration
- 5. Validating the Core Hadoop Installation
- 6. Installing Apache HBase
- 7. Installing Apache Phoenix
- 8. Installing and Configuring Apache Tez
- 9. Installing Apache Hive and Apache HCatalog
- Installing the Hive-HCatalog RPM
- Setting Directories and Permissions
- Setting Up the Hive/HCatalog Configuration Files
- Setting Up RDBMS for Use with the Hive Metastore
- Creating Directories on HDFS
- Validating the Installation
- Enabling Tez for Hive Queries
- Disabling Tez for Hive Queries
- Configuring Tez with the Capacity Scheduler
- Validating Hive-on-Tez Installation
- 10. Installing Apache Pig
- 11. Installing Apache WebHCat
- 12. Installing Apache Oozie
- 13. Installing Apache Ranger
- 14. Installing Hue
- 15. Installing Apache Sqoop
- 16. Installing Apache Mahout
- 17. Installing and Configuring Apache Flume
- 18. Installing and Configuring Apache Storm
- 19. Installing and Configuring Apache Spark
- 20. Installing and Configuring Apache Kafka
- 21. Installing Apache Accumulo
- 22. Installing Apache Falcon
- 23. Installing Apache Knox
- 24. Installing Ganglia (Deprecated)
- 25. Installing Nagios (Deprecated)
- Install the Nagios RPMs
- Install the Configuration Files
- Extract the Nagios Configuration Files
- Create the Nagios Directories
- Copy the Configuration Files
- Set the Nagios Admin Password
- Set the Nagios Admin Email Contact Address
- Register the Hadoop Configuration Files
- Set Hosts
- Set Host Groups
- Set Services
- Set Status
- Add Templeton Status and Check TCP Wrapper Commands
- Validate the Installation
- 26. Installing Apache Slider
- 27. Setting Up Security for Manual Installs
- 28. Uninstalling HDP
List of Tables
- 1.1. Define Directories for Core Hadoop
- 1.2. Define Directories for Ecosystem Components
- 1.3. Define Users and Groups for Systems
- 1.4. Typical System Users and Groups
- 1.5. yarn-utils.py Options
- 1.6. Reserved Memory Recommendations
- 1.7. Recommended Values
- 1.8. YARN and MapReduce Configuration Setting Value Calculations
- 1.9. Example Value Calculations
- 1.10. Example Value Calculations
- 1.11. NameNode Heap Size Settings
- 8.1. Tez Configuration Parameters
- 9.1. Hive Configuration Parameters
- 11.1. Hadoop core-site.xml File Properties
- 13.1. install.properties Entries
- 13.2. Properties to Update in the install.properties File
- 13.3. HDFS-Related Properties to Edit in the install.properties File
- 13.4. HBase Properties to Edit in the install.properties File
- 13.5. Hive-Related Properties to Edit in the install.properties File
- 13.6. Knox-Related Properties to Edit in the install.properties File
- 13.7. Storm-Related Properties to Edit in the install.properties File
- 14.1. Hue-Supported Browsers
- 14.2. Hue Dependencies on HDP Components
- 14.3. Variables to Configure HDFS Cluster
- 14.4. Variables to Configure the YARN Cluster
- 14.5. Beeswax Configuration Values
- 17.1. Flume 1.5.2 Dependencies
- 18.1. Required jaas.conf Sections for Cluster Nodes
- 18.2. Supported Authorizers
- 18.3. storm.yaml Configuration File Properties
- 18.4. worker-launcher.cfg File Configuration Properties
- 18.5. multitenant-scheduler.yaml Configuration File Properties
- 19.1. Spark Cluster Prerequisites
- 20.1. Kafka Configuration Properties
- 25.1. Host Group Parameters
- 25.2. Core and Monitoring Host Groups
- 25.3. Ecosystem Project Host Groups
- 27.1. Service Principals
- 27.2. Service Keytab File Names
- 27.3. General core-site.xml, Knox, and Hue
- 27.4. core-site.xml Master Node Settings -- Knox Gateway
- 27.5. core-site.xml Master Node Settings -- Hue
- 27.6. hdfs-site.xml File Property Settings
- 27.7. yarn-site.xml Property Settings
- 27.8. mapred-site.xml Property Settings
- 27.9. hbase-site.xml Property Settings -- HBase Server
- 27.10. hive-site.xml Property Settings
- 27.11. oozie-site.xml Property Settings
- 27.12. webhcat-site.xml Property Settings