Hortonworks Data Platform

Installing HDP Manually

2014-04-22


Contents

1. Getting Ready to Install
1. Meet Minimum System Requirements
1.1. Hardware Recommendations
1.2. Operating Systems Requirements
1.3. Software Requirements
1.4. Metastore Database Requirements
1.5. JDK Requirements
1.6. Virtualization and Cloud Platforms
2. Configure the Remote Repositories
3. Decide on Deployment Type
4. Collect Information
5. Prepare the Environment
5.1. Enable NTP on the Cluster
5.2. Check DNS
5.3. Disable SELinux
5.4. Disable IPTables
6. Download Companion Files
7. Define Environment Parameters
8. [Optional] Create System Users and Groups
9. Determine HDP Memory Configuration Settings
9.1. Use the HDP Utility Script to Calculate Memory Configuration Settings
9.2. Manually Calculate YARN and MapReduce Memory Configuration Settings
10. Allocate Adequate Log Space for HDP
2. Installing HDFS and YARN
1. Set Default File and Directory Permissions
2. Install the Hadoop Packages
3. Install Compression Libraries
3.1. Install Snappy
3.2. Install LZO
4. Create Directories
4.1. Create the NameNode Directories
4.2. Create the SecondaryNameNode Directories
4.3. Create DataNode and YARN NodeManager Local Directories
4.4. Create the Log and PID Directories
3. Setting Up the Hadoop Configuration
4. Validating the Core Hadoop Installation
1. Format and Start HDFS
2. Smoke Test HDFS
3. Start YARN
4. Start MapReduce JobHistory Server
5. Smoke Test MapReduce
5. Installing Zookeeper
1. Install the ZooKeeper RPMs
2. Set Directories and Permissions
3. Set Up the Configuration Files
4. Start ZooKeeper
6. Installing HBase
1. Install the HBase RPMs
2. Set Directories and Permissions
3. Set Up the Configuration Files
4. Validate the Installation
5. Starting the HBase Thrift and REST APIs
7. Installing Phoenix
8. Installing Apache Pig
1. Install the Pig RPMs
2. Set Up Configuration Files
3. Validate the Installation
9. Installing Apache Hive and Apache HCatalog
1. Install the Hive and HCatalog RPMs
2. Set Directories and Permissions
3. Set Up the Hive/HCatalog Configuration Files
3.1. Configure Hive and HiveServer2 for Tez
4. Create Directories on HDFS
5. Validate the Installation
10. Installing and Configuring Apache Tez
1. Install the Tez RPM
2. Configure Tez
2.1. Tez Configuration
2.2. Tez Configuration Parameters
2.3. Configuring Tez with the Capacity Scheduler
3. Validate the Tez Installation
4. Enable Tez for Hive Queries
5. Validate Hive-on-Tez Installation
6. Troubleshooting
11. Installing WebHCat
1. Install the WebHCat RPMs
2. Set Directories and Permissions
3. Modify WebHCat Configuration Files
4. Set Up HDFS User and Prepare WebHCat Directories On HDFS
5. Validate the Installation
12. Installing Apache Oozie
1. Install the Oozie RPMs
2. Set Directories and Permissions
3. Set Up the Oozie Configuration Files
4. Validate the Installation
13. Installing Hue
1. Prerequisites
2. Configure HDP
3. Install Hue
4. Configure Hue
4.1. Configure Web Server
4.2. Configure Hadoop
4.3. Configure Beeswax
4.4. Configure JobDesigner and Oozie
4.5. Configure UserAdmin
4.6. Configure WebHCat
5. Start Hue
6. Validate Configuration
14. Installing Apache Sqoop
1. Install the Sqoop RPMs
2. Set Up the Sqoop Configuration
3. Validate the Installation
15. Installing Mahout
16. Installing and Configuring Flume in HDP
1. Understand Flume
1.1. Flume Components
2. Install Flume
3. Prerequisites
4. Installation
5. Directories
6. Configure Flume
7. Start Flume
8. HDP and Flume
8.1. Sources
8.2. Channels
8.3. Sinks
9. A Simple Example
17. Installing and Configuring Storm
1. Install the Storm RPMs
2. Configure Storm
3. Configure Process Controller
4. Validate the Installation
18. Installing Accumulo
1. Install the Accumulo RPM
2. Configure Accumulo
3. Validate Accumulo
19. Installing Falcon
1. Install the Falcon RPM
2. Configuring Falcon Entities
3. Configuring Oozie for Falcon
4. Configuring Hive for Falcon
5. Configuring for Secure Clusters
6. Validate Falcon
20. Installing Knox
1. Install the Knox RPMs on the Knox server
2. Set up and Validate the Knox Gateway Installation
21. Installing Ganglia
1. Install the Ganglia RPMs
2. Install the Configuration Files
2.1. Extract the Ganglia Configuration Files
2.2. Copy the Configuration Files
2.3. Set Up Ganglia Hosts
2.4. Set Up Configurations
2.5. Set Up Hadoop Metrics
3. Validate the Installation
3.1. Start the Ganglia Server
3.2. Start Ganglia Monitoring on All Hosts
3.3. Confirm that Ganglia is Running
22. Installing Nagios
1. Install the Nagios RPMs
2. Install the Configuration Files
2.1. Extract the Nagios Configuration Files
2.2. Create the Nagios Directories
2.3. Copy the Configuration Files
2.4. Set the Nagios Admin Password
2.5. Set the Nagios Admin Email Contact Address
2.6. Register the Hadoop Configuration Files
2.7. Set Hosts
2.8. Set Host Groups
2.9. Set Services
2.10. Set Status
2.11. Add Templeton Status and Check TCP Wrapper Commands
3. Validate the Installation
3.1. Validate the Nagios Installation
3.2. Start Nagios and httpd
3.3. Confirm Nagios is Running
3.4. Test Nagios Services
3.5. Test Nagios Access
3.6. Test Nagios Alerts
23. Setting Up Security for Manual Installs
1. Preparing Kerberos
1.1. Kerberos Overview
1.2. Installing and Configuring the KDC
1.3. Creating the Database and Setting Up the First Administrator
1.4. Creating Service Principals and Keytab Files for HDP
2. Configuring HDP
2.1. Configuration Overview
2.2. Creating Mappings Between Principals and UNIX Usernames
2.3. Adding Security Information to Configuration Files
3. Configure secure HBase and ZooKeeper
3.1. Configure HBase Master
3.2. Create JAAS configuration files
3.3. Start HBase and ZooKeeper services
3.4. Configure secure client side access for HBase
3.5. Optional: Configure client-side operation for secure operation - Thrift Gateway
3.6. Optional: Configure client-side operation for secure operation - REST Gateway
3.7. Configure HBase for Access Control Lists (ACL)
4. Setting up One-Way Trust with Active Directory
4.1. Configure Kerberos Hadoop Realm on the AD DC
4.2. Configure the AD Domain on the KDC and Hadoop Cluster Hosts
24. Upgrade from HDP 1.3 to HDP 2.1 Manually
1. Getting Ready to Upgrade
2. Upgrade Hadoop
3. Migrate the HDP Configurations
4. Create Local Directories
5. Start HDFS
5.1. Verify HDFS filesystem health
5.2. Create HDFS Directories
5.3. Start YARN/MapReduce Services
5.4. Run Hadoop Smoke Tests
6. Upgrade ZooKeeper
7. Upgrade HBase
8. Upgrade Hive and HCatalog
9. Upgrade Oozie
10. Upgrade WebHCat (Templeton)
11. Upgrade Pig
12. Upgrade Sqoop
13. Upgrade Flume
13.1. Validate Flume
14. Upgrade Mahout
14.1. Mahout Validation
15. Upgrade Hue
16. Finalize Upgrade
17. Install New HDP 2.1 Services
25. Upgrade from HDP 2.0 to HDP 2.1 Manually
1. Getting Ready to Upgrade
2. Upgrade Hadoop
3. Start HDFS
3.1. Verify HDFS filesystem health
3.2. Start YARN/MapReduce Services
3.3. Run Hadoop Smoke Tests
4. Upgrade ZooKeeper
5. Upgrade HBase
6. Upgrade Hive and HCatalog
7. Upgrade Oozie
8. Upgrade WebHCat (Templeton)
9. Upgrade Pig
10. Upgrade Sqoop
11. Upgrade Flume
11.1. Validate Flume
12. Upgrade Mahout
12.1. Mahout Validation
13. Upgrade Hue
14. Finalize Upgrade
15. Install New HDP 2.1 Services
26. Uninstalling HDP

loading table of contents...