2. Meet Minimum System Requirements

To run the Hortonworks Data Platform, your system must meet minimum requirements.

 2.1. Hardware Recommendations

Although there is no single hardware requirement for installing HDP, there are some basic guidelines. You can see sample setups here: Hardware Recommendations for Apache Hadoop.

 2.2. Operating Systems Requirements

The following operating systems are supported:

  • 64-bit Red Hat Enterprise Linux (RHEL) 5 or 6

  • 64-bit CentOS 5 or 6

  • 64-bit SUSE Linux Enterprise Server (SLES) 11, SP1

 2.3. Software Requirements

On each of your hosts:

  • yum [for RHEL or CentOS]

  • zypper [for SLES]

  • rpm

  • scp [for multiple node installs]

  • curl

  • wget

  • unzip

  • tar

  • pdsh [for multiple node installs over many hosts]

 2.4. Configure the Remote Repository

The standard HDP install fetches the software from a remote yum repository over the Internet. To use this option, you must set up access to the remote repository and have an available Internet connection for each of your hosts.

[Note]Note

If your cluster does not have access to the Internet, or you are creating a large cluster and you want to conserve bandwidth, you can instead provide a local copy of the HDP repository that your hosts can access. For more information, see  Deployment Strategies for Data Centers with Firewalls., a separate document in this set.

  1. For each node in your cluster, download the repo configuration file hdp.repo and ambari.repo. From a terminal window, type:

    • For RHEL and CentOS 5

      wget -nv //public-repo-1.hortonworks.com/HDP/centos5/1.x/GA/1.3.0.0/hdp.repo  -O /etc/yum.repos.d/hdp.repo
       wget -nv //public-repo-1.hortonworks.com/ambari/centos5/1.x/updates/1.2.3.7/ambari.repo -O /etc/yum.repos.d/ambari.repo
      
    • For RHEL and CentOS 6

      wget -nv //public-repo-1.hortonworks.com/HDP/centos6/1.x/GA/1.3.0.0/hdp.repo -O /etc/yum.repos.d/hdp.repo
       wget -nv //public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.2.3.7/ambari.repo -O /etc/yum.repos.d/ambari.repo
      
    • For SLES

      wget -nv //public-repo-1.hortonworks.com/HDP/suse11/1.x/GA/1.3.0.0/hdp.repo -O /etc/zypp/repos.d/hdp.repo
       wget -nv //public-repo-1.hortonworks.com/ambari/suse11/1.x/updates/1.2.3.7/ambari.repo -O /etc/zypp/repos.d/ambari.repo
       
      
      
  2. Confirm the HDP repository is configured by checking the repo list.

    • For RHEL/CentOS

      yum repolist
    • For SLES

      zypper repos 

You should see something like this. Ensure you have HDP-1.3.0.0, HDP-UTILS-1.1.0.15, and AMBARI-1.2.3.7:

Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
* base: mirrors.cat.pdx.edu
* extras: linux.mirrors.es.net
* updates: mirrors.usc.edu
repo id                                   repo name                                                             status
AMBARI-1.2.3.7                            Ambari 1.2.3.7                                                        enabled:      6
HDP-1.3.0.0                               Hortonworks Data Platform Version - HDP-1.3.0                         enabled:     53
HDP-UTILS-1.1.0.15                        Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.15          enabled:     51

 2.5. Database Requirements

  • To use external database for Hive or Oozie metastore, ensure that a MySQL or Oracle or PostgreSQL database is deployed and available.

    (By default, Hive and Oozie use Derby database for the metastore.)

    • For instructions on deploying and/or configuring MySQL database instance, see here.

    • For instructions on configuring an existing Oracle database instance, see here.

      [Note]Note

      To deploy a new Oracle instance, consult your database administrator.

    • For instructions on deploying and/or configuring an existing PostgreSQL database instance, see here.

  • Ensure that your database administrator creates the following databases and users:

    • If deploying Hive:

      1. hive_dbname: Required if using MySQL database for Hive Metastore.

      2. hive_dbuser

      3. hive_dbpasswd

    • If deploying Oozie:

      1. oozie_dbname: Required if using MySQL database for Oozie Metastore.

      2. oozie_dbuser

      3. oozie_dbpasswd

 Instructions to setup MySQL database

  1. Connect to the host machine where you plan to deploy MySQL instance and from a terminal window, type:

    • For RHEL and CentOS:

      yum install mysql
    • For SLES:

      zypper install mysql
  2. Start the instance.

    • For RHEL and CentOS:

      /etc/init.d/mysqld start
    • For SLES:

      /etc/init.d/mysql start
  3. Set the root user password and remove unnecessary information from log and STDOUT.

    mysqladmin -u root password ‘$password’
    mysqladmin -u root 2>&1 >/dev/null
  4. Manually create users for MySQL.

    • As root, use mysql (or other client tool) to create the “dbuser” and grant it adequate privileges.

      (For access to Hive metastore, create hive_dbuser and for access to Oozie metastore, create oozie_dbuser.

      CREATE USER 'dbusername'@'%' IDENTIFIED BY 'dbuserpassword';
      GRANT ALL PRIVILEGES ON *.* TO 'dbusername'@'%';
      flush privileges;
    • See if you can connect to the database as that user. You are prompted to enter the dbuserpassword password above.

      mysql -u $dbusername -p

 Instructions to configure Oracle database

  • Ensure that the following SQL script is run against your Hive schema:

    $master-install-location/gsInstaller/confSupport/sql/oracle/hive-schema-0.10.0.oracle.sql

 Instructions to deploy and configure PostgreSQL database

  1. Connect to the host machine where you plan to deploy PostgreSQL instance and from a terminal window, type:

    • For RHEL and CentOS:

      yum install postgresql-server
    • For SLES:

      zypper install postgresql-server
  2. Start the instance.For RHEL and CentOS:

    /etc/init.d/postgresql start
    [Note]Note

    For some newer versions of PostgreSQL, you might need to execute the following command:

    /etc/init.d/postgresql initdb
  3. Reconfigure PostgreSQL server:

    1. Edit the /var/lib/pgsql/data/postgresql.conf file and change the value of #listen_addresses = 'localhost' to the following:

      listen_addresses = '*'
    2. Edit the /var/lib/pgsql/data/postgresql.conf file and change the port setting #port = 5432 to the following:

      port = 5432
    3. Edit the /var/lib/pgsql/data/pg_hba.conf and add the following:

      host all all 0.0.0.0/0 trust
    4. Optional - If you are using PostgreSQL v9.1 or later, add the following to the /var/lib/pgsql/data/postgresql.conf file:

      standard_conforming_strings = off

  4. Create users for PostgreSQL server:

    echo "CREATE DATABASE $dbname;" | psql -U postgres
    echo "CREATE USER $user WITH PASSWORD '$passwd';" | psql -U postgres
    echo "GRANT ALL PRIVILEGES ON DATABASE $dbname TO $user;" | psql -U postgres 
    [Note]Note

    For access to Hive metastore, create hive_dbuser and for access to Oozie metastore, create oozie_dbuser.

  5. Ensure that the following SQL script is run against your Hive schema:

    $master-install-location/gsInstaller/confSupport/sql/postgres/hive-schema-0.10.0.postgres.sql

    where $master-install-location is the master install location for Hive metastore host machine.

 2.6. JDK Requirements

Your system must have the correct JDK installed on all the nodes of the cluster. HDP requires Oracle JDK 1.6 update 31.

Use the following instructions to manually install JDK 1.6 update 31:

  1. Check the version. From a terminal window, type:

    java -version
  2. (Optional) Uninstall the Java package if the JDK version is less than v1.6 update 31.

    rpm -qa | grep java
    yum remove {java-1.*}
  3. (Optional) Verify that the default Java package is uninstalled.

    which java
  4. Download the Oracle 64-bit JDK (jdk-6u31-linux-x64.bin) from the Oracle download site:

    http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html#jdk-6u31-oth-JPR

    Accept the license agreement.

  5. Change directory to the location where you downloaded the JDK and run the install.

    mkdir /usr/jdk1.6.0_31
    cd /usr/jdk1.6.0_31
    chmod u+x $JDK_download_directory/jdk-6u31-linux-x64.bin
    $JDK_download_directory/jdk-6u31-linux-x64.bin 
    
  6. Create symbolic links (symlinks) to the JDK.

    mkdir /usr/java
    ln -s /usr/jdk1.6.0_31/jdk1.6.0_31 /usr/java/default
    ln -s /usr/java/default/bin/java /usr/bin/java
    
  7. Set up your environment to define JAVA_HOME to put the Java Virtual Machine and the Java compiler on your path.

    export JAVA_HOME=/usr/java/default
    export PATH=$JAVA_HOME/bin:$PATH
    

 2.7. Virtualization and Cloud Platforms

HDP is certified and supported when running on virtual or cloud platforms (for example, VMware vSphere or Amazon Web Services EC2) as long as the respective guest operating system (OS) is supported by HDP and any issues detected on these platforms are reproducible on the same supported OS installed on bare metal.

See Operating Systems Requirements for the list of supported operating systems for HDP.


loading table of contents...