3.2.2. Install the Repos

  • Temporarily reconfigure your firewall to allow Internet access from your mirror server host.

  • Execute the following command to download the appropriate Hortonworks yum client configuration file and save it in /etc/yum.repos.d/ directory on the mirror server host.

    Table 4.6. Deploying HDP - Option II
    Cluster OS HDP Repository Tarballs

    RHEL/CentOS/Oracle Linux 5.x

    wget http://public-repo-1.hortonworks.com/HDP/centos5/2.x/updates/2.0.6.0/hdp.repo -O /etc/yum.repos.d/hdp.repo

    wget http://public-repo-1.hortonworks.com/ambari/centos5/1.x/updates/1.4.1.25/ambari.repo -O /etc/yum.repos.d/ambari.repo

    RHEL/CentOS/Oracle Linux 6.x

    wget http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.0.6.0/hdp.repo -O /etc/yum.repos.d/hdp.repo

    wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.4.1.25/ambari.repo -O /etc/yum.repos.d/ambari.repo

    SLES 11

    wget http://public-repo-1.hortonworks.com/HDP/suse11/2.x/updates/2.0.6.0/hdp.repo -O /etc/zypp/hdp.repo

    wget http://public-repo-1.hortonworks.com/ambari/suse11/1.x/updates/1.4.1.25/ambari.repo -O /etc/zypp/ambari.repo

    Ubuntu 12.04

    wget http://public-repo-1.hortonworks.com/HDP/ubuntu12/2.x/hdp.list -O /etc/apt/sources.list.d/hdp.list

  • Create an HTTP server.

    1. On the mirror server, install an HTTP server (such as Apache httpd) using the instructions provided http://httpd.apache.org/download.cgi

    2. Activate this web server.

    3. Ensure that the firewall settings (if any) allow inbound HTTP access from your cluster nodes to your mirror server.

      [Note]Note

      If you are using EC2, make sure that SELinux is disabled.

    4. Optional - If your mirror server uses SLES, modify the default-server.conf file to enable the docs root folder listing.

      sed -e "s/Options None/Options Indexes MultiViews/ig" /etc/apache2/default-server.conf > /tmp/tempfile.tmp
      mv /tmp/tempfile.tmp /etc/apache2/default-server.conf

  • On your mirror server, create a directory for your web server.

    • For example, from a shell window, type:

      • For RHEL/CentOS/Oracle:

        mkdir –p /var/www/html/hdp/
      • For SLES:

        mkdir –p /srv/www/htdocs/rpms 
      • For Ubuntu:

        mkdir –p /var/www/html/hdp/

    • If you are using a symlink, enable the followsymlinks on your web server.

  • Copy the contents of entire HDP repository for your desired OS from the remote yum server to your local mirror server.

    • Continuing the previous example, from a shell window, type:

      • For RHEL/CentOS/Oracle:

        cd /var/www/html/hdp
      • For SLES:

        cd /srv/www/htdocs/rpms

      Then for all hosts, type:

      • HDP Repository

        reposync -r HDP-2.x
        reposync -r HDP-2.0.6.0-76
        reposync -r HDP-UTILS-1.1.0.16

        You should see both an HDP-2.0.6.0 directory and an HDP-UTILS-1.1.0.16 directory, each with several subdirectories.

      • Optional - Ambari Repository

        reposync –r ambari-1.x
        reposync –r $release_type-ambari-1.4.1.25
  • Generate appropriate metadata.

    This step defines each directory as a yum repository. From a shell window, type:

    • For RHEL/CentOS/Oracle:

      • HDP Repository:

        createrepo /var/www/html/hdp/HDP-2.0.6.0
        createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.16
      • Optional - Ambari Repository:

        createrepo /var/www/html/hdp/ambari-1.x
        createrepo /var/www/html/hdp/$release_type-ambari-1.4.1.25

    • For SLES:

      • HDP Repository:

        createrepo /srv/www/htdocs/rpms/hdp/HDP 

      • Optional - Ambari Repository:

        createrepo /srv/www/htdocs/rpms/hdp/ambari-1.x
        createrepo /srv/www/htdocs/rpms/hdp/$release_type-ambari-1.4.1.25 

    You should see a new folder called repodata inside both HDP directories.

  • Verify the configuration.

    • The configuration is successful, if you can access the above directory through your web browser.

      To test this out, browse to the following location:

      • HDP: http://$yourwebserver/hdp/HDP-2.0.6.0/

      • Optional - Ambari Repository: http://$yourwebserver/hdp/ambari/$os/1.x/updates/1.4.1.25

    • You should now see directory listing for all the HDP components.

  • At this point, you can disable external Internet access for the mirror server, so that the mirror server is again entirely within your data center firewall.

  • Depending on your cluster OS, configure the yum clients on all the nodes in your cluster

    1. Edit the repo files, changing the value of the baseurl property to the local mirror URL.

      • Edit the /etc/yum.repos.d/hdp.repo file, changing the value of the baseurl property to point to your local repositories based on your cluster OS.

        [HDP-2.x]
        name=Hortonworks Data Platform Version - HDP-2.x
        baseurl=http://$yourwebserver/HDP/$os/2.x/GA 
        gpgcheck=1
        gpgkey=http://public-repo-1.hortonworks.com/HDP/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
        enabled=1
        priority=1
        
        [HDP-UTILS-1.1.0.16]
        name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.16
        baseurl=http://$yourwebserver/HDP-UTILS-1.1.0.16/repos/$os
        gpgcheck=1
        gpgkey=http://public-repo-1.hortonworks.com/HDP/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
        enabled=1
        priority=1
        
        [HDP-2.0.6.0]
        name=Hortonworks Data Platform HDP-2.0.6.0
        baseurl=http://$yourwebserver/HDP/$os/2.x/updates/2.0.6.0
        gpgcheck=1
        gpgkey=http://public-repo-1.hortonworks.com/HDP/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
        enabled=1
        priority=1

        where

        • $yourwebserver is FQDN of your local mirror server.

        • $os can be centos5, centos6, or suse11. Use the following options table for $os parameter:

           

          Table 4.7. Options for $os parameter in repo URL

          Operating SystemValue

          CentOS 5

          centos5

          RHEL 5
          Oracle Linux 5

          CentOS 6

          centos6

          RHEL 6
          Oracle Linux 6

          SLES 11

          suse11

          Ubuntu 12

          ubuntu12


      • Edit the /etc/yum.repos.d/ambari.repo file, changing the value of the baseurl property to point to your local repositories based on your cluster OS.

        [ambari-1.x]
        name=Ambari 1.x
        baseurl=http://$yourwebserver/hdp/ambari/$os/1.x/updates/ambari.repo
        gpgcheck=0
        gpgkey=http://public-repo-1.hortonworks.com/ambari/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
        enabled=1
        priority=1
        
        [HDP-UTILS-1.1.0.16]
        name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.16
        baseurl=http://$yourwebserver/HDP-UTILS-1.1.0.16/repos/$os
        gpgcheck=0
        gpgkey=http://public-repo-1.hortonworks.com/ambari/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
        enabled=1
        priority=1
        
        [$release_type-ambari-1.4.1.25
        name=ambari-1.4.1.25 - updates
        baseurl=http://$yourwebserver/ambari/$os/1.x/updates/1.4.1.25
        gpgcheck=0
        gpgkey=http://public-repo-1.hortonworks.com/ambari/$os/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
        enabled=1
        priority=1   
                                                 
        • $yourwebserver is FQDN of your local mirror server.

        • $os can be centos5, centos6, or suse11. Use the following options table for $os parameter:

           

          Table 4.8. Options for $os parameter in repo URL

          Operating SystemValue

          CentOS 5

          centos5

          RHEL 5
          Oracle Linux 5

          CentOS 6

          centos6

          RHEL 6
          Oracle Linux 6

          SLES 11

          suse11

          Ubuntu 12

          ubuntu12


    2. Copy the yum/zypper client configuration file to all nodes in your cluster.

      • RHEL/CentOS/Oracle Linux:

        Use scp or pdsh to copy the client yum configuration file to /etc/yum.repos.d/ directory on every node in the cluster.

      • For SLES:

        On every node, invoke the following command:

        • HDP Repository: zypper addrepo -r http://$yourwebserver/hdp/HDP/suse11/2.x/updates/2.0.6.0/hdp.repo

        • Optional - Ambari Repository: zypper addrepo -r http://$yourwebserver/hdp/ambari/suse11/1.x/updates/1.4.1.25/ambari.repo

        • If using Ambari, verify the configuration by deploying Ambari server on one of the cluster nodes. yum install ambari-server

      • For Ubuntu:

        On every node, invoke the following command:

        • HDP Repository: sudo add-apt-repository 'deb http://$yourwebserver/hdp/HDP/ubuntu12/2.x/hdp.list'

  • If your cluster runs CentOS, Oracle, or RHEL and if you have multiple repositories configured in your environment, deploy the following plugin on all the nodes in your cluster.

    1. Install the plugin.

      • For RHEL and CentOs v5.x

        yum install yum-priorities
      • For RHEL and CentOs v6.x

        yum install yum-plugin-priorities
    2. Edit the /etc/yum/pluginconf.d/priorities.conf file to add the following:

      [main]
      enabled=1
      gpgcheck=0