Configuring Fault Tolerance
Also available as:
PDF
loading table of contents...

Configuring Ranger Admin HA With SSL

Set up a load-balancer with SSL and enable Ranger Admin HA on an Ambari-managed cluster.

  1. Use SSH to connect to the cluster node where you will set up the load-balancer. In this procedure, we use the IP address 172.22.71.37.
  2. Use the following command to switch to the /usr/local directory:
    cd /usr/local
  3. Download the httpd file and its dependencies (apr and apr-util):
    wget https://archive.apache.org/dist/httpd/httpd-2.4.16.tar.gz
    wget https://archive.apache.org/dist/apr/apr-1.5.2.tar.gz 
    wget https://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz
  4. Extract the contents of these files:
    tar -xvf httpd-2.4.16.tar.gz
    tar -xvf apr-1.5.2.tar.gz 
    tar -xvf apr-util-1.5.4.tar.gz
  5. Run the following commands to move apr and apr-util to the srclib directory under httpd:
    mv apr-1.5.2/ apr
    mv apr httpd-2.4.16/srclib/ 
    mv apr-util-1.5.4/ apr-util
    mv apr-util httpd-2.4.16/srclib/
  6. Install the required packages:
    yum groupinstall "Development Tools"
    yum install openssl-devel
    yum install pcre-devel
  7. Run the following commands to configure the source tree:
    cd /usr/local/httpd-2.4.16
    ./configure --enable-so --enable-ssl --with-mpm=prefork --with-included-apr
  8. Run the following command to make the build:
    make
  9. Run the install:
    make install
  10. Run the following commands to confirm the preceding configuration steps:
    cd /usr/local/apache2/bin
    ./apachectl start
    curl localhost

    This should return:

    <html><body><h1>It works!</h1></body></html>
  11. Run the following commands to create a backup conf file.
    cd /usr/local/apache2/conf
    cp httpd.conf ~/httpd.conf.backup
  12. Edit the httpd.conf file:
    vi /usr/local/apache2/conf/httpd.conf

    Make the following updates:

    • If you are not running the load-balancer on the default port 80, change the default listening port in line Listen 80 to match the port setting.

    • Un-comment the following module entries (remove the # symbol at the beginning of each line):

      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_http_module modules/mod_proxy_http.so
      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so            
      LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
      LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
      LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
      LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
      LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
      LoadModule ssl_module modules/mod_ssl.so
      Note
      Note

      If LoadModule ssl_module modules/mod_ssl.so is not available in the httpd.conf file, check to make sure that you performed all of the previous installation steps. The load balancer will not work properly without the SSL module.

    • Update the ServerAdmin email address, or comment out that line.

      #ServerAdmin you@example.com
  13. Run the following command to restart the httpd server:
    /usr/local/apache2/bin/apachectl restart

    You should now be able to use Curl or a browser to access the load-balancer server IP address (with the port configured in the httpd.conf file) using the HTTPS protocol.

  14. Enable Ranger Admin HA using Ambari. On the Ambari dashboard of the first Ranger host, select Services > Ranger, then select Service Actions > Enable Ranger Admin HA to launch the Enable Ranger Admin HA Wizard.
  15. On the Get Started page, enter the load-balancer URL and port number (in this example, 172.22.71.37:88), then click Next.
  16. On the Select Hosts page, confirm the host assignments, then click Next.
  17. Check the settings on the Review page, then click Next.
  18. Click Complete on the Install, Start, and Test page to complete the installation.
  19. When the installation is complete, the Ranger Admin instances are listed on the Ranger Summary page. Select Actions > Restart All Required to restart all services that require a restart.
    Note
    Note

    For Oracle, clear the Setup DB and DB user check box under "Advanced ranger-env" to avoid DB setup.

  20. To test the load-balancer and Ranger HA configuration, select Ranger > Service Actions > Stop on one of the Ranger hosts.
  21. Use a browser to check the load-balancer host URL (with port). You should see the Ranger Admin page.
  22. Use the following steps to generate the self-signed certificate:
    1. Switch to the directory that will contain the self-signed certificate:
      cd /tmp
    2. Generate the private key:
      openssl genrsa -out server.key 2048
    3. Generate the CSR:
      openssl req -new -key server.key -out server.csr
    4. Generate the self-signed key:
      openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
    5. Generate the keystore in PEM format:
      openssl pkcs12 -export -passout pass:ranger -in server.crt -inkey server.key -out lbkeystore.p12 -name httpd.lb.server.alias
    6. Use the keytool to convert the PEM format keystore to JKS format:
      keytool -importkeystore -deststorepass ranger -destkeypass ranger -destkeystore httpd_lb_keystore.jks -srckeystore lbkeystore.p12 -srcstoretype PKCS12 -srcstorepass ranger -alias httpd.lb.server.alias
    7. Create a truststore of the load-balancer self-signed keystore:
      keytool -export -keystore httpd_lb_keystore.jks -alias httpd.lb.server.alias -file httpd-lb-trust.cer
  23. Copy the generated key and certificate into the /usr/local/apache2/conf/ directory.
    cp server.crt /usr/local/apache2/conf/
    cp server.key /usr/local/apache2/conf/
  24. Add the following entry at the end of the /usr/local/apache2/conf/httpd.conf file to read the custom configuration file:
    Include /usr/local/apache2/conf/ranger-lb-ssl.conf
  25. Create a custom conf file for the load-balancer SSL configuration:
    vi /usr/local/apache2/conf/ranger-lb-ssl.conf

    Make the following updates:

    Add the following lines, then change the <VirtualHost *:8443> port to match the default port you set previously in the httpd.conf file.

    <VirtualHost *:8443>
    
            SSLEngine On
            SSLProxyEngine On
            SSLCertificateFile /usr/local/apache2/conf/server.crt
            SSLCertificateKeyFile /usr/local/apache2/conf/server.key
    
            #SSLCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem
            #SSLProxyCACertificateFile /usr/local/apache2/conf/ranger_lb_crt.pem
            SSLVerifyClient optional
            SSLOptions +ExportCertData
            SSLProxyVerify none
            SSLProxyCheckPeerCN off
            SSLProxyCheckPeerName off
            SSLProxyCheckPeerExpire off
            ProxyRequests off
            ProxyPreserveHost off
    
            Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
    
            <Proxy balancer://rangercluster>
                   BalancerMember http://172.22.71.39:6080 loadfactor=1 route=1
                   BalancerMember http://172.22.71.38:6080 loadfactor=1 route=2
            
    
                    Order Deny,Allow
                    Deny from none
                    Allow from all
    
                    ProxySet lbmethod=byrequests scolonpathdelim=On stickysession=ROUTEID maxattempts=1 failonstatus=500,501,502,503 nofailover=Off
            </Proxy>
    
            # balancer-manager
            # This tool is built into the mod_proxy_balancer
            # module and will allow you to do some simple
            # modifications to the balanced group via a gui
            # web interface.
            <Location /balancer-manager>
                    SetHandler balancer-manager
                    Order deny,allow
                    Allow from all
            </Location>
    
    
           ProxyPass /balancer-manager !
           ProxyPass / balancer://rangercluster/
           ProxyPassReverse / balancer://rangercluster/
    
    </VirtualHost>
    Note
    Note

    The URLs listed in the BalancerMember entries are the IP addresses of the Ranger Admin hosts. In this example, the Ranger Admin host adresses are:

    http://172.22.71.38:6080
    http://172.22.71.39:6080
  26. Run the following command to restart the httpd server:
    /usr/local/apache2/bin/apachectl restart

    If you use a browser to check the load-balancer host (with port), you should see the Ranger Admin page.

  27. Run the following command to enable Usersync to communicate with Ranger via the load-balancer. This command copies the previously generated truststore file from the /tmp directory imports the certificate into the Usersync truststore.
    keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/ranger/usersync/conf/mytruststore.jks -storepass changeit
  28. Restart Ranger Usersync.
  29. Run the following command to enable the HDFS plug-in to communicate with Ranger via the load-balancer. This command copies the previously generated truststore file from the /tmp directory imports the certificate into the HDFS truststore.
    keytool -import -file /tmp/httpd-lb-trust.cer -alias httpd.lb.server.alias -keystore /etc/hadoop/conf/ranger-plugin-truststore.jks -storepass changeit
  30. Restart HDFS.
  31. In the Ranger Admin UI, select Audit > Plugins. You should see an entry for your repo name with HTTP Response Code 200.
  32. Use SSH to connect to the KDC server host. Use the kadmin.local command to access the Kerberos CLI, then check the list of principals for each domain where Ranger Admin and the load-balancer are installed.
    kadmin.local
    kadmin.local: list_principals

    For example, if Ranger Admin is installed on <host1> and <host2>, and the load-balancer is installed on <host3>, the list returned should include the following entries:

    HTTP/ <host3>@EXAMPLE.COM
    HTTP/ <host2>@EXAMPLE.COM
    HTTP/ <host1>@EXAMPLE.COM

    If the HTTP principal for any of these hosts is not listed, use the following command to add the principal:

    kadmin.local: addprinc -randkey HTTP/<host3>@EXAMPLE.COM
    Note
    Note

    This step will need to be performed each time the Spnego keytab is regenerated.

  33. Use the following kadmin.local commands to add the HTTP Principal of each of the Ranger Admin and load-balancer nodes to the Spnego keytab file:
    kadmin.local: ktadd -norandkey -kt /etc/security/keytabs/spnego.service.keytab HTTP/ <host3>@EXAMPLE.COM
    kadmin.local: ktadd -norandkey -kt /etc/security/keytabs/spnego.service.keytab HTTP/ <host2>@EXAMPLE.COM
    kadmin.local: ktadd -norandkey -kt /etc/security/keytabs/spnego.service.keytab HTTP/ <host1>@EXAMPLE.COM

    Use the exit command to exit ktadmin.local.

  34. Run the following command to check the Spnego keytab file:
    klist -kt /etc/security/keytabs/spnego.service.keytab

    The output should include the principals of all of the nodes on which Ranger Admin and the load-balancer are installed. For example:

    Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
    KVNO Timestamp     	Principal
    ---- ----------------- --------------------------------------------------------
       1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM
       1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM
       1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM
       1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM
       1 07/22/16 06:27:31 HTTP/ <host3>@EXAMPLE.COM
       1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM
       1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM
       1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM
       1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM
       1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM
       1 07/22/16 08:37:23 HTTP/ <host2>@EXAMPLE.COM
       1 07/22/16 08:37:35 HTTP/ <host1>@EXAMPLE.COM
       1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM
       1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM
       1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM
       1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM
       1 07/22/16 08:37:36 HTTP/ <host1>@EXAMPLE.COM
  35. Use scp to copy the Spnego keytab file to every node in the cluster on which Ranger Admin and the load-balancer are installed. Verify that the /etc/security/keytabs/spnego.service.keytab file is present on all Ranger Admin and load-balancer hosts.
  36. Configure the Ranger HA Keytab file.

    When setting up Ranger in HA where Kerberos is enabled, the Ambari-managed SPNEGO keytab file is altered. Since Ambari expects the file to contain certain data, the file is now overwritten with the Ambari-cached data during some Kerberos operations. This breaks Ranger in HA.

    In HDP 3.0+, you must duplicate the SPNEGO keytab file for Ranger HA, make various changes, and add the custom property "ranger.ha.spnego.kerberos.keytab". This enables Ambari to use the relevant SPNEGO properties differently depending on whether the Ambari server or the Ambari agent is acting.

    1. Login to the load balancer node.
    2. Enter: cp /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/ranger.ha.keytab.
    3. Run kadmin.local.
    4. Add the SPNEGO principal entry of the node where the first ranger_admin is installed: ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/as-amb-21-1.openstacklocal@EXAMPLE.COM.
    5. Add the SPNEGO principal entry of the node where the second ranger_admin is installed: ktadd -norandkey -kt /etc/security/keytabs/ranger.ha.keytab HTTP/as-amb-21-1.openstacklocal@EXAMPLE.COM.
    6. Verify /etc/security/keytabs/ranger.ha.keytab contains an entry of all the required SPNEGO principals: klist -kt /etc/security/keytabs/ranger.ha.keytab.
    7. Copy the Ranger keytab file to other nodes where ranger_admin is installed: scp /etc/security/keytabs/ranger.ha.keytab.
    8. Update permission: chmod 440 /etc/security/keytabs/ranger.ha.keytab.
    9. Update ownership: chown root:hadoop /etc/security/keytabs/ranger.ha.keytab.
    10. Add config from Ambari > Ranger > Configs > Advanced > Custom ranger-admin-site: ranger.ha.spnego.kerberos.keytab=/etc/security/keytabs/ranger.ha.keytab.
    11. Restart the Ranger service.