1.6. Creating Service Principals and Keytab Files for Hadoop 1.x

Each service and sub-service in Hadoop must have its own principal. A principal name in a given realm consists of a primary name and an instance name, which in this case is the FQDN of the host that runs that service. As services do not login with a password to acquire their tickets, their principal's authentication credentials are stored in a keytab file, which is extracted from the Kerberos database and stored locally with the service principal on the service component host.

First you must create the principal, using mandatory naming conventions.

Then you must create the keytab file with that principal's information and copy the file to the keytab directory on the appropriate service host.

[Note]Note

Principals can be created either on the KDC machine itself or through the network, using an “admin” principal. The following instructions assume you are using the KDC machine and using the kadmin.local command line administration utility. Using kadmin.local on the KDC machine allows you to create principals without needing to create a separate "admin" principal before you start.

  1. Open the kadmin.local utility on the KDC machine

    /usr/sbin/kadmin.local
  2. Create the service principals:

    $kadmin.local
    addprinc -randkey $primary_name/$fully.qualified.domain.name@EXAMPLE.COM

    The -randkey option is used to generate the password.

    Note that in the example each service principal's primary name has appended to it the instance name, the FQDN of the host on which it runs. This provides a unique principal name for services that run on multiple hosts, like DataNodes and TaskTrackers. The addition of the host name serves to distinguish, for example, a request from DataNode A from a request from DataNode B. This is important for two reasons:

    • If the Kerberos credentials for one DataNode are compromised, it does not automatically lead to all DataNodes being compromised

    • If multiple DataNodes have exactly the same principal and are simultaneously connecting to the NameNode, and if the Kerberos authenticator being sent happens to have same time stamp, then the authentication would be rejected as a replay request.

    The principal name must match the values in the table below:

     

    Table 1.2. Service Principals

    ServiceComponentMandatory Principal Name

    HDFS

    NameNode

    nn/$FQDN

    HDFS

    NameNode HTTP

    HTTP/$FQDN

    HDFSSecondaryNameNodenn/$FQDN
    HDFSSecondaryNameNode HTTPHTTP/$FQDN

    HDFS

    DataNode

    dn/$FQDN

    MapReduceJobTrackerjt/$FQDN
    MapReduceTaskTrackertt/$FQDN

    Oozie

    Oozie Server

    oozie/$FQDN

    Oozie

    Oozie HTTP

    HTTP/$FQDN

    Hive

    Hive Metastore

    HiveServer2

    hive/$FQDN

    HiveWebHCat

    HTTP/$FQDN

    HBase

    MasterServer

    hbase/$FQDN

    HBase

    RegionServer

    hbase/$FQDN

    ZooKeeper

    ZooKeeper

    zookeeper/$FQDN

    Nagios ServerNagiosnagios/$FQDN

    For example: To create the principal for a DataNode service, issue this command:

    $kadmin.local
    addprinc -randkey dn/$DataNode-Host@EXAMPLE.COM 
  3.  In addition you must create four special principals for Ambari's own use.

    [Note]Note

    The names in table below can be customized in the Customize Services step of the Ambari Install Wizard. If this is the case in your installation, the principal names should match the customized names. For example, if the HDFS Service User has been set to hdfs1, the respective principal for the Ambari HDFS User should also be created as hdfs1.

    These principals do not need the FQDN appended to the primary name:

     

    Table 1.3. Ambari Principals

    UserDefault Principal Name
    Ambari User[a]ambari
    Ambari Smoke Test Userambari-qa
    Ambari HDFS Userhdfs
    Ambari HBase Userhbase

    [a] This principal is used with the JAAS configuration. See Setting Up JAAS for Ambari for more information.


  4. Once the principals are created in the database, you can extract the related keytab files for transfer to the appropriate host:

    $kadmin.local
    xst -norandkey -k $keytab_file_name $primary_name/fully.qualified.domain.name@EXAMPLE.COM
    [Note]Note

    Some older versions of Kerberos do not support the xst -norandkey option. You can use the command without the -norandkey flag, except in cases where you need to merge two principals with the same name into a single keytab file for a single host. In this case, you can use the two step kadmin/kutil procedure. This description assumes MIT Kerberos. If you are using another version, please check your documentation.

    1. Extract the keytab file information:

      $kadmin
      xst -k $keytab_file_name-temp1 $primary_name/fully.qualified.domain.name@EXAMPLE.COM
      xst -k $keytab_file_name-temp2 $primary_name/fully.qualified.domain.name@EXAMPLE.COM
    2. Merge the keytabs into a single file. :

       $kutil
      rkt $keytab_file_name-temp1
      rkt $keytab_file_name-temp2
      wkt $keytab_file_name 
      clear 

    You must use the mandatory names for the keytab_file_name variable shown in this table. Adjust the principal names if necessary.

     

    Table 1.4. Service Keytab File Names

    ComponentPrincipal NameMandatory Keytab File Name

    NameNode

    nn/$FQDNnn.service.keytab
    NameNode HTTPHTTP/$FQDNspnego.service.keytab

    SecondaryNameNode

    nn/$FQDN nn.service.keytab
    SecondaryNameNode HTTPHTTP/$FQDNspnego.service.keytab
    DataNodedn/$FQDNdn.service.keytab

    JobTracker

    jt/$FQDN

    jt.service.keytab

    TaskTracker

    tt/$FQDN

    tt.service.keytab

    Oozie Serveroozie/$FQDNoozie.service.keytab
    Oozie HTTPHTTP/$FQDNspnego.service.keytab

    Hive Metastore

    HiveServer2

    hive/$FQDN

    hive.service.keytab

    WebHCatHTTP/$FQDNspnego.service.keytab

    HBase Master Server

    hbase/$FQDN

    hbase.service.keytab

    HBase RegionServer

    hbase/$FQDN

    hbase.service.keytab

    ZooKeeper

    zookeeper/$FQDN

    zk.service.keytab

    Nagios Servernagios/$FQDNnagios.service.keytab
    Ambari User[a]ambariambari.keytab
    Ambari Smoke Test Userambari-qa smokeuser.headless.keytab
    Ambari HDFS Userhdfs hdfs.headless.keytab
    Ambari HBase Userhbasehbase.headless.keytab

    [a] This principal is used with the JAAS configuration. See Setting Up JAAS for Ambari for more information.


    For example: To create the keytab files for NameNode HTTP, issue this command:

    xst -norandkey -k spnego.service.keytab HTTP/<namenode-host>
    [Note]Note

    If you have a large cluster, you may want to create a script to automate creating your principals and keytabs. The Ambari Web GUI can help. See Create Principals and Keytabs for more information.

  5. When the keytab files have been created, on each host create a directory for them and set appropriate permissions.

    mkdir -p /etc/security/keytabs/
    chown root:hadoop /etc/security/keytabs
    chmod 750 /etc/security/keytabs
  6. Copy the appropriate keytab file to each host. If a host runs more than one component (for example, both TaskTracker and DataNode), copy keytabs for both components. The Ambari Smoke Test User, the Ambari HDFS User, and the Ambari HBase User keytabs should be copied to all hosts.

  7. Set appropriate permissions for the keytabs.

    [Important]Important

    If you have customized service user names, replace the default values below with your appropriate service user, group, and keytab names.

    1. Optionally, if you have Set up JAAS for Ambari on the Ambari server host:

      chown ambari:ambari /etc/security/keytabs/ambari.keytab
      chmod 400 /etc/security/keytabs/ambari.keytab 

    2. On the HDFS NameNode and SecondaryNameNode hosts:

      chown hdfs:hadoop /etc/security/keytabs/nn.service.keytab 
      chmod 400 /etc/security/keytabs/nn.service.keytab
      chown root:hadoop /etc/security/keytabs/spnego.service.keytab 
      chmod 440 /etc/security/keytabs/spnego.service.keytab

    3. On the HDFS NameNode host, for the Ambari Test Users:

      chown ambari-qa:hadoop /etc/security/keytabs/smokeuser.headless.keytab
      chmod 440 /etc/security/keytabs/smokeuser.headless.keytab
      chown hdfs:hadoop /etc/security/keytabs/hdfs.headless.keytab
      chmod 440 /etc/security/keytabs/hdfs.headless.keytab
      chown hbase:hadoop /etc/security/keytabs/hbase.headless.keytab
      chmod 440 /etc/security/keytabs/hbase.headless.keytab
    4. On each host that runs an HDFS DataNode:

      chown hdfs:hadoop /etc/security/keytabs/dn.service.keytab 
      chmod 400 /etc/security/keytabs/dn.service.keytab

    5. On the host that runs the MapReduce JobTracker:

      chown mapred:hadoop /etc/security/keytabs/jt.service.keytab 
      chmod 400 /etc/security/keytabs/jt.service.keytab
    6. On each host that runs a MapReduce TaskTracker:

      chown mapred:hadoop /etc/security/keytabs/tt.service.keytab 
      chmod 400 /etc/security/keytabs/tt.service.keytab
    7. On the host that runs the Oozie Server:

      chown oozie:hadoop /etc/security/keytabs/oozie.service.keytab 
      chmod 400 /etc/security/keytabs/oozie.service.keytab
      chown root:hadoop /etc/security/keytabs/spnego.service.keytab 
      chmod 440 /etc/security/keytabs/spnego.service.keytab
    8. On the host that runs the Hive Metastore, HiveServer2 and WebHCat:

      chown hive:hadoop /etc/security/keytabs/hive.service.keytab 
      chmod 400 /etc/security/keytabs/hive.service.keytab
      chown root:hadoop /etc/security/keytabs/spnego.service.keytab 
      chmod 440 /etc/security/keytabs/spnego.service.keytab
    9. On hosts that run the HBase MasterServer, RegionServer and ZooKeeper:

      chown hbase:hadoop /etc/security/keytabs/hbase.service.keytab 
      chmod 400 /etc/security/keytabs/hbase.service.keytab
      chown zookeeper:hadoop /etc/security/keytabs/zk.service.keytab 
      chmod 400 /etc/security/keytabs/zk.service.keytab
    10. On the host that runs the Nagios server:

      chown nagios:nagios /etc/security/keytabs/nagios.service.keytab
      chmod 400 /etc/security/keytabs/nagios.service.keytab

  8. Verify that the correct keytab files and principals are associated with the correct service using the klist command. For example, on the NameNode:

    klist -k -t /etc/security/keytabs/nn.service.keytab

    Do this on each respective service in your cluster.