Security
Also available as:
PDF
loading table of contents...

Configuring Knox for NiFi

Creating Knox Certificates using the TLS Toolkit

About This Task

You must create Knox certificates so that Knox and NiFi can use two-way SSL communication to achieve a secure proxy relationship.

Steps

  1. As the Knox user, create a nifi-ca-config.json file on each Knox node, in a location accessible to Knox.

    For example, to create the file on a Knox node at /home/knox using the vi text editor, enter the following:

    sudo su - knox
    vi /home/knox/nifi-ca-config.json 
  2. Populate the nifi-ca-config.json file with the following information:

    {
      "dn" : "CN=$KNOX_HOSTNAME, OU=KNOX",
      "keyStore" : "/home/knox/knox-nifi-keystore.jks",
      "keyStoreType" : "jks",
      "keyStorePassword" : "$KEYSTORE_PASSWORD",
      "keyPassword" : "$KEY_PASSWORD",
      "token" : "$NIFI_CA_TOKEN_VALUE",
      "caHostname" : "$NIFI_CA_HOSTNAME",
      "port" : $NIFI_CA_PORT,
      "trustStore" : "/home/knox/knox-nifi-truststore.jks",
      "trustStorePassword" : "$TRUSTSTORE_PASSWORD",
      "trustStoreType" : "jks"
    }
    

    • $KNOX_HOSTNAME is the FQDN for the cluster node on which Knox is installed.

    • $KEYSTORE_PASSWORD is the password you want the TLS Toolkit to use for the KeyStore and TrustStore for Knox when communicating with NiFi.

    • $KEY_PASSWORD is required when SSL is enabled. If you are using the NiFi Certificate Authority (CA), this value is automatically generated if the field is left blank. You can find this value in the Key password field in the NiFi Advanced nifi-ambari-ssl-config configuration section.

    • $NIFI_CA_TOKEN_VALUE is the token that NiFi CA uses to verify a NiFi node identity before issuing certificates. You can find this value in the NiFi CA Token field in the NiFi Advanced nifi-ambari-ssl-config configuration section.

    • $NIFI_CA_HOSTNAME is the FQDN for the cluster node on which the NiFi CA is installed.

    • $NIFI_CA_PORT is the port used by the NiFi CA.

      You can find this value in the NiFi Certificate Authority port field in the NiFi Advanced nifi-ambari-ssl-config configuration section.

    • $TRUSTSTORE_PASSWORD is the truststore password.

    [Note]Note

    You can set keyStorePassword, keyPassword, and trustStorePassword to the Knox Master Secret to make it easier to import the keyStore and trustStore created by the NiFi Certificate Authority into the Knox keystore.

Example nifi-ca-config.json file

{
  "days" : 1095,
  "keySize" : 2048,
  "keyPairAlgorithm" : "RSA",
  "signingAlgorithm" : "SHA256WITHRSA",
  "dn" : "CN=slo-hdf-test5.field.hortonworks.com, OU=KNOX",
  "domainAlternativeNames" : null,
  "keyStore" : "/home/knox/knox-nifi-keystore.jks",
  "keyStoreType" : "jks",
  "keyStorePassword" : "admin",
  "keyPassword" : "admin",
  "token" : "token",
  "caHostname" : "slo-hdf-test5.field.hortonworks.com",
  "port" : 10443,
  "dnPrefix" : "CN=",
  "dnSuffix" : ", OU=NIFI",
  "reorderDn" : true,
  "trustStore" : "/home/knox/knox-nifi-truststore.jks",
  "trustStorePassword" : "admin",
  "trustStoreType" : "jks"
}

Creating Certificates for Knox

About This Task

You must create the keystore and truststore used by Knox when proxying NiFi.

Before You Begin

Ensure that Java is set correctly for your environment.

Steps

  1. As the root user, start the NiFi TLS Toolkit:For the location of the TLS Toolkit, see the HDF Release Notes for release-specific download information.

    /var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/bin/tls-toolkit.sh 
    client --subjectAlternativeNames "CN=$KNOX_HOSTNAME, OU=KNOX" 
    -F 
    -f /home/knox/config.json
    

    The toolkit requests a new certificate and creates two new files containing the keystore and truststore:

    /home/knox/knox-nifi-keystore.jks
    /home/knox/knox-nifi-truststore.jks
    
  2. Import the Knox certificate for NiFi into the Knox gateway.jks file:

    keytool 
    -importkeystore 
    -srckeystore /home/knox/knox-nifi-keystore.jks 
    -destkeystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks 
    -deststoretype JKS 
    -srcstorepass $KEYSTORE_PASSWORD 
    -deststorepass $KNOX_MASTER_PASSWORD
    

    The gateway.jks file should now contain a PrivateKeyEntry for NiFi.

  3. Import the NiFi CA truststore into the Knox gateway.jks file:

    keytool 
    -importkeystore 
    -srckeystore /home/knox/knox-nifi-truststore.jks 
    -destkeystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks 
    -deststoretype JKS 
    -srcstorepass $KEYSTORE_PASSWORD 
    -deststorepass $KNOX_MASTER_PASSWORD
    

    The gateway.jks file should now contain a trustedCertEntry for NiFi.

    Knox uses the gateway.jks file to fince secure certificates in the truststore.

  4. Verify that the proper keys are in the gateway.jks file:

    keytool 
    -keystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks 
    -storepass $KEYSTORE_PASSWORD 
    -list 
    -v
    

    You should see nifi-key and nifi-cert added to the gateway-identity key.

More Information

HDF Release Notes

Verify the Knox NiFi Service Configuration

If you are running HDP 2.6.5, the /usr/hdp/current/knox-server/data/services/nifi/1.4.0/service.xml file should contain the following content. You do not need to manually edit this file. <dispatch classname="org.apache.hadoop.gateway.dispatch.NiFiDispatch" ha-classname="org.apache.hadoop.gateway.dispatch.NiFiHaDispatch" use-two-way-ssl="false"/>

If you are running HDP 2.6.4 or earlier, set the use-two-way-ssl attribute to true.

In the /usr/hdp/current/knox-server/data/services/nifi/1.4.0/service.xml file, make sure the dispatch element contains the following: <dispatch classname="org.apache.hadoop.gateway.dispatch.NiFiDispatch" use-two-way-ssl="true"/>

Creating an Advanced Topology

Steps

  1. As the Knox user, create flow-management.xml in /usr/hdp/current/knox-server/conf/topologies.

  2. Add the following content to flow-management.xml:

    If you have modified the ShiroProvider in the topology specified in the Advanced Topology section of the Ambari Knox configs, and want to authenticate Flow Management topology users in the same manner as Advanced Topology users, ensure that those modifications are reflected in your added content.

    <?xml version="1.0" encoding="utf-8"?>
    <!--
      Licensed to the Apache Software Foundation (ASF) under one or more
      contributor license agreements.  See the NOTICE file distributed with
      this work for additional information regarding copyright ownership.
      The ASF licenses this file to You under the Apache License, Version 2.0
      (the "License"); you may not use this file except in compliance with
      the License.  You may obtain a copy of the License at
    
          http://www.apache.org/licenses/LICENSE-2.0
    
      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License.
    -->
    <topology>
      <gateway>
        <provider>
          <role>authentication</role>
          <name>ShiroProvider</name>
          <enabled>true</enabled>
          <param>
            <name>sessionTimeout</name>
            <value>30</value>
          </param>
          <param>
            <name>redirectToUrl</name>
            <value>/gateway/knoxsso/knoxauth/login.html</value>
          </param>
          <param>
            <name>restrictedCookies</name>
            <value>rememberme,WWW-Authenticate</value>
          </param>
          <param>
            <name>main.ldapRealm</name>
            <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
          </param>
          <param>
            <name>main.ldapContextFactory</name>
            <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
          </param>
          <param>
            <name>main.ldapRealm.contextFactory</name>
            <value>$ldapContextFactory</value>
          </param>
          <param>
            <name>main.ldapRealm.userDnTemplate</name>
            <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
          </param>
          <param>
            <name>main.ldapRealm.contextFactory.url</name>
            <value>ldap://localhost:33389</value>
          </param>
          <param>
            <name>main.ldapRealm.authenticationCachingEnabled</name>
            <value>false</value>
          </param>
          <param>
            <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
            <value>simple</value>
          </param>
          <param>
            <name>urls./**</name>
            <value>authcBasic</value>
          </param>
        </provider>
        <provider>
          <role>identity-assertion</role>
          <name>Default</name>
          <enabled>true</enabled>
        </provider>
      </gateway>
      <service>
        <role>NIFI</role>
        <url>$NIFI_HTTP_SCHEME://$NIFI_HOST:$NIFI_HTTP_SCHEME_PORT</url>
        <param name="useTwoWaySsl" value="true"/>
      </service>
    </topology>
    
    • $NIFI_HTTP_SCHEME specifies whether NiFi is communicating over HTTP or HTTPS.

    • $NIFI_HOST is the FQDN of the host machine on which NiFi is running.

    • $NIFI_HTTP_SCHEME_PORT specifies the port on which NiFi is running. This value varies depending on HTTP or HTTPS usage. In Advanced nifi-ambari-ssl-config, if Enable SSL? is selected, use the port specified byNiFi HTTP port (SSL), and if not, use the port specified by NiFi HTTP port (non-SSL). The port values are available in Advanced nifi-ambari-config.

  3. Save the configuration and restart Knox, or continue by configuring Knox SSO.

Configuring Knox SSO

Steps

  1. Navigate to Advaned knoxsso-topology and, in the KNOXSSO service definition, edit

    • the Knox SSO token time-to-live value:

      <param>
         <name>knoxsso.token.ttl</name>
         <value>36000000</value>
      </param>
      
    • Update the whitelist redirect regex property with a regex value that represents the host or domain in which the NiFi host is running:

      <param>
         <name>knoxsso.redirect.whitelist.regex</name>
         <value>^https?:\/\/(hdf-test\.field\.hortonworks\.com|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value>
      </param>
      
  2. If you want to use Knox SSO authentication, perform the following steps:

    1. On each cluster node with Knox installed, replace the ShiroProvider federation provider in the flow-management.xml file with the following content:

      <provider>
         <role>federation</role>
         <name>SSOCookieProvider</name>
         <enabled>true</enabled>
         <param>
            <name>sso.authentication.provider.url</name>
            <value>https://$KNOX_HOSTNAME:$KNOX_PORT/gateway/knoxsso/api/v1/websso</value>
         </param>
      </provider>
      

      where:

      • $KNOX_HOSTNAME is the FQDN of the Knox host.

      • $KNOX_PORT is the port Knox is using.

      Your new flow-managementso.xml file looks similar to the following:

      <?xml version="1.0" encoding="utf-8"?>
      <!--
        Licensed to the Apache Software Foundation (ASF) under one or more
        contributor license agreements.  See the NOTICE file distributed with
        this work for additional information regarding copyright ownership.
        The ASF licenses this file to You under the Apache License, Version 2.0
        (the "License"); you may not use this file except in compliance with
        the License.  You may obtain a copy of the License at
      
            http://www.apache.org/licenses/LICENSE-2.0
      
        Unless required by applicable law or agreed to in writing, software
        distributed under the License is distributed on an "AS IS" BASIS,
        WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        See the License for the specific language governing permissions and
        limitations under the License.
      -->
      <topology>
        <gateway>
          <provider>
            <role>federation</role>
            <name>SSOCookieProvider</name>
            <enabled>true</enabled>
            <param>
              <name>sso.authentication.provider.url</name>
              <value>https://hdf-test.field.hortonworks.com:8443/gateway/knoxsso/api/v1/websso</value>
            </param>
          </provider>
          <provider>
            <role>identity-assertion</role>
            <name>Default</name>
            <enabled>true</enabled>
          </provider>
        </gateway>
        <service>
          <role>NIFI</role>
          <url>https://hdf-test-nifi.field.hortonworks.com:9091</url>
          <param name="useTwoWaySsl" value="true" />
        </service>
      </topology>
      
    2. If you want to access NiFi directly and still use Knox SSO:

      1. Export the Knox SSO certificate:

         $KNOX_INSTALL_DIR/bin/knoxcli.sh export-cert 
      2. Set the following properties in the Advanced nifi-properties section in Ambari:

        nifi.security.user.knox.url=https://$KNOX_HOSTNAME:$KNOX_PORT/gateway/knoxsso/api/v1/websso
        nifi.security.user.knox.publicKey=<path-to>/gateway-identity.pem
        nifi.security.user.knox.cookieName=hadoop-jwt
        nifi.security.user.knox.audiences=
        

        The cookieName property must align with what is configured in Knox. The audiences property is used to only accept tokens from a particular audience. The audiences value is configured as part of Knox SSO.

  3. Save the configuration and restart Knox.