Test the LDAP connection use a curl command from the Knox Gateway to lists the
contents of the directory tmp/test
(or use a directory that exists in your
environment):
curl -i -k -u $ldap_user:$password -X GET \ 'https://$gateway_host:8443/$gateway_path/$cluster_name/webhdfs/api/v1/tmp/test?op=LISTSTATUS'
where the variables in the above command match actual items in your environment:
$ldap_user
and$password
is an actual user account in your environment (the LDAP provider binds to the UserDN).$gateway_host
and$gateway_path
matches your gateway configuration.$cluster_name
matches the name of the cluster topology descriptor file of the cluster you are trying to access.
Example outputs related to authentication messages:
Successful Authentication. The following requests and logs show normal operation:
Client-side successful request to WebHDFS using HDP Sandbox, the Knox sample Apache Directory, and the Knox Test Account
guest
:curl -i -k -u guest:guest-password -X GET 'https://sandbox:8443/gateway/sandbox/webhdfs/v1/user/?op=LISTSTATUS' HTTP/1.1 200 OK Set-Cookie: JSESSIONID=2hf99emf1dr31mzhjmpimwf1w;Path=/gateway/sandbox;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache Expires: Fri, 06 Jun 2014 20:26:53 GMT Date: Fri, 06 Jun 2014 20:26:53 GMT Pragma: no-cache Expires: Fri, 06 Jun 2014 20:26:53 GMT Date: Fri, 06 Jun 2014 20:26:53 GMT Pragma: no-cache Server: Jetty(6.1.26) Content-Type: application/json Content-Length: 1515 {"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":9,"fileId":16388,"group":"hdfs","length":0, "modificationTime":1398090362268,"owner":"ambari-qa","pathSuffix":"ambari-qa","permission":"770","replication":0,"type" :"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":17117,"group":"guest","length":0,"modificationTime" :1398176498899,"owner":"guest","pathSuffix":"guest","permission":"755","replication":0,"type":"DIRECTORY"},{"accessTime" :0,"blockSize":0,"childrenNum":0,"fileId":16949,"group":"hdfs","length":0,"modificationTime":1398090186250,"owner":"hcat", "pathSuffix":"hcat","permission":"755","replication":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":0, "fileId":16418,"group":"hdfs","length":0,"modificationTime":1398089829227,"owner":"hive","pathSuffix":"hive","permission": "700","replication":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":17019,"group":"hue","length" :0,"modificationTime":1398176493665,"owner":"hue","pathSuffix":"hue","permission":"755","replication":0,"type":"DIRECTORY"}, {"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16441,"group":"hdfs","length":0,"modificationTime":1398089926561, "owner":"oozie","pathSuffix":"oozie","permission":"775","replication":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0, "childrenNum":0,"fileId":17012,"group":"root","length":0,"modificationTime":1398176445256,"owner":"root","pathSuffix":"root", "permission":"755","replication":0,"type":"DIRECTORY"}]}}[root@sandbox xasecure-ha
The output above shows a success authentication and returns the content list.
Client-side request with WebHDFS error. Using the Knox sample environment on Sandbox. The user authenticates and requests a list from WebHDFS on a directory does not exist:
curl -i -k -u guest:guest-password -X GET 'https://sandbox:8443/gateway/sandbox/webhdfs/v1/user/myuser?op=LISTSTATUS' HTTP/1.1 404 Not Found Set-Cookie: JSESSIONID=10n9m664r778o1otump5s3jjqb;Path=/gateway/sandbox;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache Expires: Fri, 06 Jun 2014 20:57:26 GMT Date: Fri, 06 Jun 2014 20:57:26 GMT Pragma: no-cache Expires: Fri, 06 Jun 2014 20:57:26 GMT Date: Fri, 06 Jun 2014 20:57:26 GMT Pragma: no-cache Server: Jetty(6.1.26) Content-Type: application/json Content-Length: 151 {"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException", "message":"File /user/myuser does not exist."}}[ro...
Server Log examples. Each time a user tries to access a Hadoop REST API through the Knox Gateway, the attempt is captured in the audit log. For example:
14/06/06 13:26:53 ||c05b42e8-06aa-4673-959e-37743573dfdc|audit|WEBHDFS|guest|||access|uri|/gateway/sandbox/webhdfs/v1/user/?op=LISTSTATUS|success|Response st atus: 200 14/06/06 13:31:45 ||d63cb431-9cb9-4bd4-8d8c-26202dd1a71f|audit|WEBHDFS|guest|||access|uri|/gateway/sandbox/webhdfs/v1/user/?op=LISTSTATUS|unavailable| 14/06/06 13:57:26 ||0409c32a-2b29-4a05-8798-0420682e930b|audit|WEBHDFS||||access|uri|/gateway/sandbox/webhdfs/v1/user/myuser?op=LISTSTATUS|unavailable| 14/06/06 13:57:26 ||0409c32a-2b29-4a05-8798-0420682e930b|audit|WEBHDFS|guest|||authentication|uri|/gateway/sandbox/webhdfs/v1/user/myuser?op=LISTSTATUS|succe ss| 14/06/06 13:57:26 ||0409c32a-2b29-4a05-8798-0420682e930b|audit|WEBHDFS|guest|||dispatch|uri|http://sandbox.hortonworks.com:50070/webhdfs/v1/user/myuser?user. name=guest&op=LISTSTATUS|success|Response status: 404 14/06/06 13:57:26 ||0409c32a-2b29-4a05-8798-0420682e930b|audit|WEBHDFS|guest|||access|uri|/gateway/sandbox/webhdfs/v1/user/myuser?op=LISTSTATUS|success|Response status: 404
Authentication failures. When the user cannot be authenticated, the request is rejected with an HTTP status of 401 unauthorized regardless of the reason. The client receives the same error message (shown below) in the case of invalid credentials and if the Knox Gateway CANNOT establish a connection to the LDAP service:
curl -i -k -u guest:password -X GET 'https://sandbox:8443/gateway/sandbox/webhdfs/v1/user/?op=LISTSTATUS' HTTP/1.1 401 Unauthorized WWW-Authenticate: BASIC realm="application" Content-Length: 0 Server: Jetty(8.1.14.v20131031)
To verify connectivity to the LDAP service, enable debug on the Shiro provider and check the logs.
Log example for LDAP service connection issues. The gateway log with the Shiro provider debug enable contains an error messages similar to the ones below:
2014-06-06 16:06:55,831 DEBUG authc.BasicHttpAuthenticationFilter (BasicHttpAuthenticationFilter.java:createToken(308)) - Attempting to execute login with headers [Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=] 2014-06-06 16:06:55,832 DEBUG ldap.JndiLdapRealm (JndiLdapRealm.java:queryForAuthenticationInfo(369)) - Authenticating user 'guest' through LDAP 2014-06-06 16:06:55,832 DEBUG ldap.JndiLdapContextFactory (JndiLdapContextFactory.java:getLdapContext(488)) - Initializing LDAP context using URL [ldap://localhost:33389] and principal [uid=guest,ou=people,dc=hadoop,dc=apache,dc=org] with pooling disabled 2014-06-06 16:06:55,838 DEBUG servlet.SimpleCookie (SimpleCookie.java:addCookieHeader(226)) - Added HttpServletResponse Cookie [rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0; Expires=Thu, 05-Jun-2014 23:06:55 GMT] 2014-06-06 16:06:55,839 DEBUG authc.BasicHttpAuthenticationFilter (BasicHttpAuthenticationFilter.java:sendChallenge(274)) - Authentication required: sending 401 Authentication challenge response.
Log example for invalid credentials:
14/06/06 13:31:45 ||d63cb431-9cb9-4bd4-8d8c-26202dd1a71f|audit|WEBHDFS|guest|||access|uri|/gateway/sandbox/webhdfs/v1/user/?op=LISTSTATUS|success|Response st atus: 401