Troubleshooting Kerberos

Java errors from Kerberos problems can be obscure and difficult to interpret. To see the Kerberos error messages, enable Kerberos debugging by adding these settings to the JVM:

-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true -Djavax.net.debug=all

Enable Kerberos debugging for the HPE Ezmeral Data Fabric-provided maprcli and Hadoop clients by adding the following line to the /opt/mapr/conf/env_override.sh shell script:

#MAPRLOGIN_OPTS="${MAPRLOGIN_OPTS} -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true -Djavax.net.debug=all"

The env.sh script reads this file as part of its execution. For more information, see About env_override.sh.

Capture the Kerberos error to research the issue.

The following sections list common Kerberos error conditions.

Incorrect JVM

Nodes often have multiple JVMs installed. The HPE Ezmeral Data Fabric env.sh script automatically configures a JVM to use. To change the automatically-configured JVM, set the value of the JAVA_HOME environment variable in the /opt/mapr/conf/env_override.sh file. The env.sh script reads this file as part of its execution. For more information, see About env_override.sh.

Incorrect Server Name

The following error message is caused by an incorrect CLDB server name in the mapr.login.conf file. The error message mentions passwords, but the error condition is unrelated to password authentication.

2018-04-25 16:46:02,324 ERROR MapRLoginServlet [185087767@qtp-648535353-2]: Failed to obtain kerberos identity, continuing anyway...
        javax.security.auth.login.LoginException: Unable to obtain password from user
        
        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:789)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:654)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)
        at com.mapr.fs.cldb.http.login.MapRLoginServlet.init(MapRLoginServlet.java:73)
        at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
        at org.mortbay.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:339)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Invalid or missing keytab file

The keytab file must be consistent with the key versions of the Kerberos principal. The following example shows an inconsistent keytab file:

kadmin:  getprinc mapr/realm1
        Principal: mapr/realm1@mapr
        Expiration date: [never]
        Last password change: Thu May 23 15:36:01 PDT 2013
        Password expiration date: [none]
        Maximum ticket life: 0 days 10:00:00
        Maximum renewable life: 7 days 00:00:00
        Last modified: Thu May 23 15:36:01 PDT 2013 (mapr/admin@mapr)
        Last successful authentication: Thu May 23 19:31:59 PDT 2013
        Last failed authentication: Thu May 23 15:35:41 PDT 2013
        Failed password attempts: 0
        Number of keys: 8
        Key: vno 15, aes256-cts-hmac-sha1-96, no salt
        Key: vno 15, arcfour-hmac, no salt
        Key: vno 15, des3-cbc-sha1, no salt
        Key: vno 15, des-cbc-crc, no salt
        Key: vno 15, des-cbc-md5, Version 4
        Key: vno 15, des-cbc-md5, Version 5 - No Realm
        Key: vno 15, des-cbc-md5, Version 5 - Realm Only
        Key: vno 15, des-cbc-md5, AFS version 3
        MKey: vno 1
        
        ktutil:  rkt mapr.keytab
        ktutil:  l
        slot KVNO Principal
        ---- ---- ---------------------------------------------------------------------
        1   14             mapr/realm1@mapr
        2   14             mapr/realm1@mapr
        3   14             mapr/realm1@mapr
        4   14             mapr/realm1@mapr
        ktutil:  q

Note that the key versions in the Kerberos principal /realm1 are 15, and the versions in the keytab file are 14. This mismatch can result in errors about missing keys or mismatched encryption.

NOTE

This error state can also be caused by the /opt/mapr/conf/mapr.keytab file not being owned by the user mapr or not being present. The keytab file is owned by root at generation. Be sure to use the chown command to set the mapr user as the owner:

$ chown mapr:mapr /opt/mapr/conf/mapr.keytab

Incompatible encryption on Java runtime

Incompatible cryptography between the KDC and the JDK results in failed handshakes, leading to errors similar to the following:

Caused by: javax.security.auth.login.LoginException: Unable to obtain Principal Name for authentication

With debugging active, the following message is displayed:

>>>DEBUG <CCacheInputStream>  client principal is username@hostname
        >>>DEBUG <CCacheInputStream> server principal is X-CACHECONF:/krb5_ccache_conf_data/fast_avail/krbtgt/user@hostname
        >>>DEBUG <CCacheInputStream> key type: 0
        >>>DEBUG <CCacheInputStream> auth time: Wed Dec 31 16:00:00 PST 1969
        >>>DEBUG <CCacheInputStream> start time: null
        >>>DEBUG <CCacheInputStream> end time: Wed Dec 31 16:00:00 PST 1969
        >>>DEBUG <CCacheInputStream> renew_till time: null
        >>> CCacheInputStream: readFlags() 
        >>> unsupported key type found the default TGT: 18
        Negotiate support not initiated, will fallback to other scheme if allowed. Reason:

This debug message indicates that the problem is an unsupported key type.

Incompatible encryption errors can occur due to a keytab file that is not present or contains outdated keys.

Be sure to update the Java jurisdiction policy file. Jurisdiction policy files are available from Oracle.

A persistent encryption incompatibility problem may require you to edit the krb5.conf file to ensure compatible algorithms between Java and Kerberos.

Bugs in Java

The following error occurs in Java version 1.6.0_25. Upgrade to 1.6.0_45 to resolve the error.

java.io.IOException: extra data given to DerValue constructor
        at sun.security.util.DerValue.init(DerValue.java:368)
        at sun.security.util.DerValue.<init>(DerValue.java:277)
        at sun.security.krb5.internal.Ticket.<init>(Ticket.java:81)
        at sun.security.krb5.internal.ccache.CCacheInputStream.readData(CCacheInputStream.java:250)
        at sun.security.krb5.internal.ccache.CCacheInputStream.readCred(CCacheInputStream.java:357)
        at sun.security.krb5.internal.ccache.FileCredentialsCache.load(FileCredentialsCache.java:225)
        at sun.security.krb5.internal.ccache.FileCredentialsCache.acquireInstance(FileCredentialsCache.java:104)
        at sun.security.krb5.internal.ccache.CredentialsCache.getInstance(CredentialsCache.java:75)
        at sun.security.krb5.Credentials.acquireTGTFromCache(Credentials.java:304)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:589)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
        at javax.security.auth.login.LoginContext$5.run(LoginContext.java:706)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:703)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:575)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:554)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:528)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1656)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1649)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1517)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:235)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
        at org.apache.hadoop.fs.FsShell.init(FsShell.java:87)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:1808)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:1967)
        13/05/10 15:24:00 DEBUG security.SaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos principal name is hdfs/qa-node24@dev-maprtech
        13/05/10 15:24:00 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
NOTE

Starting with the 4.0.1 release of the HPE Ezmeral Data Fabric software, Java 6 is deprecated in favor of Java 7 and Java 8.

Kerberos and PAM validation

Standard Kerberos implementations are predicated on access to elevated user privileges that are not present on secure HPE Ezmeral Data Fabric clusters. In a HPE Ezmeral Data Fabric cluster, the Control System console and other components call PAM as an ordinary user process. This discrepancy in expected and actual privileges can cause a variety of obscure file permisson errors. Since different Kerberos PAM modules are available, error reports can vary.

To diagnose this issue, attempt starting the Control System as the root user, or clear out the /tmp folder. If there are no problems when starting the Control System as root, or if clearing out the /tmp folder enables a single login before errors appear again, the problem may lie in the Kerberos PAM configuration.

To resolve this condition, prevent Kerberos from creating a ticket file. HPE Ezmeral Data Fabric security does not use Kerberos tickets. The Kerberos KDC is used to validate passwords. Typically the configuration file for PAM is in the /etc/pam.d directory. See the documentation for your specific Kerberos PAM module for more information.