Getting the following errors from spark-shell or from listing out valid KMS keys?

tom@mds.xyz@cm-r01en01:~] 🙂 $ hadoop key list
19/09/17 23:56:43 INFO util.KerberosName: No auth_to_local rules applied to tom@MDS.XYZ
Cannot list keys for KeyProvider: org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@e350b40
list [-provider ] [-strict] [-metadata] [-help]:

The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately if
the provider requires a password and none is given.
Executing command failed with the following exception: AuthorizationException: User:tom@MDS.XYZ not allowed to do 'GET_KEYS'
tom@mds.xyz@cm-r01en01:~] 🙁 $

Or the following message entry?

19/09/17 22:17:25 DEBUG ipc.Client: Negotiated QOP is :auth
19/09/17 22:17:25 DEBUG ipc.Client: IPC Client (1322600748) connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8020 from tom@MDS.XYZ: starting, having connections 1
19/09/17 22:17:25 DEBUG ipc.Client: IPC Client (1322600748) connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8020 from tom@MDS.XYZ sending #0 org.apache.hadoop.hdfs.protocol.ClientProtocol.getDelegationToken
19/09/17 22:17:25 DEBUG ipc.Client: IPC Client (1322600748) connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8020 from tom@MDS.XYZ got value #0
19/09/17 22:17:25 DEBUG ipc.ProtobufRpcEngine: Call: getDelegationToken took 650ms
19/09/17 22:17:25 INFO util.KerberosName: No auth_to_local rules applied to tom@MDS.XYZ
19/09/17 22:17:25 INFO hdfs.DFSClient: Created token for tom@MDS.XYZ: HDFS_DELEGATION_TOKEN owner=tom@MDS.XYZ, renewer=yarn, realUser=, issueDate=1568773045589, maxDate=1569377845589, sequenceNumber=56, masterKeyId=62 on 192.168.0.133:8020
19/09/17 22:17:25 DEBUG ipc.Client: IPC Client (1322600748) connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8020 from tom@MDS.XYZ sending #1 org.apache.hadoop.hdfs.protocol.ClientProtocol.getServerDefaults
19/09/17 22:17:25 DEBUG ipc.Client: IPC Client (1322600748) connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8020 from tom@MDS.XYZ got value #1
19/09/17 22:17:25 DEBUG ipc.ProtobufRpcEngine: Call: getServerDefaults took 2ms
19/09/17 22:17:25 DEBUG kms.KMSClientProvider: KMSClientProvider created for KMS url: http://cm-r01nn01.mws.mds.xyz:16000/kms/v1/ delegation token service: kms://http@cm-r01nn01.mws.mds.xyz:16000/kms canonical service: 192.168.0.134:16000.
19/09/17 22:17:25 DEBUG kms.LoadBalancingKMSClientProvider: Created LoadBalancingKMSClientProvider for KMS url: kms://http@cm-r                           01nn01.mws.mds.xyz:16000/kms with 1 providers. delegation token service: kms://http@cm-r01nn01.mws.mds.xyz:16000/kms, canonical service: 192.168.0.134:16000
19/09/17 22:17:25 DEBUG kms.KMSClientProvider: Current UGI: tom@MDS.XYZ (auth:KERBEROS)
19/09/17 22:17:25 DEBUG kms.KMSClientProvider: Login UGI: tom@MDS.XYZ (auth:KERBEROS)
19/09/17 22:17:25 DEBUG security.UserGroupInformation: PrivilegedAction as:tom@MDS.XYZ (auth:KERBEROS) from:org.apache.hadoop.c                           rypto.key.kms.KMSClientProvider.getDelegationToken(KMSClientProvider.java:1029)
19/09/17 22:17:25 DEBUG kms.KMSClientProvider: Getting new token from http://cm-r01nn01.mws.mds.xyz:16000/kms/v1/, renewer:yarn/cm-r01nn02.mws.mds.xyz@MWS.MDS.XYZ
19/09/17 22:17:25 DEBUG web.DelegationTokenAuthenticator: No delegation token found for url=http://cm-r01nn01.mws.mds.xyz:16000/kms/v1/?op=GETDELEGATIONTOKEN&renewer=yarn%2Fcm-r01nn02.mws.mds.xyz%40MWS.MDS.XYZ, token=, authenticating with class org.apach                           e.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticator$1
19/09/17 22:17:25 DEBUG client.KerberosAuthenticator: JDK performed authentication on our behalf.
19/09/17 22:17:25 DEBUG client.AuthenticatedURL: Cannot parse cookie header:
java.lang.IllegalArgumentException: Empty cookie header string
 

Solve it by adjusting your KMS settings to include the groups and users that will run your commands as follows:

Name: hadoop.kms.acl.GET_KEYS
Value: kmsadmin,kmsadmingroup,hdfs,cdhadmins@mds.xyz,nixadmins@mds.xyz,cdhadmins,nixadmins,tom@MDS.XYZ
Description: ACL for get-keys operations.

And test using:

tom@mds.xyz@cm-r01en01:~] 🙂 $ hadoop key list
19/09/18 07:20:23 INFO util.KerberosName: No auth_to_local rules applied to tom@MDS.XYZ
Listing keys for KeyProvider: org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@121314f7
tom@mds.xyz@cm-r01en01:~] 🙂 $

Cheers,
TK