org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
Getting this?
[root@cm-r01en01 ~]# hdfs dfs -ls /
19/08/25 22:43:19 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
ls: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "cm-r01en01.mws.mds.xyz/192.168.0.140"; destination host is: "cm-r01nn02.mws.mds.xyz":8020;
[root@cm-r01en01 ~]#
Fix it by doing the following:
-
Ensure the following setting is commented out and following two settings exist in the krb5.conf:
# default_ccache_name = KEYRING:persistent:%{uid}
renew_lifetime = 7d
forwardable = true -
Stop the cluster and CM.
-
Regenerate the Cluster Kerberos Credentials in Administration – Security.
-
Start CM and CDH services.
-
Try the procedure again.
Try the operation again:
[root@cm-r01en01 ~]# ls -altri /var/run/cloudera-scm-agent/process/*hdfs*/hdfs.keytab
40996 -rw——-. 1 hdfs hdfs 534 Aug 25 01:58 /var/run/cloudera-scm-agent/process/1016-hdfs-NFSGATEWAY/hdfs.keytab
57096 -rw——-. 1 hdfs hdfs 534 Aug 25 02:01 /var/run/cloudera-scm-agent/process/1089-hdfs-NFSGATEWAY/hdfs.keytab
17388393 -rw——-. 1 hdfs hdfs 534 Aug 25 08:48 /var/run/cloudera-scm-agent/process/1174-hdfs-NFSGATEWAY/hdfs.keytab
17814727 -rw——-. 1 hdfs hdfs 534 Aug 25 20:16 /var/run/cloudera-scm-agent/process/1249-hdfs-NFSGATEWAY/hdfs.keytab
17871689 -rw——-. 1 hdfs hdfs 534 Aug 25 21:29 /var/run/cloudera-scm-agent/process/1329-hdfs-NFSGATEWAY/hdfs.keytab
[root@cm-r01en01 ~]# kinit -kt /var/run/cloudera-scm-agent/process/1329-hdfs-NFSGATEWAY/hdfs.keytab hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ
[root@cm-r01en01 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ
Valid starting Expires Service principal
08/25/2019 22:44:06 08/26/2019 22:44:06 krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
renew until 09/01/2019 22:44:06
[root@cm-r01en01 ~]#
[root@cm-r01en01 ~]#
[root@cm-r01en01 ~]#
[root@cm-r01en01 ~]# hdfs dfs -ls /
Found 4 items
drwxr-xr-x – hbase hbase 0 2019-08-25 21:30 /hbase
drwxrwxr-x – solr solr 0 2019-08-13 00:41 /solr
drwxrwxrwt – hdfs supergroup 0 2019-08-17 21:28 /tmp
drwxr-xr-x – hdfs supergroup 0 2019-08-17 22:38 /user
[root@cm-r01en01 ~]# klist -fe
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ
Valid starting Expires Service principal
08/25/2019 22:44:06 08/26/2019 22:44:06 krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
renew until 09/01/2019 22:44:06, Flags: FRIA
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[root@cm-r01en01 ~]#
Note the flags above FRIA. The R stands for renewable, a requirements for Cloudera.
UNVERIFIED
An alternate solution to this could be to set the following to privacy however we never tried it.
hadoop.rpc.protection
Cheers,
TK
You must be logged in to post a comment.