Getting this?

[root@cm-r01en01 ~]# hdfs dfs -ls /
19/08/25 22:43:19 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
ls: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "cm-r01en01.mws.mds.xyz/192.168.0.140"; destination host is: "cm-r01nn02.mws.mds.xyz":8020;
[root@cm-r01en01 ~]#

Fix it by doing the following:

• Ensure the following setting is commented out and following two settings exist in the krb5.conf:

  # default_ccache_name = KEYRING:persistent:%{uid}

  renew_lifetime = 7d
  forwardable = true

• Stop the cluster and CM.

• Regenerate the Cluster Kerberos Credentials in Administration – Security.

• Start CM and CDH services.

• Try the procedure again.

Try the operation again:

[root@cm-r01en01 ~]# ls -altri /var/run/cloudera-scm-agent/process/*hdfs*/hdfs.keytab
   40996 -rw——-. 1 hdfs hdfs 534 Aug 25 01:58 /var/run/cloudera-scm-agent/process/1016-hdfs-NFSGATEWAY/hdfs.keytab
   57096 -rw——-. 1 hdfs hdfs 534 Aug 25 02:01 /var/run/cloudera-scm-agent/process/1089-hdfs-NFSGATEWAY/hdfs.keytab
17388393 -rw——-. 1 hdfs hdfs 534 Aug 25 08:48 /var/run/cloudera-scm-agent/process/1174-hdfs-NFSGATEWAY/hdfs.keytab
17814727 -rw——-. 1 hdfs hdfs 534 Aug 25 20:16 /var/run/cloudera-scm-agent/process/1249-hdfs-NFSGATEWAY/hdfs.keytab
17871689 -rw——-. 1 hdfs hdfs 534 Aug 25 21:29 /var/run/cloudera-scm-agent/process/1329-hdfs-NFSGATEWAY/hdfs.keytab
[root@cm-r01en01 ~]# kinit -kt /var/run/cloudera-scm-agent/process/1329-hdfs-NFSGATEWAY/hdfs.keytab hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ
[root@cm-r01en01 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ

Valid starting       Expires              Service principal
08/25/2019 22:44:06  08/26/2019 22:44:06  krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
        renew until 09/01/2019 22:44:06
[root@cm-r01en01 ~]#
[root@cm-r01en01 ~]#
[root@cm-r01en01 ~]#
[root@cm-r01en01 ~]# hdfs dfs -ls /
Found 4 items
drwxr-xr-x   – hbase hbase               0 2019-08-25 21:30 /hbase
drwxrwxr-x   – solr  solr                0 2019-08-13 00:41 /solr
drwxrwxrwt   – hdfs  supergroup          0 2019-08-17 21:28 /tmp
drwxr-xr-x   – hdfs  supergroup          0 2019-08-17 22:38 /user
[root@cm-r01en01 ~]# klist -fe
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ

Valid starting       Expires              Service principal
08/25/2019 22:44:06  08/26/2019 22:44:06  krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
        renew until 09/01/2019 22:44:06, Flags: FRIA
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[root@cm-r01en01 ~]#

Note the flags above FRIA.  The R stands for renewable, a requirements for Cloudera.

UNVERIFIED

An alternate solution to this could be to set the following to privacy however we never tried it.  

hadoop.rpc.protection

Cheers,
TK