Are you getting this result?

[root@cm-r01en01 process]# systemctl restart cloudera-scm-agent
[root@cm-r01en01 process]#
[root@cm-r01en01 process]#
[root@cm-r01en01 process]#
[root@cm-r01en01 process]# kinit -kt ./1401-hdfs-NFSGATEWAY/hdfs.keytab hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ
[root@cm-r01en01 process]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: host/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ

Valid starting       Expires              Service principal
12/31/1969 19:00:00  12/31/1969 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
[root@cm-r01en01 process]#
 

Solve it by restarting nfs-ganesha on the NFS server!   I know what you're saying but keep reading.

Turns out our NFS server was stuck on one of the three cluster nodes making up our NFS cluster.  This affected the cloudera-scm-agent (CMA) . It couldn't properly communicate with or report back to the CMS ( Cloudera Manager Server ).  CMA does FS checks.  It got stuck trying to read the NFS mount.  

Restarting nfs-ganesha allowed NFS reads again.  CMA could then do it's FS checks and report back.  Renewing the ticket after that fixed the above issue:

[root@cm-r01en01 process]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ

Valid starting       Expires              Service principal
08/31/2019 07:30:14  09/01/2019 07:30:14  krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
        renew until 09/07/2019 07:30:14
[root@cm-r01en01 process]#

Cheers,
TK