Getting this?
INFO:desktop.kt_renewer:Renewing kerberos ticket to work around kerberos 1.8.1: /usr/bin/kinit -R -c /var/run/hue/hue_krb5_ccache
kinit: KDC can't fulfill requested option while renewing credentials
ERROR:desktop.kt_renewer:Couldn't renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for 'hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ' is still renewable:
$ klist -f -c /var/run/hue/hue_krb5_ccache
If the 'renew until' date is the same as the 'valid starting' date, the ticket cannot be renewed. Please check your KDC configuration, and the ticket renewal policy (maxrenewlife) for the 'hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ' and `krbtgt' principals.
Resolve it by adding the following lines to the /etc/krb5.conf file on the KDC servers:
[libdefaults]
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
Then regenerate the kerberos credentials by browsing to Administration – Security – Kerberos Credentials, checking off all hosts and regenerating the Kerberos Credentails for all.
Once that is done, restart the Kerberos Ticket Renewer.
Didn't work? Set the Maximum Renewable Life for Principals to 7 days from 5 days and set the Hue Keytab Renewal Interval to 7 days:
Hue Keytab Renewal Interval
reinit_frequency
However this did not work for us either. The real issue was in this message on the KDC server:
Aug 24 21:26:50 idmipa03.mws.mds.xyz krb5kdc[12023](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 192.168.0.140: TICKET NOT RENEWABLE: authtime 0, hu /cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ for krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ, KDC can't fulfill requested option
We can see this through the cache:
[root@cm-r01en02 ~]# klist -fe /var/run/hue/hue_krb5_ccache
Ticket cache: FILE:/var/run/hue/hue_krb5_ccache
Default principal: hue/cm-r01en02.mws.mds.xyz@MWS.MDS.XYZ
Valid starting Expires Service principal
08/24/2019 21:14:07 08/25/2019 21:14:07 krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
Flags: FIA, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
You have new mail in /var/spool/mail/root
[root@cm-r01en02 ~]#
Hence why when we try to kinit with the ticket, we get the above renewal error message:
[root@cm-r01en01 ~]# kinit -R -c /var/run/hue/hue_krb5_ccache
kinit: KDC can't fulfill requested option while renewing credentials
[root@cm-r01en01 ~]#
It is missing the R ( renewable ) flag . For this we'll need to modify one of the Cloudera scripts to ensure our tickets are renewable by adding +allow_renewable to the code that creates the principals:
68 # Set the maxrenewlife for the principal, if given. There is no interface
69 # offered by the IPA to set it, so we use KADMIN as suggested in a few IPA
70 # related forums.
71 #set +e
72 KADMIN="kadmin -k -t $CMF_KEYTAB_FILE -p $CMF_PRINCIPAL -r $CMF_REALM"
73
74 if [ $MAX_RENEW_LIFE -gt 0 ]; then
75 mkdir /tmp/kadmin/;
76 klist >> /tmp/kadmin/klist.log;
77 pwd >> /tmp/kadmin/pwd.log;
78 whoami >> /tmp/kadmin/whoami.log;
79 who am i >> /tmp/kadmin/who-am-i.log;
80 echo "Running: $KADMIN -q \"modprinc -maxrenewlife \"$MAX_RENEW_LIFE sec\" $PRINCIPAL\" " >> /tmp/kadmin/kadmin-command.log;
81 /bin/cp $CMF_KEYTAB_FILE $CMF_PRINCIPAL /tmp/kadmin/;
82 $KADMIN -q "modprinc -maxrenewlife \"$MAX_RENEW_LIFE sec\" +allow_renewable $PRINCIPAL"
83 fi
84 #set -e
Save the code and redistribute it:
[root@awx01 ansible]# vi adhoc/gen_credentials_ipa.sh
[root@awx01 ansible]# ansible cm* -i infra -m copy -a 'src=adhoc/gen_credentials_ipa.sh dest=/opt/cloudera/cm/bin/gen_credentials_ipa.sh'
Once this is done, regenerate all the kerberos principals in Administration – Security – Kerberos Credentials.
Another issue here is the Encryption Types and possibly the renewable life:
kadmin.local: getprinc krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
Principal: krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
Expiration date: [never]
Last password change: [never]
Password expiration date: [never]
Maximum ticket life: 7 days 00:00:00
Maximum renewable life: 14 days 00:00:00
Last modified: Mon Feb 04 22:19:28 EST 2019 (db_creation@MWS.MDS.XYZ)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 1, aes256-cts-hmac-sha1-96
Key: vno 1, aes128-cts-hmac-sha1-96
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH LOCKDOWN_KEYS
Policy: [none]
kadmin.local: modprinc -maxrenewlife 90day krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
Principal "krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ" modified.
kadmin.local:
kadmin.local:
kadmin.local: getprinc krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
Principal: krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
Expiration date: [never]
Last password change: [never]
Password expiration date: [never]
Maximum ticket life: 7 days 00:00:00
Maximum renewable life: 90 days 00:00:00
Last modified: Sat Aug 24 22:45:03 EDT 2019 (admin/admin@MWS.MDS.XYZ)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 1, aes256-cts-hmac-sha1-96
Key: vno 1, aes128-cts-hmac-sha1-96
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH LOCKDOWN_KEYS
Policy: [none]
kadmin.local:
However the encryption types were of highest concern. In our Cloudera Kerberos configuration, we didn't have these thereby getting:
Aug 24 22:43:38 idmipa03.mws.mds.xyz krb5kdc[12022](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 192.168.0.140: NEEDED_PREAUTH: hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ for krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ, Additional pre-authentication required
Aug 24 22:43:38 idmipa03.mws.mds.xyz krb5kdc[12022](info): closing down fd 11
Aug 24 22:43:38 idmipa03.mws.mds.xyz krb5kdc[12023](info): preauth (encrypted_timestamp) verify failure: Preauthentication failed
Aug 24 22:43:38 idmipa03.mws.mds.xyz krb5kdc[12023](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 192.168.0.140: PREAUTH_FAILED: hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ for krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ, Preauthentication failed
Aug 24 22:43:38 idmipa03.mws.mds.xyz krb5kdc[12023](info): closing down fd 11
when we tried to run:
[root@cm-r01en01 ~]# /bin/kinit -k -t /run/cloudera-scm-agent/process/749-hue-KT_RENEWER/hue.keytab -c /var/run/hue/hue_krb5_ccache hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ
kinit: Preauthentication failed while getting initial credentials
[root@cm-r01en01 ~]#
Add the encryption types to the configuration:
Administration – Settings
Kerberos Encryption Types
aes256-cts-hmac-sha1-96
aes128-cts-hmac-sha1-96
Again, regenerate the Kerberos Credentials and check the kerberos cache again:
[root@cm-r01en01 ~]# /bin/kinit -k -t /run/cloudera-scm-agent/process/904-hue-KT_RENEWER/hue.keytab -c /var/run/hue/hue_krb5_ccache hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ
[root@cm-r01en01 ~]#
Tried adjusting the kdc.conf with no luck either:
[root@idmipa04 ~]# cat /var/kerberos/krb5kdc/kdc.conf | grep default_principal_flags
default_principal_flags = +preauth, +renewable
[root@idmipa04 ~]#
But no luck. Ultimately, we ended up setting the flags of the /etc/krb5.conf on each CM machine instead:
[root@cm-r01en01 ~]# vi /etc/krb5.conf
[libdefaults]
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
And the same on cm-r01en02. That worked!
[root@cm-r01en01 ~]# klist -fe
Ticket cache: KEYRING:persistent:0:krb_ccache_Wg0x02u
Default principal: hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ
Valid starting Expires Service principal
08/25/2019 01:02:14 08/26/2019 01:02:14 krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ
renew until 09/01/2019 01:02:14, Flags: FRIA
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[root@cm-r01en01 ~]#
Cheers,
TK
