Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) ()
FreeIPA replication failes for about 13 minutes with no activity on the first IDM server. Not clear why at first.
Feb 12 10:06:56 idmipa01 named-pkcs11[2529]: zone nix.mds.xyz/IN: sending notifies (serial 1518448016)
Feb 12 10:07:06 idmipa01 named-pkcs11[2529]: error (chase DS servers) resolving 'mds.xyz/DS/IN': 192.168.0.224#53
Feb 12 10:07:14 idmipa01 ns-slapd: [12/Feb/2018:10:07:14.130840773 -0500] – ERR – NSMMReplicationPlugin – bind_and_check_pwp – agmt="cn=meToidmipa02.nix.mds.xyz" (idmipa02:389) – Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) ()
Feb 12 10:20:01 idmipa01 systemd: Created slice user-0.slice.
Feb 12 10:20:01 idmipa01 systemd: Starting user-0.slice.
The problem was again with NTP and time/date settings.
[root@idmipa02 log]# date
Wed Feb 14 00:05:58 EST 2018
[root@idmipa02 log]#
[root@idmipa01 log]# date
Wed Feb 14 00:00:14 EST 2018
You have new mail in /var/spool/mail/root
[root@idmipa01 log]#
Over 5 minute difference. Checking further we see the following in the logs:
Feb 12 10:13:00 idmipa02 rc.local: Error resolving ca.pool.ntp.org: Name or service not known (-2)
Feb 12 10:13:00 idmipa02 rc.local: 12 Feb 10:13:00 ntpdate[963]: Can't find host ca.pool.ntp.org: Name or service not known (-2)
Feb 12 10:13:00 idmipa02 rc.local: 12 Feb 10:13:00 ntpdate[963]: no servers can be used, exiting
So we need to keep the time between the two masters in sync otherwise this replication issue will reoccur. But we need to ensure our NTP servers are resolvable. So we may need to put extra conditions in our NTP servers. We have:
[root@idmipa01 log]# cat /etc/rc.local |grep -Evi "#"
touch /var/lock/subsys/local
ntpdate -u ca.pool.ntp.org;
[root@idmipa01 log]#
But we should use a single IP in case of failure (We are using NLB on our AD DC servers and we noted a failure on that host earlier which we just fixed.):
[root@idmipa01 log]# cat /etc/rc.local |grep -Evi "#"
touch /var/lock/subsys/local
ntpdate -u ca.pool.ntp.org || ntpdate -u 206.108.0.132 || ntpdate -u 159.203.8.72;
[root@idmipa01 log]#
This gives us some safety in case the name can't be resolved due to DNS issues. We will also reconfigure our NTP servers as follows:
[root@idmipa02 log]# grep -Evi "#" /etc/ntp.conf | sed -e "/^$/d"
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
fudge 127.127.1.0 stratum 10
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
restrict 127.0.0.1
restrict ::1
driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log
server 0.ca.pool.ntp.org prefer
server 1.ca.pool.ntp.org
server 2.ca.pool.ntp.org
server 3.ca.pool.ntp.org
server 198.50.139.209
server 207.210.46.249
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
disable monitor
[root@idmipa02 log]#
and
[root@idmipa01 log]# grep -Evi "#" /etc/ntp.conf|sed -e "/^$/d"
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
fudge 127.127.1.0 stratum 10
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
restrict 127.0.0.1
restrict ::1
driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log
server 207.210.46.249
server 198.50.139.209
server 0.ca.pool.ntp.org
server 1.ca.pool.ntp.org
server 2.ca.pool.ntp.org
server 3.ca.pool.ntp.org prefer
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
disable monitor
[root@idmipa01 log]#
Noticed the preferred NTP servers are different on each of our NTP servers. We're attempting to prevent a scenario where the same external NTP server is polled twice from two different servers simultaneously. No clear evidence if this causes an issue but setting an alternate preferred server for each of our NTP servers prevents that from occurring just in case it could ever be true. We also add 2 IP's from one the domains above in case DNS errors cause us issues. We will be immune to this if it were ever to come up. The difference is significant:
[root@idmipa02 log]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*LOCAL(0) .LOCL. 10 l 4 64 1 0.000 0.000 0.000
k8s-w04.tblflp. 152.2.133.55 2 u 3 64 1 21.943 906.098 0.000
echo.baxterit.n 213.251.128.249 2 u 2 64 1 39.255 908.220 0.000
k8s-w01.tblflp. 152.2.133.55 2 u 1 64 1 18.415 903.549 0.000
portal.switch.c 213.251.128.249 2 u – 64 1 16.560 901.799 0.000
mirror3.rafal.c .INIT. 16 u – 64 0 0.000 0.000 0.000
198.50.139.209 .INIT. 16 u – 64 0 0.000 0.000 0.000
[root@idmipa02 log]#
[root@idmipa01 log]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*LOCAL(0) .LOCL. 10 l 34 64 1 0.000 0.000 0.000
198.50.139.209 35.73.197.144 2 u 33 64 1 19.071 -84.149 0.000
mirror3.rafal.c 53.27.192.223 2 u 32 64 1 18.490 -56.439 0.000
ns522433.ip-158 18.26.4.105 2 u 31 64 1 17.833 -80.900 0.000
echo.baxterit.n 213.251.128.249 2 u 30 64 1 16.688 -82.694 0.000
209.115.181.102 206.108.0.133 2 u 29 64 1 72.834 -82.194 0.000
mongrel.ahem.ca .INIT. 16 u – 64 0 0.000 0.000 0.000
[root@idmipa01 log]#
Good Luck!
Cheers,
TK