Header Shadow Image


Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) ()

FreeIPA replication failes for about 13 minutes with no activity on the first IDM server.  Not clear why at first.

Feb 12 10:06:56 idmipa01 named-pkcs11[2529]: zone nix.mds.xyz/IN: sending notifies (serial 1518448016)
Feb 12 10:07:06 idmipa01 named-pkcs11[2529]: error (chase DS servers) resolving 'mds.xyz/DS/IN': 192.168.0.224#53
Feb 12 10:07:14 idmipa01 ns-slapd: [12/Feb/2018:10:07:14.130840773 -0500] – ERR – NSMMReplicationPlugin – bind_and_check_pwp – agmt="cn=meToidmipa02.nix.mds.xyz" (idmipa02:389) – Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) ()
Feb 12 10:20:01 idmipa01 systemd: Created slice user-0.slice.
Feb 12 10:20:01 idmipa01 systemd: Starting user-0.slice.

The problem was again with NTP and time/date settings.

[root@idmipa02 log]# date
Wed Feb 14 00:05:58 EST 2018
[root@idmipa02 log]#

 

[root@idmipa01 log]# date
Wed Feb 14 00:00:14 EST 2018
You have new mail in /var/spool/mail/root
[root@idmipa01 log]#

Over 5 minute difference.  Checking further we see the following in the logs:

Feb 12 10:13:00 idmipa02 rc.local: Error resolving ca.pool.ntp.org: Name or service not known (-2)
Feb 12 10:13:00 idmipa02 rc.local: 12 Feb 10:13:00 ntpdate[963]: Can't find host ca.pool.ntp.org: Name or service not known (-2)
Feb 12 10:13:00 idmipa02 rc.local: 12 Feb 10:13:00 ntpdate[963]: no servers can be used, exiting

So we need to keep the time between the two masters in sync otherwise this replication issue will reoccur.  But we need to ensure our NTP servers are resolvable.  So we may need to put extra conditions in our NTP servers.  We have:

[root@idmipa01 log]# cat /etc/rc.local |grep -Evi "#"

touch /var/lock/subsys/local
ntpdate -u ca.pool.ntp.org;
[root@idmipa01 log]#

But we should use a single IP in case of failure (We are using NLB on our AD DC servers and we noted a failure on that host earlier which we just fixed.):

[root@idmipa01 log]# cat /etc/rc.local |grep -Evi "#"

touch /var/lock/subsys/local
ntpdate -u ca.pool.ntp.org || ntpdate -u 206.108.0.132 || ntpdate -u 159.203.8.72;

[root@idmipa01 log]#

This gives us some safety in case the name can't be resolved due to DNS issues.  We will also reconfigure our NTP servers as follows:

[root@idmipa02 log]# grep -Evi "#" /etc/ntp.conf | sed -e "/^$/d"
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
fudge   127.127.1.0 stratum 10
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
restrict 127.0.0.1
restrict ::1
driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log
server 0.ca.pool.ntp.org prefer
server 1.ca.pool.ntp.org
server 2.ca.pool.ntp.org
server 3.ca.pool.ntp.org

server 198.50.139.209

server 207.210.46.249
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
disable monitor
[root@idmipa02 log]#

and

[root@idmipa01 log]# grep -Evi "#" /etc/ntp.conf|sed -e "/^$/d"
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
fudge   127.127.1.0 stratum 10
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
restrict 127.0.0.1
restrict ::1
driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log

server 207.210.46.249
server 198.50.139.209
server 0.ca.pool.ntp.org
server 1.ca.pool.ntp.org
server 2.ca.pool.ntp.org
server 3.ca.pool.ntp.org prefer
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
disable monitor
[root@idmipa01 log]#

Noticed the preferred NTP servers are different on each of our NTP servers.  We're attempting to prevent a scenario where the same external NTP server is polled twice from two different servers simultaneously.  No clear evidence if this causes an issue but setting an alternate preferred server for each of our NTP servers prevents that from occurring just in case it could ever be true.  We also add 2 IP's from one the domains above in case DNS errors cause us issues.  We will be immune to this if it were ever to come up. The difference is significant:

[root@idmipa02 log]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*LOCAL(0)        .LOCL.          10 l    4   64    1    0.000    0.000   0.000
 k8s-w04.tblflp. 152.2.133.55     2 u    3   64    1   21.943  906.098   0.000
 echo.baxterit.n 213.251.128.249  2 u    2   64    1   39.255  908.220   0.000
 k8s-w01.tblflp. 152.2.133.55     2 u    1   64    1   18.415  903.549   0.000
 portal.switch.c 213.251.128.249  2 u    -   64    1   16.560  901.799   0.000
 mirror3.rafal.c .INIT.          16 u    -   64    0    0.000    0.000   0.000
 198.50.139.209  .INIT.          16 u    -   64    0    0.000    0.000   0.000
[root@idmipa02 log]#

[root@idmipa01 log]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*LOCAL(0)        .LOCL.          10 l   34   64    1    0.000    0.000   0.000
 198.50.139.209  35.73.197.144    2 u   33   64    1   19.071  -84.149   0.000
 mirror3.rafal.c 53.27.192.223    2 u   32   64    1   18.490  -56.439   0.000
 ns522433.ip-158 18.26.4.105      2 u   31   64    1   17.833  -80.900   0.000
 echo.baxterit.n 213.251.128.249  2 u   30   64    1   16.688  -82.694   0.000
 209.115.181.102 206.108.0.133    2 u   29   64    1   72.834  -82.194   0.000
 mongrel.ahem.ca .INIT.          16 u    -   64    0    0.000    0.000   0.000
[root@idmipa01 log]#

Good Luck!

Cheers,
TK

Leave a Reply

You must be logged in to post a comment.


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License