Header Shadow Image


iPhone Bricked: Update or Recovery

Had the misfortune of experiencing of doing the Apple equivalent of bricking my iPhone while doing an iOS update.  Why did I do an iOS update?  Well here's how I did this without any data loss. 

Read the rest of this entry »

Lost Thunderbird Settings and Folders

Lost Thunderbird settings and folders?  All settings in our Thunderbird were reset after a string of events including recent upgrade and some random reboots from apparent hardware issues.  Irrespective, file corruption occurred. 

Read the rest of this entry »

INFO: failed to start postgres / Permissions should be u=rwx (0700).

Getting this?

 root ? / ? data ? patroni ? systemctl status patroni
? patroni.service – Runners to orchestrate a high-availability PostgreSQL
   Loaded: loaded (/etc/systemd/system/patroni.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-07-12 17:52:48 EDT; 17s ago
 Main PID: 10991 (patroni)
   CGroup: /system.slice/patroni.service
           ??10991 /usr/bin/python2 /bin/patroni /etc/patroni.yml

Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1184        C/9E000098        no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1185        C/BE662D30        no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1186        C/DE0BD128        no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1187        C/E0577308        no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1188        C/E20393C8        no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 2020-07-12 17:53:04,277 INFO: starting as a secondary
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 2020-07-12 17:53:04,837 INFO: postmaster pid=11199
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: psql01.nix.mds.xyz:5432 – no response
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 2020-07-12 17:53:04,884 INFO: Lock owner: postgresql1; I am postgresql0
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 2020-07-12 17:53:04,891 INFO: failed to start postgres
 root ? / ? data ? patroni ?

Use this handy line to figure out why:

 root ? / ? data ? patroni ? log ? sudo su – postgres
Last login: Sun Oct 20 14:48:12 EDT 2019 on pts/0
-bash-4.2$ /usr/pgsql-10/bin/postgres -D /data/patroni –config-file=/data/patroni/postgresql.conf –listen_addresses=192.168.0.108 –max_worker_processes=8 –max_locks_per_transaction=64 –wal_level=replica –cluster_name=postgres –wal_log_hints=on –max_wal_senders=10 –track_commit_timestamp=off –max_prepared_transactions=0 –port=5432 –max_replication_slots=10 –max_connections=100 -d 5
2020-07-12 17:56:35.685 EDT [12071] FATAL:  data directory "/data/patroni" has group or world access
2020-07-12 17:56:35.685 EDT [12071] DETAIL:  Permissions should be u=rwx (0700).
2020-07-12 17:56:35.685 EDT [12071] DEBUG:  shmem_exit(1): 0 before_shmem_exit callbacks to make
2020-07-12 17:56:35.685 EDT [12071] DEBUG:  shmem_exit(1): 0 on_shmem_exit callbacks to make
2020-07-12 17:56:35.685 EDT [12071] DEBUG:  proc_exit(1): 0 callbacks to make
2020-07-12 17:56:35.685 EDT [12071] DEBUG:  exit(1)
-bash-4.2$ logout
 root ? / ? data ? patroni ? log ?

Fix it using:

 root ? / ? data ? chmod 700 patroni
 root ? / ? data ? systemctl restart patroni
 root ? / ? data ? 

Check the status:

 root ? / ? data ? systemctl status patroni
? patroni.service – Runners to orchestrate a high-availability PostgreSQL
   Loaded: loaded (/etc/systemd/system/patroni.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-07-12 17:57:10 EDT; 2min 36s ago
 Main PID: 12226 (patroni)
   CGroup: /system.slice/patroni.service
           ??12226 /usr/bin/python2 /bin/patroni /etc/patroni.yml
           ??12275 /usr/pgsql-10/bin/postgres -D /data/patroni –config-file=/data/patroni/postgresql.conf –hot_standby=on –listen_addre…
           ??12277 postgres: postgres: logger process
           ??12278 postgres: postgres: startup process   recovering 000004A50000000C000000E5
           ??12281 postgres: postgres: checkpointer process
           ??12282 postgres: postgres: writer process
           ??12283 postgres: postgres: stats collector process
           ??12287 postgres: postgres: postgres postgres 10.3.0.108(35052) idle

Jul 12 17:59:14 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:14,696 INFO: no action.  i am a secondary and i am following a leader
Jul 12 17:59:24 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:24,691 INFO: Lock owner: postgresql1; I am postgresql0
Jul 12 17:59:24 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:24,692 INFO: does not have lock
Jul 12 17:59:24 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:24,697 INFO: no action.  i am a secondary and i am following a leader
Jul 12 17:59:34 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:34,691 INFO: Lock owner: postgresql1; I am postgresql0
Jul 12 17:59:34 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:34,691 INFO: does not have lock
Jul 12 17:59:34 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:34,696 INFO: no action.  i am a secondary and i am following a leader
Jul 12 17:59:44 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:44,692 INFO: Lock owner: postgresql1; I am postgresql0
Jul 12 17:59:44 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:44,692 INFO: does not have lock
Jul 12 17:59:44 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:44,699 INFO: no action.  i am a secondary and i am following a leader
 root ? / ? data ? 

 

HTH!  

Thx,
TK

Bash: PowerLine Configuration under a User Account

Let's setup Powerline to make our prompts look like this in CentOS!

https://i2.wp.com/www.microdevsys.com/WordPressImages/powerline-configuration-introduction.JPG?ssl=1

How to do this?  Follow the following set of steps to configure this within a non privilidged user account without having to modify many target server root owned files or install any packages in the target UNIX systems.

  • Install powerline using pip3 Python 3 installer:

    [tom@mds.xyz@awx01:~] :)$ pip3 install –user powerline-status
    Collecting powerline-status
      Using cached https://files.pythonhosted.org/packages/9c/30/8bd3c62642778af9ad813a526c6ff7dd20ad6fab94ca389265/powerline-status-2.7.tar.gz
    Installing collected packages: powerline-status
      Running setup.py install for powerline-status … done
    Successfully installed powerline-status-2.7
    [tom@mds.xyz@awx01:~] :)$

     

  • Find the installed powerline direcctories. This is needed to configure .bash_profile 

    [tom@mds.xyz@awx01:~] :)$ pip3 show powerline-status
    Name: powerline-status
    Version: 2.7
    Summary: The ultimate statusline/prompt utility.
    Home-page: https://github.com/powerline/powerline
    Author: Kim Silkebaekken
    Author-email: kim.silkebaekken+vim@gmail.com
    License: MIT
    Location: /n/mds.xyz/tom/.local/lib/python3.6/site-packages
    Requires:
    [tom@mds.xyz@awx01:~] :)$

    ?

  • Next, add the following lines to your .bash_profile.  It's ok to leave the previous .bash_profile settings in place.  They'll be overwritten.

    [tom@mds.xyz@awx01:~] :)$ cat .bash_profile |tail -n5
    export PATH=$PATH:$HOME/Library/Python/2.7/bin
    powerline-daemon -q
    POWERLINE_BASH_CONTINUATION=1
    POWERLINE_BASH_SELECT=1
    . ./.local/lib/python3.6/site-packages/powerline/bindings/bash/powerline.sh
    [tom@mds.xyz@awx01:~] :)$

     

  • If running on an X Windows system, such as Gnome or KDE, install a set of fonts in the home folder of the user:

    [tom@mds.xyz@awx01:~] :)$ wget https://github.com/powerline/fonts/archive/master.zip
    [tom@mds.xyz@awx01:~] :($ unzip master.zip

    [tom@mds.xyz@awx01:~/fonts] :)$ ./install.sh
    Copying fonts…
    Powerline fonts installed to /n/mds.xyz/tom/.local/share/fonts
    [tom@mds.xyz@awx01:~/fonts] :)$

     

  • This next part occurs in Windows 10.  Grab the set of fonts below and install them in Windows 10.  https://github.com/powerline/fonts :  A few examples:

    Adding Croscore fonts for Powerline (Chrome OS core fonts)
    https://github.com/powerline/fonts/blob/master/Arimo/

    DejaVu Sans Mono for Powerline 
    https://github.com/powerline/fonts/tree/master/DejaVuSansMono

    Droid Sans Mono for Powerline
    https://github.com/powerline/fonts/tree/master/DroidSansMono
     

  • Select the installed fonts in PuTTy:

    Within PuTTy (Putty Configuration) -> Window -> Apearance -> Font settings -> Change

    Select above-installed fonts.

  • Login to a host.

  • Enjoy your new command line!

BONUS

Below is a one line ansible command to update the .bash_profile as root:

ansible 'awx01*' -i /ansible/infra -m shell -a "yum install python3 -y" -become -u root

ansible 'awx01*' -i /ansible/infra -m shell -a "pip3 install –user powerline-status" -become -u root

ansible 'awx01*' -i /ansible/infra -m shell -a "if ! grep -q powerline ~/.bash_profile; then echo -ne \"export PATH=\\\$PATH:$HOME/.local/bin/\\npowerline-daemon -q\\nPOWERLINE_BASH_CONTINUATION=1\\nPOWERLINE_BASH_SELECT=1\\n. /root/.local/lib/python3.6/site-packages/powerline/bindings/bash/powerline.sh\\n\" >> ~/.bash_profile; fi" -become -u root

Modify the host parameter to just '*' once you feel comfortable with the commands. This is how it looks when done:

https://www.microdevsys.com/WordPressImages/powerline-configuration-rootJPG

Have Fun!
TK

Cloudera: No Java JDK is detected on the host.

Getting this?  

"No Java JDK is detected on the host."  

One reason for this is a missing symlink /usr/java/latest :  

[root@cm-awn01 java]# ls -l /usr/java
total 0
drwxr-xr-x 7 root root 245 May 11 00:39 jdk1.8.0_181-cloudera  
lrwxrwxrwx 1 root root  21 May 27 13:27 latest -> jdk1.8.0_181-cloudera  
[root@cm-awn01 java]#  

GL,
SC

 

Cloudera and Azure: WrongHost: Peer certificate subjectAltName does not match host, expected , got DNS:host01.dom.com, DNS:host02.dom.com, DNS:host03.dom.com

So you're getting this while trying to connect Cloud Hosts to your local Cloudera Infrastructure?

WrongHost: Peer certificate subjectAltName does not match host, expected dhcp-100-0-0-100.remote.user.isp.com, got DNS:srv-c01.cdh.local.hst, DNS:cm-r01nn01.cdh.local.hst, DNS:cm-r01nn02.cdh.local.hst

Read the rest of this entry »

ERROR    (10 skipped) Error sending messages to firehose (retry): mgmt-HOSTMONITOR

Getting this?

[24/May/2020 23:08:13 +0000] 5385 MonitorDaemon-Reporter throttling_logger ERROR    (10 skipped) Error sending messages to firehose (retry): mgmt-HOSTMONITOR-a6c8a202b717eae93da5e0a53f184c3a
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 125, in _send
    self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
    result = self.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
    response = self.conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

modify the line slightly to see exactly what host or port it's trying:

    try:
      if self._requestor is None:
        self._transceiver = avro.ipc.HTTPTransceiver(self._address,
                                                     self._port)
        self._requestor = avro.ipc.Requestor(FIREHOSE_MESSAGE_PROTOCOL,
                                             self._transceiver)
      initial_requestor_bytes = self._requestor.get_requestor_bytes_sent()
      self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
      self._last_message_transmit_duration_gauge.set_value(
        (time.time() – start) * 1000)
      self._message_transmit_succeeded_counter.increment()
      self._requestor_bytes_sent.increment(
        self._requestor.get_requestor_bytes_sent() – initial_requestor_bytes)
      return True
    except BadStatusLine, ex:
      # We've lost our connection. In practice this usually means the server has
      # closed a connection that we expect to be open because of HTTP keep-alive.
      # We will do a single silent retry. If the problem persistest there, we'll
      # log.
      self._reset()
      if retryOnBadStatusLine:
        return self._send(messages, retryOnBadStatusLine=False)
      self._message_transmit_failed_counter.increment()
      # THROTTLED_LOG.exception("Error sending messages to firehose (retry): " +
      #                        self.name)

      THROTTLED_LOG.exception("Error sending messages to firehose (retry): %s .  Address: %s .  Port: %s" % ( self.name, self._address, self._port ))
      return False
    except Exception:
      THROTTLED_LOG.exception("Error sending messages to firehose: " + self.name)
      self._reset()
      self._message_transmit_failed_counter.increment()
      return False

Now when you start things up, you'll get some more meaningfull messages:

[24/May/2020 23:26:07 +0000] 6934 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the HOSTMONITOR.
[24/May/2020 23:26:08 +0000] 6934 MonitorDaemon-Reporter throttling_logger ERROR    Error sending messages to firehose (retry): mgmt-HOSTMONITOR-a6c8a202b717eae93da5e0a53f184c3a .  Address: cm-r01en02.mws.mds.xyz .  Port: 9995
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 125, in _send
    self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
    result = self.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
    response = self.conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''
^C
[root@cm-awn01 pki]# nc -vz cm-r01en02.mws.mds.xyz 9995
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 108.168.115.113:9995.
Ncat: 0 bytes sent, 0 bytes received in 0.05 seconds.
[root@cm-awn01 pki]#

Notice the text in blue above.  Keeping it in mind, consider this Haproxy configuration:

listen cm9995
        log                             127.0.0.1:514   local0          debug
        bind                            srv-c01:9995
        mode tcp
        option tcplog
        server cm-r01en01.mws.mds.xyz cm-r01en01.mws.mds.xyz check
        server cm-r01en02.mws.mds.xyz cm-r01en02.mws.mds.xyz check

Notice that we have TCP in the HAproxy but perhaps CMA expects HTTP?  Try setting it to HTTP:

 

ERR NSMMReplicationPlugin CSN not found, we aren’t as up to date, or we purged

Getting below errr?

May 24 13:49:11 idmipa03 ns-slapd: [24/May/2020:13:49:11.182396698 -0400] – ERR – NSMMReplicationPlugin – changelog program – repl_plugin_name_cl – agmt="cn=meToidmipa04.mws.mds.xyz" (idmipa04:389): CSN 5dd194af000000040000 not found, we aren't as up to date, or we purged
May 24 13:49:11 idmipa03 ns-slapd: [24/May/2020:13:49:11.183726430 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=meToidmipa04.mws.mds.xyz" (idmipa04:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.

or the following error?

[root@idmipa04 ~]# ipa-replica-manage force-sync –from idmipa03.mws.mds.xyz -vvv
ipa: INFO: Setting agreement cn=meToidmipa04.mws.mds.xyz,cn=replica,cn=dc\=mws\,dc\=mds\,dc\=xyz,cn=mapping tree,cn=config schedule to 2358-2359 0 to force synch
ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn=meToidmipa04.mws.mds.xyz,cn=replica,cn=dc\=mws\,dc\=mds\,dc\=xyz,cn=mapping tree,cn=config
ipa: INFO: Replication Update in progress: FALSE: status: Error (18) Replication error acquiring replica: Incremental update transient warning.  Backing off, will retry update later. (transient warning): start: 0: end: 0
[root@idmipa04 ~]#

Read the rest of this entry »

kernel: ns-slapd: segfault at ip sp error 4 in libc-2.17.so

Getting this?

kernel: ns-slapd: segfault at <ADDR> ip <ALPHA> sp <ALPHA> error 4 in libc-2.17.so

Check free memory (/var/log/dirsrv/slapd-MWS-MDS-XYZ/errors):

[root@idmipa04 slapd-MWS-MDS-XYZ]# cat errors|tail -n 30
[23/May/2020:16:33:18.519974074 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist
[23/May/2020:16:33:18.522332851 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist
[23/May/2020:16:33:18.759212393 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
[23/May/2020:16:33:18.773571691 -0400] – ERR – cos-plugin – cos_dn_defs_cb – Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mws,dc=mds,dc=xyz–no CoS Templates found, which should be added before the CoS Definition.
[23/May/2020:16:33:18.820082920 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
[23/May/2020:16:39:06.851785150 -0400] – ERR – memory allocator – malloc of 2152941454 bytes failed; OS error 12 (Cannot allocate memory)
The server has probably allocated all available virtual memory. To solve
this problem, make more virtual memory available to your server, or reduce
one or more of the following server configuration settings:
  nsslapd-cachesize        (Database Settings – Maximum entries in cache)
  nsslapd-cachememsize     (Database Settings – Memory available for cache)
  nsslapd-dbcachesize      (LDBM Plug-in Settings – Maximum cache size)
  nsslapd-import-cachesize (LDBM Plug-in Settings – Import cache size).
Can't recover; calling exit(1).

Regards,
TK

DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s

Getting this?

/var/log/ipaupgrade.log
2020-05-23T23:32:58Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-05-23T23:32:58Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 56, in run
    raise admintool.ScriptError(str(e))

?2020-05-23T23:16:22Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
2020-05-23T23:16:22Z ERROR CA did not start in 300.0s
2020-05-23T23:16:22Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

/var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host idmipa04.mws.mds.xyz port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)

It's likely because you have the following set:

[root@idmipa04 ca]# grep -Ei "nsslapd-port|nsslapd-security" /etc/dirsrv/slapd-MWS-MDS-XYZ/dse.ldif
nsslapd-port: 0
nsslapd-security: off
[root@idmipa04 ca]#

These need to be set to:

nsslapd-port: 389
nsslapd-security: on

But this did not work.  Checking certs expiration all shows dates in the future:

[root@idmipa04 ~]# getcert list|grep expires
        expires: 2021-02-05 07:37:13 UTC
        expires: 2021-02-05 07:37:42 UTC
        expires: 2021-01-25 03:22:30 UTC
        expires: 2021-01-25 03:21:37 UTC
        expires: 2021-01-25 03:21:36 UTC
        expires: 2021-01-25 03:21:37 UTC
        expires: 2039-02-05 03:21:36 UTC
        expires: 2021-01-25 07:40:56 UTC
        expires: 2021-02-05 07:42:11 UTC
[root@idmipa04 ~]#

Lastly, check for port 636 and 389 through netstat:

[root@idmipa04 pki-tomcat]# netstat -pnltu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1069/sshd
tcp        0      0 0.0.0.0:88              0.0.0.0:*               LISTEN      1089/krb5kdc
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1537/master
tcp6       0      0 :::22                   :::*                    LISTEN      1069/sshd
tcp6       0      0 :::88                   :::*                    LISTEN      1089/krb5kdc
tcp6       0      0 ::1:25                  :::*                    LISTEN      1537/master
tcp6       0      0 :::8443                 :::*                    LISTEN      16371/java
tcp6       0      0 :::443                  :::*                    LISTEN      15941/httpd
tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      16371/java
tcp6       0      0 127.0.0.1:8009          :::*                    LISTEN      16371/java
tcp6       0      0 :::8080                 :::*                    LISTEN      16371/java
tcp6       0      0 :::80                   :::*                    LISTEN      15941/httpd
udp        0      0 0.0.0.0:88              0.0.0.0:*                           1089/krb5kdc
udp6       0      0 :::88                   :::*                                1089/krb5kdc

If missing, start the directory server:

[root@idmipa04 pki-tomcat]# systemctl start dirsrv@MWS-MDS-XYZ.service

Check for the IP once started:

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 10s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-…

May 24 01:44:55 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 2
May 24 01:44:57 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:44:57.329920836 -0400] – ERR…d.
May 24 01:44:57 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:44:57.331112434 -0400] – ERR…d.
May 24 01:45:00 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:00.339593970 -0400] – ERR…d.
May 24 01:45:00 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:00.340490104 -0400] – ERR…d.
May 24 01:45:03 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:03.348216609 -0400] – ERR…d.
May 24 01:45:03 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:03.354849567 -0400] – ERR…d.
Hint: Some lines were ellipsized, use -l to show in full.
[root@idmipa04 pki-tomcat]#

Check the ports are listening:

[root@idmipa04 pki-tomcat]# netstat -pnltu|grep 18625
tcp6       0      0 :::636                  :::*                    LISTEN      18625/ns-slapd
tcp6       0      0 :::389                  :::*                    LISTEN      18625/ns-slapd

Check the error logs for the service:

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 28s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid

May 24 01:45:09 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:09.372741696 -0400] – ERR – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389) – clcache_load_buffer – Can't locate CSN 5c7bc2730000ffffffff in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
May 24 01:45:09 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:09.373677051 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389): Missing data encountered. If the error persists the replica must be reinitialized.
[root@idmipa04 pki-tomcat]#

If you see the above, reinitialize the system:

[root@idmipa04 pki-tomcat]# ipa-csreplica-manage re-initialize –from idmipa03.mws.mds.xyz
Directory Manager password:

Update in progress, 3 seconds elapsed
Update succeeded

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 4min 29s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid

May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.687759236 -0400] – WARN – NSMMReplicationPlugin – replica_reload_ruv – New data for replica o=ipaca does not match the data in the changelog.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: Recreating the changelog file. This could affect replication with replica's  consumers in which case the consumers should be reinitialized.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.721328728 -0400] – ERR – cos-plugin – cos_dn_defs_cb – Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mws,dc=mds,dc=xyz–no CoS Templates found, which should be added before the CoS Definition.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.727578549 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728113208 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding replication changelog RUV complete.  Result 0 (Success)
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728579987 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728985312 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding replication changelog RUV complete.  Result 0 (Success)
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 1
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 2
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 3

[root@idmipa04 pki-tomcat]#

Your FreeIPA server should now be back up?  Let's try that and see what happens.

/var/log/ipaupgrade.log
2020-05-24T06:00:06Z DEBUG request POST http://idmipa04.mws.mds.xyz:8080/ca/admin/ca/getStatus
2020-05-24T06:00:06Z DEBUG request body ''
2020-05-24T06:00:06Z DEBUG response status 200
2020-05-24T06:00:06Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: application/xml
Content-Length: 168
Date: Sun, 24 May 2020 06:00:06 GMT

2020-05-24T06:00:06Z DEBUG response body '<?xml version="1.0" encoding="UTF-8" standalone="no"?><XMLResponse><State>1</State><Type>CA</Type><Status>running</Status><Version>10.5.17-6.el7</Version></XMLResponse>'
2020-05-24T06:00:06Z INFO The IPA services were upgraded
2020-05-24T06:00:06Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z INFO The ipa-server-upgrade command was successful

Confirming the command now succeeded as expected:

[root@idmipa04 pki-tomcat]# ipactl start
IPA version error: data needs to be upgraded (expected version '4.6.6-11.el7.centos', current version '4.6.4-10.el7.centos.2')
Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@idmipa04 pki-tomcat]#

RELATED ERRORS:

The following errors were seen alongside the above-mentioned entries.

/var/log/ipaupgrade.log
2020-05-02T12:50:40Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1

2020-05-23T21:07:50Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

/var/log/pki/pki-tomcat/localhost.2020-05-24.log
SEVERE: Exception Processing /ca/admin/ca/getStatus
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable

SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception
org.jboss.resteasy.spi.UnhandledException: org.jboss.resteasy.core.NoMessageBodyWriterFoundFailure: Could not find MessageBodyWriter for response object of type: com.netscape.certsrv.base.PKIException$Data of media type: application/x-www-form-urlencoded

/var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host idmipa04.mws.mds.xyz port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)

/var/log/dirsrv/slapd-MWS-MDS-XYZ/errors
[24/May/2020:01:02:41.912364232 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389): Missing data encountered. If the error persists the replica must be reinitialized.

[23/May/2020:00:40:23.025920441 -0400] – ERR – set_krb5_creds – Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)

Cheers,
TK
 


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License