iPhone Bricked: Update or Recovery
Had the misfortune of experiencing of doing the Apple equivalent of bricking my iPhone while doing an iOS update. Why did I do an iOS update? Well here's how I did this without any data loss.
Lost Thunderbird Settings and Folders
Lost Thunderbird settings and folders? All settings in our Thunderbird were reset after a string of events including recent upgrade and some random reboots from apparent hardware issues. Irrespective, file corruption occurred.
INFO: failed to start postgres / Permissions should be u=rwx (0700).
Getting this?
root ? / ? data ? patroni ? systemctl status patroni
? patroni.service – Runners to orchestrate a high-availability PostgreSQL
Loaded: loaded (/etc/systemd/system/patroni.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2020-07-12 17:52:48 EDT; 17s ago
Main PID: 10991 (patroni)
CGroup: /system.slice/patroni.service
??10991 /usr/bin/python2 /bin/patroni /etc/patroni.yml
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1184 C/9E000098 no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1185 C/BE662D30 no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1186 C/DE0BD128 no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1187 C/E0577308 no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 1188 C/E20393C8 no recovery target specified
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 2020-07-12 17:53:04,277 INFO: starting as a secondary
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 2020-07-12 17:53:04,837 INFO: postmaster pid=11199
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: psql01.nix.mds.xyz:5432 – no response
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 2020-07-12 17:53:04,884 INFO: Lock owner: postgresql1; I am postgresql0
Jul 12 17:53:04 psql01.nix.mds.xyz patroni[10991]: 2020-07-12 17:53:04,891 INFO: failed to start postgres
root ? / ? data ? patroni ?
Use this handy line to figure out why:
root ? / ? data ? patroni ? log ? sudo su – postgres
Last login: Sun Oct 20 14:48:12 EDT 2019 on pts/0
-bash-4.2$ /usr/pgsql-10/bin/postgres -D /data/patroni –config-file=/data/patroni/postgresql.conf –listen_addresses=192.168.0.108 –max_worker_processes=8 –max_locks_per_transaction=64 –wal_level=replica –cluster_name=postgres –wal_log_hints=on –max_wal_senders=10 –track_commit_timestamp=off –max_prepared_transactions=0 –port=5432 –max_replication_slots=10 –max_connections=100 -d 5
2020-07-12 17:56:35.685 EDT [12071] FATAL: data directory "/data/patroni" has group or world access
2020-07-12 17:56:35.685 EDT [12071] DETAIL: Permissions should be u=rwx (0700).
2020-07-12 17:56:35.685 EDT [12071] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make
2020-07-12 17:56:35.685 EDT [12071] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make
2020-07-12 17:56:35.685 EDT [12071] DEBUG: proc_exit(1): 0 callbacks to make
2020-07-12 17:56:35.685 EDT [12071] DEBUG: exit(1)
-bash-4.2$ logout
root ? / ? data ? patroni ? log ?
Fix it using:
root ? / ? data ? chmod 700 patroni
root ? / ? data ? systemctl restart patroni
root ? / ? data ?
Check the status:
root ? / ? data ? systemctl status patroni
? patroni.service – Runners to orchestrate a high-availability PostgreSQL
Loaded: loaded (/etc/systemd/system/patroni.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2020-07-12 17:57:10 EDT; 2min 36s ago
Main PID: 12226 (patroni)
CGroup: /system.slice/patroni.service
??12226 /usr/bin/python2 /bin/patroni /etc/patroni.yml
??12275 /usr/pgsql-10/bin/postgres -D /data/patroni –config-file=/data/patroni/postgresql.conf –hot_standby=on –listen_addre…
??12277 postgres: postgres: logger process
??12278 postgres: postgres: startup process recovering 000004A50000000C000000E5
??12281 postgres: postgres: checkpointer process
??12282 postgres: postgres: writer process
??12283 postgres: postgres: stats collector process
??12287 postgres: postgres: postgres postgres 10.3.0.108(35052) idle
Jul 12 17:59:14 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:14,696 INFO: no action. i am a secondary and i am following a leader
Jul 12 17:59:24 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:24,691 INFO: Lock owner: postgresql1; I am postgresql0
Jul 12 17:59:24 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:24,692 INFO: does not have lock
Jul 12 17:59:24 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:24,697 INFO: no action. i am a secondary and i am following a leader
Jul 12 17:59:34 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:34,691 INFO: Lock owner: postgresql1; I am postgresql0
Jul 12 17:59:34 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:34,691 INFO: does not have lock
Jul 12 17:59:34 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:34,696 INFO: no action. i am a secondary and i am following a leader
Jul 12 17:59:44 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:44,692 INFO: Lock owner: postgresql1; I am postgresql0
Jul 12 17:59:44 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:44,692 INFO: does not have lock
Jul 12 17:59:44 psql01.nix.mds.xyz patroni[12226]: 2020-07-12 17:59:44,699 INFO: no action. i am a secondary and i am following a leader
root ? / ? data ?
HTH!
Thx,
TK
Bash: PowerLine Configuration under a User Account
Let's setup Powerline to make our prompts look like this in CentOS!
How to do this? Follow the set of steps below to configure this within a non-privilidged user account without having to modify many target server root-owned files or install any packages in the target UNIX systems.
-
Install gcc and glibc. This will ensure powerline is compiled into a binary, not the slow python version.
-
(Recommended) Install powerline using pip3 but sourcing the files from Github:
[tom@mds.xyz@awx01:~] :($ pip3 install --user git+https://github.com/powerline/powerline.git@master Collecting git+https://github.com/powerline/powerline.git@master Cloning https://github.com/powerline/powerline.git (to master) to /tmp/pip-moyw5ssr-build Installing collected packages: powerline-status Running setup.py install for powerline-status ... done Successfully installed powerline-status-2.7 [tom@mds.xyz@awx01:~] :)$This gives the best chance that the C ELF binary get's compiled. Python version, per discussion with the maintainers, is slow due to the Python3 Socket Implementation in the code.
-
(Deprecated) Install powerline using pip3 Python 3 installer:
[tom@mds.xyz@awx01:~] :)$ pip3 install –user powerline-status
Collecting powerline-status
Using cached https://files.pythonhosted.org/packages/9c/30/8bd3c62642778af9ad813a526c6ff7dd20ad6fab94ca389265/powerline-status-2.7.tar.gz
Installing collected packages: powerline-status
Running setup.py install for powerline-status … done
Successfully installed powerline-status-2.7
[tom@mds.xyz@awx01:~] :)$
-
Find the installed powerline direcctories. This is needed to configure .bash_profile
[tom@mds.xyz@awx01:~] :)$ pip3 show powerline-status
Name: powerline-status
Version: 2.7
Summary: The ultimate statusline/prompt utility.
Home-page: https://github.com/powerline/powerline
Author: Kim Silkebaekken
Author-email: kim.silkebaekken+vim@gmail.com
License: MIT
Location: /n/mds.xyz/tom/.local/lib/python3.6/site-packages
Requires:
[tom@mds.xyz@awx01:~] :)$
? -
Next, add the following lines to your .bash_profile. It's ok to leave the previous .bash_profile settings in place. They'll be overwritten.
[tom@mds.xyz@awx01:~] :)$ cat .bash_profile |tail -n5
export PATH=$PATH:$HOME/Library/Python/2.7/bin
powerline-daemon -q
POWERLINE_BASH_CONTINUATION=1
POWERLINE_BASH_SELECT=1
. ./.local/lib/python3.6/site-packages/powerline/bindings/bash/powerline.sh
[tom@mds.xyz@awx01:~] :)$
-
If running on an X Windows system, such as Gnome or KDE, install a set of fonts in the home folder of the user:
[tom@mds.xyz@awx01:~] :)$ wget https://github.com/powerline/fonts/archive/master.zip
[tom@mds.xyz@awx01:~] :($ unzip master.zip[tom@mds.xyz@awx01:~/fonts] :)$ ./install.sh
Copying fonts…
Powerline fonts installed to /n/mds.xyz/tom/.local/share/fonts
[tom@mds.xyz@awx01:~/fonts] :)$
-
This next part occurs in Windows 10. Grab the set of fonts below and install them in Windows 10. https://github.com/powerline/fonts : A few examples:
Adding Croscore fonts for Powerline (Chrome OS core fonts)
https://github.com/powerline/fonts/blob/master/Arimo/DejaVu Sans Mono for Powerline
https://github.com/powerline/fonts/tree/master/DejaVuSansMonoDroid Sans Mono for Powerline
https://github.com/powerline/fonts/tree/master/DroidSansMono
-
Select the installed fonts in PuTTy:
Within PuTTy (Putty Configuration) -> Window -> Apearance -> Font settings -> Change
Select above-installed fonts.
-
Login to a host.
-
Enjoy your new command line!
BONUS
Below is a one line ansible command to update the .bash_profile as root:
ansible 'awx01*' -i /ansible/infra -m shell -a "yum install python3 -y" -become -u root
ansible 'awx01*' -i /ansible/infra -m shell -a "pip3 install –user powerline-status" -become -u root
ansible 'awx01*' -i /ansible/infra -m shell -a "if ! grep -q powerline ~/.bash_profile; then echo -ne \"export PATH=\\\$PATH:$HOME/.local/bin/\\npowerline-daemon -q\\nPOWERLINE_BASH_CONTINUATION=1\\nPOWERLINE_BASH_SELECT=1\\n. /root/.local/lib/python3.6/site-packages/powerline/bindings/bash/powerline.sh\\n\" >> ~/.bash_profile; fi" -become -u root
Modify the host parameter to just '*' once you feel comfortable with the commands. This is how it looks when done:
Have Fun!
TK
Cloudera: No Java JDK is detected on the host.
Getting this?
"No Java JDK is detected on the host."
One reason for this is a missing symlink /usr/java/latest :
[root@cm-awn01 java]# ls -l /usr/java
total 0
drwxr-xr-x 7 root root 245 May 11 00:39 jdk1.8.0_181-cloudera
lrwxrwxrwx 1 root root 21 May 27 13:27 latest -> jdk1.8.0_181-cloudera
[root@cm-awn01 java]#
GL,
SC
Cloudera and Azure: WrongHost: Peer certificate subjectAltName does not match host, expected , got DNS:host01.dom.com, DNS:host02.dom.com, DNS:host03.dom.com
So you're getting this while trying to connect Cloud Hosts to your local Cloudera Infrastructure?
WrongHost: Peer certificate subjectAltName does not match host, expected dhcp-100-0-0-100.remote.user.isp.com, got DNS:srv-c01.cdh.local.hst, DNS:cm-r01nn01.cdh.local.hst, DNS:cm-r01nn02.cdh.local.hst
ERROR (10 skipped) Error sending messages to firehose (retry): mgmt-HOSTMONITOR
Getting this?
[24/May/2020 23:08:13 +0000] 5385 MonitorDaemon-Reporter throttling_logger ERROR (10 skipped) Error sending messages to firehose (retry): mgmt-HOSTMONITOR-a6c8a202b717eae93da5e0a53f184c3a
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 125, in _send
self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
result = self.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
response = self.conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
modify the line slightly to see exactly what host or port it's trying:
try:
if self._requestor is None:
self._transceiver = avro.ipc.HTTPTransceiver(self._address,
self._port)
self._requestor = avro.ipc.Requestor(FIREHOSE_MESSAGE_PROTOCOL,
self._transceiver)
initial_requestor_bytes = self._requestor.get_requestor_bytes_sent()
self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
self._last_message_transmit_duration_gauge.set_value(
(time.time() – start) * 1000)
self._message_transmit_succeeded_counter.increment()
self._requestor_bytes_sent.increment(
self._requestor.get_requestor_bytes_sent() – initial_requestor_bytes)
return True
except BadStatusLine, ex:
# We've lost our connection. In practice this usually means the server has
# closed a connection that we expect to be open because of HTTP keep-alive.
# We will do a single silent retry. If the problem persistest there, we'll
# log.
self._reset()
if retryOnBadStatusLine:
return self._send(messages, retryOnBadStatusLine=False)
self._message_transmit_failed_counter.increment()
# THROTTLED_LOG.exception("Error sending messages to firehose (retry): " +
# self.name)
THROTTLED_LOG.exception("Error sending messages to firehose (retry): %s . Address: %s . Port: %s" % ( self.name, self._address, self._port ))
return False
except Exception:
THROTTLED_LOG.exception("Error sending messages to firehose: " + self.name)
self._reset()
self._message_transmit_failed_counter.increment()
return False
Now when you start things up, you'll get some more meaningfull messages:
[24/May/2020 23:26:07 +0000] 6934 MonitorDaemon-Reporter firehoses INFO Creating a connection to the HOSTMONITOR.
[24/May/2020 23:26:08 +0000] 6934 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose (retry): mgmt-HOSTMONITOR-a6c8a202b717eae93da5e0a53f184c3a . Address: cm-r01en02.mws.mds.xyz . Port: 9995
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 125, in _send
self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
result = self.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
response = self.conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
^C
[root@cm-awn01 pki]# nc -vz cm-r01en02.mws.mds.xyz 9995
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 108.168.115.113:9995.
Ncat: 0 bytes sent, 0 bytes received in 0.05 seconds.
[root@cm-awn01 pki]#
Notice the text in blue above. Keeping it in mind, consider this Haproxy configuration:
listen cm9995
log 127.0.0.1:514 local0 debug
bind srv-c01:9995
mode tcp
option tcplog
server cm-r01en01.mws.mds.xyz cm-r01en01.mws.mds.xyz check
server cm-r01en02.mws.mds.xyz cm-r01en02.mws.mds.xyz check
Notice that we have TCP in the HAproxy but perhaps CMA expects HTTP? Try setting it to HTTP:
ERR NSMMReplicationPlugin CSN not found, we aren’t as up to date, or we purged
Getting below errr?
May 24 13:49:11 idmipa03 ns-slapd: [24/May/2020:13:49:11.182396698 -0400] – ERR – NSMMReplicationPlugin – changelog program – repl_plugin_name_cl – agmt="cn=meToidmipa04.mws.mds.xyz" (idmipa04:389): CSN 5dd194af000000040000 not found, we aren't as up to date, or we purged
May 24 13:49:11 idmipa03 ns-slapd: [24/May/2020:13:49:11.183726430 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=meToidmipa04.mws.mds.xyz" (idmipa04:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
or the following error?
[root@idmipa04 ~]# ipa-replica-manage force-sync –from idmipa03.mws.mds.xyz -vvv
ipa: INFO: Setting agreement cn=meToidmipa04.mws.mds.xyz,cn=replica,cn=dc\=mws\,dc\=mds\,dc\=xyz,cn=mapping tree,cn=config schedule to 2358-2359 0 to force synch
ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn=meToidmipa04.mws.mds.xyz,cn=replica,cn=dc\=mws\,dc\=mds\,dc\=xyz,cn=mapping tree,cn=config
ipa: INFO: Replication Update in progress: FALSE: status: Error (18) Replication error acquiring replica: Incremental update transient warning. Backing off, will retry update later. (transient warning): start: 0: end: 0
[root@idmipa04 ~]#
kernel: ns-slapd: segfault at ip sp error 4 in libc-2.17.so
Getting this?
kernel: ns-slapd: segfault at <ADDR> ip <ALPHA> sp <ALPHA> error 4 in libc-2.17.so
Check free memory (/var/log/dirsrv/slapd-MWS-MDS-XYZ/errors):
[root@idmipa04 slapd-MWS-MDS-XYZ]# cat errors|tail -n 30
[23/May/2020:16:33:18.519974074 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist
[23/May/2020:16:33:18.522332851 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist
[23/May/2020:16:33:18.759212393 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
[23/May/2020:16:33:18.773571691 -0400] – ERR – cos-plugin – cos_dn_defs_cb – Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mws,dc=mds,dc=xyz–no CoS Templates found, which should be added before the CoS Definition.
[23/May/2020:16:33:18.820082920 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
[23/May/2020:16:39:06.851785150 -0400] – ERR – memory allocator – malloc of 2152941454 bytes failed; OS error 12 (Cannot allocate memory)
The server has probably allocated all available virtual memory. To solve
this problem, make more virtual memory available to your server, or reduce
one or more of the following server configuration settings:
nsslapd-cachesize (Database Settings – Maximum entries in cache)
nsslapd-cachememsize (Database Settings – Memory available for cache)
nsslapd-dbcachesize (LDBM Plug-in Settings – Maximum cache size)
nsslapd-import-cachesize (LDBM Plug-in Settings – Import cache size).
Can't recover; calling exit(1).
Regards,
TK
DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
Getting this?
/var/log/ipaupgrade.log
2020-05-23T23:32:58Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-05-23T23:32:58Z DEBUG File “/usr/lib/python2.7/site-packages/ipapython/admintool.py”, line 178, in execute
return_value = self.run()
File “/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py”, line 56, in run
raise admintool.ScriptError(str(e))
?2020-05-23T23:16:22Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
2020-05-23T23:16:22Z ERROR CA did not start in 300.0s
2020-05-23T23:16:22Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
/var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host idmipa04.mws.mds.xyz port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)
It’s likely because you have the following set:
[root@idmipa04 ca]# grep -Ei “nsslapd-port|nsslapd-security” /etc/dirsrv/slapd-MWS-MDS-XYZ/dse.ldif
nsslapd-port: 0
nsslapd-security: off
[root@idmipa04 ca]#
These need to be set to:
nsslapd-port: 389
nsslapd-security: onBut this did not work. Checking certs expiration all shows dates in the future:
[root@idmipa04 ~]# getcert list|grep expires
expires: 2021-02-05 07:37:13 UTC
expires: 2021-02-05 07:37:42 UTC
expires: 2021-01-25 03:22:30 UTC
expires: 2021-01-25 03:21:37 UTC
expires: 2021-01-25 03:21:36 UTC
expires: 2021-01-25 03:21:37 UTC
expires: 2039-02-05 03:21:36 UTC
expires: 2021-01-25 07:40:56 UTC
expires: 2021-02-05 07:42:11 UTC
[root@idmipa04 ~]#
Lastly, check for port 636 and 389 through netstat:
[root@idmipa04 pki-tomcat]# netstat -pnltu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1069/sshd
tcp 0 0 0.0.0.0:88 0.0.0.0:* LISTEN 1089/krb5kdc
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1537/master
tcp6 0 0 :::22 :::* LISTEN 1069/sshd
tcp6 0 0 :::88 :::* LISTEN 1089/krb5kdc
tcp6 0 0 ::1:25 :::* LISTEN 1537/master
tcp6 0 0 :::8443 :::* LISTEN 16371/java
tcp6 0 0 :::443 :::* LISTEN 15941/httpd
tcp6 0 0 127.0.0.1:8005 :::* LISTEN 16371/java
tcp6 0 0 127.0.0.1:8009 :::* LISTEN 16371/java
tcp6 0 0 :::8080 :::* LISTEN 16371/java
tcp6 0 0 :::80 :::* LISTEN 15941/httpd
udp 0 0 0.0.0.0:88 0.0.0.0:* 1089/krb5kdc
udp6 0 0 :::88 :::* 1089/krb5kdc
If missing, start the directory server:
[root@idmipa04 pki-tomcat]# systemctl start dirsrv@MWS-MDS-XYZ.service
Check for the IP once started:
[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 10s ago
Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
Main PID: 18625 (ns-slapd)
Status: “slapd started: Ready to process requests”
CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-…
May 24 01:44:55 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 2
May 24 01:44:57 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:44:57.329920836 -0400] – ERR…d.
May 24 01:44:57 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:44:57.331112434 -0400] – ERR…d.
May 24 01:45:00 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:00.339593970 -0400] – ERR…d.
May 24 01:45:00 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:00.340490104 -0400] – ERR…d.
May 24 01:45:03 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:03.348216609 -0400] – ERR…d.
May 24 01:45:03 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:03.354849567 -0400] – ERR…d.
Hint: Some lines were ellipsized, use -l to show in full.
[root@idmipa04 pki-tomcat]#
Check the ports are listening:
[root@idmipa04 pki-tomcat]# netstat -pnltu|grep 18625
tcp6 0 0 :::636 :::* LISTEN 18625/ns-slapd
tcp6 0 0 :::389 :::* LISTEN 18625/ns-slapd
Check the error logs for the service:
[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 28s ago
Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
Main PID: 18625 (ns-slapd)
Status: “slapd started: Ready to process requests”
CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 24 01:45:09 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:09.372741696 -0400] – ERR – agmt=”cn=caToidmipa03.mws.mds.xyz” (idmipa03:389) – clcache_load_buffer – Can’t locate CSN 5c7bc2730000ffffffff in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
May 24 01:45:09 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:09.373677051 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt=”cn=caToidmipa03.mws.mds.xyz” (idmipa03:389): Missing data encountered. If the error persists the replica must be reinitialized.
[root@idmipa04 pki-tomcat]#
If you see the above, reinitialize the system:
[root@idmipa04 pki-tomcat]# ipa-csreplica-manage re-initialize –from idmipa03.mws.mds.xyz
Directory Manager password:
Update in progress, 3 seconds elapsed
Update succeeded
[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 4min 29s ago
Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
Main PID: 18625 (ns-slapd)
Status: “slapd started: Ready to process requests”
CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.687759236 -0400] – WARN – NSMMReplicationPlugin – replica_reload_ruv – New data for replica o=ipaca does not match the data in the changelog.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: Recreating the changelog file. This could affect replication with replica’s consumers in which case the consumers should be reinitialized.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.721328728 -0400] – ERR – cos-plugin – cos_dn_defs_cb – Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mws,dc=mds,dc=xyz–no CoS Templates found, which should be added before the CoS Definition.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.727578549 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728113208 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding replication changelog RUV complete. Result 0 (Success)
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728579987 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728985312 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding replication changelog RUV complete. Result 0 (Success)
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 1
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 2
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 3
[root@idmipa04 pki-tomcat]#
Your FreeIPA server should now be back up? Let’s try that and see what happens.
/var/log/ipaupgrade.log
2020-05-24T06:00:06Z DEBUG request POST http://idmipa04.mws.mds.xyz:8080/ca/admin/ca/getStatus
2020-05-24T06:00:06Z DEBUG request body ”
2020-05-24T06:00:06Z DEBUG response status 200
2020-05-24T06:00:06Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: application/xml
Content-Length: 168
Date: Sun, 24 May 2020 06:00:06 GMT
2020-05-24T06:00:06Z DEBUG response body ‘<?xml version=”1.0″ encoding=”UTF-8″ standalone=”no”?><XMLResponse><State>1</State><Type>CA</Type><Status>running</Status><Version>10.5.17-6.el7</Version></XMLResponse>’
2020-05-24T06:00:06Z INFO The IPA services were upgraded
2020-05-24T06:00:06Z DEBUG Loading StateFile from ‘/var/lib/ipa/sysupgrade/sysupgrade.state’
2020-05-24T06:00:06Z DEBUG Saving StateFile to ‘/var/lib/ipa/sysupgrade/sysupgrade.state’
2020-05-24T06:00:06Z DEBUG Loading StateFile from ‘/var/lib/ipa/sysupgrade/sysupgrade.state’
2020-05-24T06:00:06Z DEBUG Saving StateFile to ‘/var/lib/ipa/sysupgrade/sysupgrade.state’
2020-05-24T06:00:06Z INFO The ipa-server-upgrade command was successful
Confirming the command now succeeded as expected:
[root@idmipa04 pki-tomcat]# ipactl start
IPA version error: data needs to be upgraded (expected version ‘4.6.6-11.el7.centos’, current version ‘4.6.4-10.el7.centos.2’)
Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@idmipa04 pki-tomcat]#
RELATED ERRORS:
The following errors were seen alongside the above-mentioned entries.
/var/log/ipaupgrade.log
2020-05-02T12:50:40Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command ‘/bin/systemctl start dirsrv@MWS-MDS-XYZ.service’ returned non-zero exit status 1
2020-05-23T21:07:50Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
/var/log/pki/pki-tomcat/localhost.2020-05-24.log
SEVERE: Exception Processing /ca/admin/ca/getStatus
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception
org.jboss.resteasy.spi.UnhandledException: org.jboss.resteasy.core.NoMessageBodyWriterFoundFailure: Could not find MessageBodyWriter for response object of type: com.netscape.certsrv.base.PKIException$Data of media type: application/x-www-form-urlencoded
/var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host idmipa04.mws.mds.xyz port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)
/var/log/dirsrv/slapd-MWS-MDS-XYZ/errors
[24/May/2020:01:02:41.912364232 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt=”cn=caToidmipa03.mws.mds.xyz” (idmipa03:389): Missing data encountered. If the error persists the replica must be reinitialized.
[23/May/2020:00:40:23.025920441 -0400] – ERR – set_krb5_creds – Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
Cheers,
TK
