Restore VM's from snapshot.  Yes, this is a new attempt at restoring some FreeIPA hosts that have been, ahem, neglected slightly to the point where things expired and don't work.  A few unexpected reboots and FS corruption didn't help the matter either.  Regardless, the recovery will in many ways show off the restoration capabilities of.FreeIPA which have certinly grew with the product.  Once again we see the following in the debug logs:

# tail -f /var/log/pki/pki-tomcat/ca/debug -n 200
Could not connect to LDAP server host idmipa01.nix.mds.xyz port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
Internal Database Error encountered: Could not connect to LDAP server host idmipa01.nix.mds.xyz port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
        at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)

Use idmipa01 to fix certificates.  Set idmipa01 as renewal master, if not already:

# ipa config-mod \
      –ca-renewal-master-server idmipa01.nix.mds.xyz \
      | grep 'CA renewal master'

Set idmipa02 as following the renewal master (idmipa01 is designated / defacto master in the cluster)

[ idmipa01 ]
# ipa config-show | grep 'IPA CA renewal master'
  IPA CA renewal master: idmipa02.nix.mds.xyz

[ idmipa02 ]
# ipa config-show | grep 'IPA CA renewal master'
  IPA CA renewal master: idmipa02.nix.mds.xyz

[ idmipa02 ]
# ipa config-mod \
      –ca-renewal-master-server idmipa01.nix.mds.xyz \
      | grep 'CA renewal master'

Once this is done, certs appear with expiration dates as below:

# getcert list|grep -Ei expire
        expires: 2022-09-12 03:14:57 UTC
        expires: 2020-10-03 20:04:58 UTC
        expires: 2022-09-12 03:13:58 UTC
        expires: 2036-11-21 07:32:02 UTC
        expires: 2022-09-12 03:13:48 UTC
        expires: 2022-09-12 03:13:47 UTC
        expires: 2022-10-05 23:00:29 UTC
        expires: 2022-10-05 23:00:59 UTC
        expires: 2023-09-26 00:54:45 UTC

Start the IPA service ignoring failures:

# ipactl restart –ignore-service-failure

Follow steps on this RH blog:

https://access.redhat.com/solutions/3357261

# systemctl stop ntpd

# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
do
  certdate=$(date -d "`certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after | cut -d: -f2-`" +%s )
  echo "$nickname – $certdate"
  [[ ${newdate:-99999999999} -gt “${certdate}” ]] && newdate=$certdate
done
date –set="`date –date=@$[newdate – 86400]`"

# systemctl restart certmonger

We are greeted with the following since the site certificate is valid only in the future:

# getcert list|grep -Ei expire
        expires: 2020-10-03 20:05:47 UTC
        expires: 2020-10-03 20:04:58 UTC
        expires: 2022-09-12 03:13:58 UTC
        expires: 2036-11-21 07:32:02 UTC
        expires: 2022-09-12 03:13:48 UTC
        expires: 2022-09-12 03:13:47 UTC
        ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining:  Peer's Certificate has expired.).
        expires: 2022-10-05 23:00:29 UTC
        ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining:  Peer's Certificate has expired.).
        expires: 2022-10-05 23:00:59 UTC
        expires: 2023-09-26 00:54:45 UTC

# openssl s_client -showcerts -connect idmipa01.nix.mds.xyz:443
CONNECTED(00000003)
depth=1 O = NIX.MDS.XYZ, CN = Certificate Authority
verify return:1
depth=0 O = NIX.MDS.XYZ, CN = idmipa01.nix.mds.xyz
verify error:num=9:certificate is not yet valid
notBefore=Oct  4 23:00:59 2020 GMT
verify return:1
depth=0 O = NIX.MDS.XYZ, CN = idmipa01.nix.mds.xyz
notBefore=Oct  4 23:00:59 2020 GMT
verify return:1
—
Certificate chain
 0 s:/O=NIX.MDS.XYZ/CN=idmipa01.nix.mds.xyz
   i:/O=NIX.MDS.XYZ/CN=Certificate Authority

[ …. ]

We notice that the date on the host was set to:

Fri Oct  2 20:12:43 EDT 2020

which is pior to the earliest date in the certificates:

# getcert list|grep -Ei expire
        expires: 2020-10-03 20:05:47 UTC
        expires: 2020-10-03 20:04:58 UTC

However, the Apache / HTTPD SSL Certificate is only valid after:

notBefore=Oct  4 23:00:59 2020 GMT

So we either need to update the HTTPD certificate or move the date past Oct 4th 2020.  Let's set the date to Oct 4th: 

# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
do
  certdate=$(date -d "`certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after | cut -d: -f2-`" +%s )
  echo "$nickname – $certdate"
  [[ ${newdate:-99999999999} -gt “${certdate}” ]] && newdate=$certdate
done
date –set="`date –date=@$[newdate + 172800]`"

Restart certmonger and check status:

# systemctl restart certmonger

check status:

# getcert list

Still expired.  Did some reading:

https://frasertweedale.github.io/blog-redhat/posts/2019-05-24-ipa-cert-fix.html

and decided to try:

# ipa-cert-fix

                          WARNING

ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA.  It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.

The following certificates will be renewed:

Dogtag ca_ocsp_signing certificate:
  Subject: CN=OCSP Subsystem,O=NIX.MDS.XYZ
  Serial:  17
  Expires: 2020-10-03 20:04:58

Dogtag ca_audit_signing certificate:
  Subject: CN=CA Audit,O=NIX.MDS.XYZ
  Serial:  15
  Expires: 2020-10-03 20:05:47

Enter "yes" to proceed: yes
Proceeding.
Renewed Dogtag ca_ocsp_signing certificate:
  Subject: CN=OCSP Subsystem,O=NIX.MDS.XYZ
  Serial:  31
  Expires: 2022-09-26 00:11:14

Renewed Dogtag ca_audit_signing certificate:
  Subject: CN=CA Audit,O=NIX.MDS.XYZ
  Serial:  32
  Expires: 2022-09-26 00:12:16

Becoming renewal master.
The ipa-cert-fix command was successful

Which was apparently successful though failed to renew things:

# getcert list|grep -Ei expire
        expires: 2020-10-03 20:05:47 UTC *
        expires: 2020-10-03 20:04:58 UTC *
        expires: 2022-09-12 03:13:58 UTC
        expires: 2036-11-21 07:32:02 UTC
        expires: 2022-09-12 03:13:48 UTC
        expires: 2022-09-12 03:13:47 UTC
        expires: 2022-10-05 23:00:29 UTC
        expires: 2022-10-05 23:00:59 UTC
        expires: 2023-09-26 00:54:45 UTC

Restart certmonger which now captures correct dates:

# systemctl restart certmonger
# getcert list|grep -Ei expire
        expires: 2022-09-26 00:12:16 UTC *
        expires: 2022-09-26 00:11:14 UTC *
        expires: 2022-09-12 03:13:58 UTC
        expires: 2036-11-21 07:32:02 UTC
        expires: 2022-09-12 03:13:48 UTC
        expires: 2022-09-12 03:13:47 UTC
        expires: 2022-10-05 23:00:29 UTC
        expires: 2022-10-05 23:00:59 UTC
        expires: 2023-09-26 00:54:45 UTC

Restart IPA services ignoring failures in the process, while still maintaining the reset date of Oct 5th 2020 (Today is Sep 25 2022)

# ipactl restart –ignore-service-failure
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

This is the part where I realize the system was using the hwclcok date not the 'date' date (facepalm):

# date
Mon Oct  5 20:18:46 EDT 2020
# hwclock
Sun 25 Sep 2022 09:35:45 PM EDT  -0.321095 seconds

Hence why cert dates came back with 2022.  Whatever, let's set the date back.  It may work but let's check the UI certs.  Copy the openssl output certificate portions from above into a file and run the following:

# cat site-cert.pem
—–BEGIN CERTIFICATE—–
MIIEmzCCA4OgAwI…………………………………………………HIFvjW5pjp58mflhQ==
—–END CERTIFICATE—–
 1 s:/O=NIX.MDS.XYZ/CN=Certificate Authority
   i:/O=NIX.MDS.XYZ/CN=Certificate Authority
—–BEGIN CERTIFICATE—–
MIIDkDCC……………………………………………………………..w0T37yu7pbxM
LGclqw==
—–END CERTIFICATE—–

Check the site cert extracted from the above command:

# openssl x509 -enddate -startdate -noout -in site-cert.pem
notAfter=Oct  5 23:00:59 2022 GMT
notBefore=Oct  4 23:00:59 2020 GMT

Cert appears good until 2022 Oct 5th which we are not yet in.  Let's set the date forwards a tad:

# hwclock –set –date "Fri Sep 25 21:49:00 EDT 2022"; date -s "Fri Sep 25 21:49:00 EDT 2022"
# systemctl restart ntpd
# ntpdate -s 192.168.0.12                                                                            # My NTP host.

Now try a status and a restart as well:

# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

But checking the certs again, seeing two more that are older then Sep 25 2022:

# getcert list|grep -Ei expire
        expires: 2022-09-26 00:12:16 UTC
        expires: 2022-09-26 00:11:14 UTC 
        expires: 2022-09-12 03:13:58 UTC *
        expires: 2036-11-21 07:32:02 UTC
        expires: 2022-09-12 03:13:48 UTC *
        expires: 2022-09-12 03:13:47 UTC *
        expires: 2022-10-05 23:00:29 UTC
        expires: 2022-10-05 23:00:59 UTC
        expires: 2023-09-26 00:54:45 UTC

Need to move the dates back again to a day prior and renew again:

# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
do
  certdate=$(date -d "`certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after | cut -d: -f2-`" +%s )
  echo "$nickname – $certdate"
  [[ ${newdate:-99999999999} -gt “${certdate}” ]] && newdate=$certdate
done
date –set="`date –date=@$[newdate + 86400]`"

Well that above command failed:

Sun Oct  4 20:04:58 EDT 2020

There is no certs with that date:

# getcert list|grep -Ei expire
        expires: 2022-09-26 00:12:16 UTC
        expires: 2022-09-26 00:11:14 UTC
        expires: 2022-09-12 03:13:58 UTC
        expires: 2036-11-21 07:32:02 UTC
        expires: 2022-09-12 03:13:48 UTC
        expires: 2022-09-12 03:13:47 UTC
        expires: 2022-10-05 23:00:29 UTC
        expires: 2022-10-05 23:00:59 UTC
        expires: 2023-09-26 00:54:45 UTC

So let's try a modified copy:

# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
do
  certdate=$(certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after)
  echo $certdate;
done

Somehow this script is basing this off of the current date?  I won't reverse engineer it and set the date manually instead:

# hwclock –set –date "Fri Sep 12 01:00:00 EDT 2022"; date -s "Fri Sep 12 01:00:00 EDT 2022"

You should see certificates in submitting status now:

# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20180122053031':
        status: SUBMITTING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=CA Audit,O=NIX.MDS.XYZ
        expires: 2022-09-26 00:12:16 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20180122053032':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=OCSP Subsystem,O=NIX.MDS.XYZ
        expires: 2022-09-26 00:11:14 UTC
        eku: id-kp-OCSPSigning
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20180122053033':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=CA Subsystem,O=NIX.MDS.XYZ
        expires: 2022-09-12 03:13:58 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20180122053034':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=Certificate Authority,O=NIX.MDS.XYZ
        expires: 2036-11-21 07:32:02 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20180122053035':
        status: SUBMITTING
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=IPA RA,O=NIX.MDS.XYZ
        expires: 2022-09-12 03:13:48 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20180122053036':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2022-09-12 03:13:47 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20180122053037':
        status: SUBMITTING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2022-10-05 23:00:29 UTC
        principal name: ldap/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv NIX-MDS-XYZ
        track: yes
        auto-renew: yes
Request ID '20180122053042':
        status: SUBMITTING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2022-10-05 23:00:59 UTC
        principal name: HTTP/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
Request ID '20180122053135':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: SelfSign
        issuer: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2023-09-26 00:54:45 UTC
        principal name: krbtgt/NIX.MDS.XYZ@NIX.MDS.XYZ
        certificate template/profile: KDCs_PKINIT_Certs
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
        track: yes
        auto-renew: yes

Wait a bit and check again if they were successfully processed.  Or not:

ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 907 (RPC failed at server.  cannot connect to 'https://idmipa01.nix.mds.xyz:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)).

Let's try the following command again:

# ipa-cert-fix -v

But we get this instead:

INFO: Restoring previous LDAP configuration
ERROR: Unable to find CSR for sslserver cert

ipapython.admintool: DEBUG:   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 128, in run
    replicate_dogtag_certs(subject_base, ca_subject_dn, certs)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 251, in replicate_dogtag_certs
    cert = x509.load_certificate_from_file(cert_path)
  File "/usr/lib/python2.7/site-packages/ipalib/x509.py", line 425, in load_certificate_from_file
    with open(filename, mode='rb') as f:

ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: IOError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt'
ipapython.admintool: ERROR: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt'
ipapython.admintool: ERROR: The ipa-cert-fix command failed.

And we fix with this article:

https://access.redhat.com/solutions/4852721

Following the document steps, convert the cert accordingly:

# grep -A 19 csr /var/lib/certmonger/requests/20180122053033
csr=—–BEGIN NEW CERTIFICATE REQUEST—–
 MIIDJTCCAg………………………………………………X1cWBn+CU=
 —–END NEW CERTIFICATE REQUEST—–
spkac=MIICQDCCASgwgg…………….p78JfKV2/VHxXJTULg==
# vi 1.txt

# cat 1.txt
ca.subsystem.certreq=MIIDJTCCAg0CAQA…………….45oAX1cWBn+CU=

Make backups of anything you modify, whether or not you have snapshots. (Snapshots will cause you to restart from the beginning of this article 😛 )

# cp -ip /etc/pki/pki-tomcat/ca/CS.cfg /etc/pki/pki-tomcat/ca/CS.cfg-backup01

Confirm if the line exists:

# grep -Ei ca.subsystem.certreq /etc/pki/pki-tomcat/ca/CS.cfg
# 

It should not otherwise you wouldn't get the above error:

Add the ca.subsystem.certreq= below the ca.subsystem.cert= line in /etc/pki/pki-tomcat/ca/CS.cfg:

# grep -Ei ca.subsystem.certreq /etc/pki/pki-tomcat/ca/CS.cfg
ca.subsystem.certreq=MIIDJTCCAg0CAQ………………………..X1cWBn+CU=

Let's try the command again:

# ipa-cert-fix

But no luck:

# ipa-cert-fix
[ ….. ]
Enter "yes" to proceed: yes
Proceeding.
[Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt'
The ipa-cert-fix command failed.

Let's move the dates back again, manually:

# hwclock –set –date "Fri Sep 12 01:00:00 EDT 2022"; date -s "Fri Sep 12 01:00:00 EDT 2022"

# ipa-cert-fix
[ ….. ]
Enter "yes" to proceed: yes
Proceeding.
Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –cert ca_ocsp_signing –cert ca_audit_signing –extra-cert 25' returned non-zero exit status 1
The ipa-cert-fix command failed.

did not work.  So moving slightly ahead:

# hwclock –set –date "Fri Sep 13 01:00:00 EDT 2022"; date -s "Fri Sep 13 01:00:00 EDT 2022"

results in absolutely nothing.  So trying with a different date:

# hwclock –set –date "Fri Sep 11 04:00:00 EDT 2022"; date -s "Fri Sep 11 04:00:00 EDT 2022"
Sun Sep 11 04:00:00 EDT 2022

Resulted in a successfully started host:

# ipactl restart –ignore-service-failure
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

Hmm, ok we're on to something here.  Now that the services started fully, let's use the following:

# ipa-cert-fix

                          WARNING

ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA.  It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.

The following certificates will be renewed:

Dogtag sslserver certificate:
  Subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
  Serial:  22
  Expires: 2022-09-12 03:13:47

Dogtag subsystem certificate:
  Subject: CN=CA Subsystem,O=NIX.MDS.XYZ
  Serial:  26
  Expires: 2022-09-12 03:13:58

IPA IPA RA certificate:
  Subject: CN=IPA RA,O=NIX.MDS.XYZ
  Serial:  25
  Expires: 2022-09-12 03:13:48

Enter "yes" to proceed: yes
Proceeding.
Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –extra-cert 25' returned non-zero exit status 1
The ipa-cert-fix command failed.

# pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –extra-cert 25
INFO: Loading password config: /etc/pki/pki-tomcat/password.conf
INFO: Fixing the following system certs: [‘sslserver’, ‘subsystem’]
INFO: Renewing the following additional certs: [’25’]
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Stopping the instance to proceed with system cert renewal
INFO: Configuring LDAP password authentication
INFO: Setting pkidbuser password via ldappasswd
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Selftests disabled for subsystems: ca
INFO: Resetting password for uid=ipara,ou=people,o=ipaca
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Creating a temporary sslserver cert
INFO: Getting sslserver cert info for ca
INFO: Trying to create a new temp cert for sslserver.
INFO: Generate temp SSL certificate
INFO: Getting sslserver cert info for ca
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Unable to find CSR for sslserver cert

Looks like it expects more CSR's.  In this case:

Request ID '20180122053036':
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        expires: 2022-09-12 03:13:47 UTC

Doesn't have a CSR, so we add one`:

# cat /var/lib/certmonger/requests/20180122053036
csr=—–BEGIN NEW CERTIFICATE REQUEST—–
 MIIDIzCCAgsCA……………………………………xQ/FFfh

Then convert it to the following one liner in the scrxipt with nickname ca.sslserver.certreq in this case:

ca.sslserver.certreq=MIIDIzCCAgsCA…………………………………………………………5XsHg07A8

But, alas, I had a copy in another cert:

csr=—–BEGIN NEW CERTIFICATE REQUEST—–
 MIIDIzCC……………………………………………..kxQ/FFfh
 —–END NEW CERTIFICATE REQUEST—–

# grep certreq /etc/pki/pki-tomcat/ca/CS.cfg
ca.sslserver.certreq=MIIDIzCCAgsCA……………………………………………………………………..V5XsHg07A8

NOTE the missing FFfh characters in the CSR vs what I typed in the CS.cfg.  Hence got this:

# ipa-cert-fix -v
[ …………… ]
SASL SSF: 0
INFO: Creating a temporary sslserver cert
INFO: Getting sslserver cert info for ca
INFO: Trying to create a new temp cert for sslserver.
INFO: Generate temp SSL certificate
INFO: Getting sslserver cert info for ca
INFO: CSR for sslserver has been written to /tmp/tmpYQSMJk/sslserver.csr
INFO: Getting signing cert info for ca
INFO: CA cert written to /tmp/tmpYQSMJk/ca_certificate.crt
INFO: AKI: 0x1F737CF691BC6D8F93ACA3599FB6DBAB35AED71D
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Failed to generate CA-signed temp SSL certificate. RC: 255

ipapython.admintool: DEBUG:   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 117, in run
    run_cert_fix(certs, extra_certs)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 245, in run_cert_fix
    ipautil.run(cmd, raiseonerr=True)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 563, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))

ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: CalledProcessError: Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –extra-cert 25' returned non-zero exit status 1
ipapython.admintool: ERROR: Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –extra-cert 25' returned non-zero exit status 1
ipapython.admintool: ERROR: The ipa-cert-fix command failed.

Editing and ensuring it's correct this time, using a one liner to properly set it up:

# grep -A 19 csr /var/lib/certmonger/requests/20180122053036|grep -v spkac|grep -v "-"|tr '\n' ' '|sed -e "s/ //g"
MIIDIzCCAgs…………………………………………………………….zHkxQ/FFfh

seams to have allowed IPA to restart properly:

# ipa-cert-fix -v
[ ………………….. ]
INFO: Starting the instance with renewed certs

Renewed Dogtag sslserver certificate:
  Subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
  Serial:  34
  Expires: 2024-08-31 09:03:43

Renewed Dogtag subsystem certificate:
  Subject: CN=CA Subsystem,O=NIX.MDS.XYZ
  Serial:  35
  Expires: 2024-08-31 09:03:43

Renewed IPA IPA RA certificate:
  Subject: CN=IPA RA,O=NIX.MDS.XYZ
  Serial:  36
  Expires: 2024-08-31 09:03:44

ipalib.backend: DEBUG: Created connection context.ldap2_139668384537744
ipalib.backend: DEBUG: Destroyed connection context.ldap2_139668384537744
Becoming renewal master.
ipalib.install.sysrestore: DEBUG: Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
ipalib.install.sysrestore: DEBUG: Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=ipactl restart
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service

ipapython.ipautil: DEBUG: stderr=ipa: INFO: The ipactl command was successful

ipapython.admintool: INFO: The ipa-cert-fix command was successful

yet no change to the certs above.  Trying the renew option now:

# ipa-cacert-manage renew

Following this page:

https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/

ran the following:

# getcert modify-ca -c dogtag-ipa-ca-renew-agent -e '/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -vv'

java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  faliled: nickname: ocspSigningCert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  failed: nickname: ocspSigningCert cert-pki-ca

[11/Sep/2022:23:10:10][localhost-startStop-1]: SignedAuditLogger: event SELFTESTS_EXECUTION
[11/Sep/2022:23:10:10][localhost-startStop-1]: SelfTestSubsystem: Shutting down server due to selftest failure: Certutils.verifySystemCertValidityByNickname:  faliled: nickname: ocspSigningCert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  failed: nickname: ocspSigningCert cert-pki-ca
java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  faliled: nickname: ocspSigningCert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  failed: nickname: ocspSigningCert cert-pki-ca

The below errors could have been when IPA services were stopped while the ipactl restart command was executed:

Request ID '20180122053035':
        status: CA_UNREACHABLE
        ca-error: Error 7 connecting to http://idmipa01.nix.mds.xyz:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=IPA RA,O=NIX.MDS.XYZ
        expires: 2022-09-12 03:13:48 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20180122053036':
        status: CA_UNREACHABLE
        ca-error: Error 7 connecting to http://idmipa01.nix.mds.xyz:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2022-09-12 03:13:47 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes

Tried resubmitting manually, perhaps the service was offline when it attempted upon ipactl restart execution:

# getcert resubmit -i 20180122053036
Resubmitting "20180122053036" to "dogtag-ipa-ca-renew-agent".

# getcert resubmit -i 20180122053035
Resubmitting "20180122053035" to "dogtag-ipa-ca-renew-agent".

And this time those two certs are ok:

Request ID '20180122053035':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=IPA RA,O=NIX.MDS.XYZ
        expires: 2024-08-31 09:03:44 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20180122053036':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2024-08-31 09:03:43 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes

This moves us forward to the last two:

Request ID '20180122053037':
        status: CA_UNREACHABLE
        ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 907 (RPC failed at server.  cannot connect to 'https://idmipa01.nix.mds.xyz:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2022-10-05 23:00:29 UTC
        principal name: ldap/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv NIX-MDS-XYZ
        track: yes
        auto-renew: yes
Request ID '20180122053042':
        status: CA_UNREACHABLE
        ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 907 (RPC failed at server.  cannot connect to 'https://idmipa01.nix.mds.xyz:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2022-10-05 23:00:59 UTC
        principal name: HTTP/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
Request ID '20180122053135':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: SelfSign
        issuer: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2023-09-26 00:54:45 UTC
        principal name: krbtgt/NIX.MDS.XYZ@NIX.MDS.XYZ
        certificate template/profile: KDCs_PKINIT_Certs
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
        track: yes
        auto-renew: yes

Let's repeat the resubmission for these 2 as well.  This time the error changed:

Request ID '20180122053037':
        status: CA_UNREACHABLE
        ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 4016 (RPC failed at server.  Failed to authenticate to CA REST API).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2022-10-05 23:00:29 UTC
        principal name: ldap/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv NIX-MDS-XYZ
        track: yes
        auto-renew: yes
Request ID '20180122053042':
        status: CA_UNREACHABLE
        ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 4016 (RPC failed at server.  Failed to authenticate to CA REST API).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2022-10-05 23:00:59 UTC
        principal name: HTTP/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes

Reading, this could help:

https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/

# ipa-certupdate
trying https://idmipa01.nix.mds.xyz/ipa/session/json
[try 1]: Forwarding 'ca_is_enabled/1' to json server 'https://idmipa01.nix.mds.xyz/ipa/session/json'
[try 1]: Forwarding 'ca_find/1' to json server 'https://idmipa01.nix.mds.xyz/ipa/session/json'
Systemwide CA database updated.
Systemwide CA database updated.
The ipa-certupdate command was successful

It appears to have done something.  Let's check what that is:

Request ID '20180122053037':
        status: CA_UNREACHABLE
        ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 4016 (RPC failed at server.  Failed to authenticate to CA REST API).

Request ID '20180122053042':
        status: CA_UNREACHABLE
        ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 4016 (RPC failed at server.  Failed to authenticate to CA REST API).

not much.  Hmm.  Running a manual resubmit appears to have done something:

# getcert resubmit -i 20180122053042
Resubmitting "20180122053042" to "IPA".

# getcert resubmit -i 20180122053037
Resubmitting "20180122053037" to "IPA".

New dates are posted for the certs, which looks promising:

Request ID '20180122053037':
        status: POST_SAVED_CERT
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2024-09-26 05:16:52 UTC
        principal name: ldap/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv NIX-MDS-XYZ
        track: yes
        auto-renew: yes
Request ID '20180122053042':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
        subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
        expires: 2024-09-26 05:16:38 UTC
        principal name: HTTP/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes

Let's check the final result but before that, let's check the date.  It seems odd that it picked 09-26 above but I don't care as long as it works properly:

# date
Mon Sep 26 01:18:19 EDT 2022

Seems ipactl restart or start did the date change hence the date of 09-26.  Let's check the certs now:

# getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053031':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053032':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053033':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053034':
        status: MONITORING
        stuck: no
        expires: 2042-09-11 09:07:22 UTC
Request ID '20180122053035':
        status: MONITORING
        stuck: no
        expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053036':
        status: MONITORING
        stuck: no
        expires: 2024-08-31 09:03:43 UTC
Request ID '20180122053037':
        status: MONITORING
        stuck: no
        expires: 2024-09-26 05:16:52 UTC
Request ID '20180122053042':
        status: MONITORING
        stuck: no
        expires: 2024-09-26 05:16:38 UTC
Request ID '20180122053135':
        status: MONITORING
        stuck: no
        expires: 2023-09-26 00:54:45 UTC

And we are done.  Seems our certs are all renewed now and our IDMIPA host is back to a working state.  At least idmipa01 is!  Let's fix the replica:

[idmipa01] # ipa-replica-manage list -v
idmipa01.nix.mds.xyz: master
idmipa02.nix.mds.xyz: master

[idmipa02 ] # ipa-replica-manage list -v
idmipa02.nix.mds.xyz
idmipa01.nix.mds.xyz: replica
  last update status: Error (18) Replication error acquiring replica: Incremental update transient warning.  Backing off, will retry update later. (transient warning)
  last update ended: 1970-01-01 00:00:00+00:00

using this command:

# ipa-replica-manage re-initialize –from idmipa01.nix.mds.xyz

But alas, no, it's master/master setup:

# ipa-replica-manage re-initialize –from idmipa02.nix.mds.xyz
'idmipa02.nix.mds.xyz' has no replication agreement for 'idmipa02.nix.mds.xyz'

[idmipa01]
# sha256sum /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg
14553b94d6fad6350ce1cf2896c757657d638c8a08b250d13eac6bbacf5d3383  /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg

[ idmipa02 ]
sha256sum /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg
14553b94d6fad6350ce1cf2896c757657d638c8a08b250d13eac6bbacf5d3383  /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg

Reissue the following:

# ipa-replica-install –setup-ca –setup-dns –forwarder=192.168.0.224 /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg
Your system may be partly configured.
Run /usr/sbin/ipa-server-install –uninstall to clean up.

ipapython.admintool: ERROR    IPA server is already configured on this system.
If you want to reinstall the IPA server, please uninstall it first using 'ipa-server-install –uninstall'.
ipapython.admintool: ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

However, it complained. Let's try to find another way.  So rebooted instead to see if that will work.  Nothing happened, apparently, though I did not fully check.  However, running the following worked well:

[ idmipa02 ]

# ipa-replica-manage re-initialize –from idmipa01.nix.mds.xyz
Directory Manager password:

Update in progress, 4 seconds elapsed
Update succeeded

# getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
        status: MONITORING
        stuck: no
        expires: 2036-11-21 07:32:02 UTC
Request ID '20180122053642':
        status: MONITORING
        stuck: no
        expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
        status: NEED_TO_SUBMIT
        stuck: no
        expires: 2022-08-27 17:23:10 UTC
Request ID '20180122053644':
        status: CA_UNREACHABLE
        stuck: no
        expires: 2022-09-29 17:22:58 UTC
Request ID '20180122053649':
        status: CA_UNREACHABLE
        stuck: no
        expires: 2022-09-29 17:22:45 UTC
Request ID '20180122053742':
        status: MONITORING
        stuck: no
        expires: 2023-09-26 00:54:54 UTC

And a full restart went perfectly well:

# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

But giving the above a few moments, certs still didn't update after some time.  Trying to run the following:

# ipa-certupdate

This got me further but one is still unreachable with error:

# getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
        status: MONITORING
        stuck: no
        expires: 2042-09-11 09:07:22 UTC
Request ID '20180122053642':
        status: MONITORING
        stuck: no
        expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
        status: CA_UNREACHABLE
        stuck: no
        expires: 2022-08-27 17:23:10 UTC
Request ID '20180122053644':
        status: MONITORING
        stuck: no
        expires: 2024-09-27 23:42:10 UTC
Request ID '20180122053649':
        status: MONITORING
        stuck: no
        expires: 2024-09-27 23:41:58 UTC
Request ID '20180122053742':
        status: MONITORING
        stuck: no
        expires: 2023-09-26 00:54:54 UTC

A more detailed look from getcert list :

Request ID '20180122053643':
        status: CA_UNREACHABLE
        ca-error: Error 60 connecting to https://idmipa02.nix.mds.xyz:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
        stuck: no

Resubmit did nothing:

# getcert resubmit -i 20180122053643

Checking the CA we receive:

# ipa ca-show ipa  -v
Usage: ipa [global-options] ca-show NAME [options]

# ipa ca-show ipa
ipa: ERROR: Failed to authenticate to CA REST API

Digging into the getcert list and /var/log/pki/pki-tomcat/ca/debug logs further, we get the following messages:

# getcert list
ca-error: Error 60 connecting to https://idmipa02.nix.mds.xyz:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.

# /var/log/pki/pki-tomcat/ca/debug

[28/Sep/2022:00:59:32][localhost-startStop-1]: CertUtils: verifySystemCertsByTag() failed: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  faliled: nickname: Server-Cert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  failed: nickname: Server-Cert cert-pki-ca
[28/Sep/2022:00:59:32][localhost-startStop-1]: SignedAuditLogger: event CIMC_CERT_VERIFICATION
[28/Sep/2022:00:59:32][localhost-startStop-1]: SignedAuditLogger: event CIMC_CERT_VERIFICATION
java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  faliled: nickname: Server-Cert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  failed: nickname: Server-Cert cert-pki-ca

[28/Sep/2022:00:59:32][localhost-startStop-1]: SelfTestSubsystem: Shutting down server due to selftest failure: Certutils.verifySystemCertValidityByNickname:  faliled: nickname: Server-Cert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  failed: nickname: Server-Cert cert-pki-ca
java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  faliled: nickname: Server-Cert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname:  failed: nickname: Server-Cert cert-pki-ca

[28/Sep/2022:00:59:33][http-bio-8080-exec-1]: Failed to read product version String. java.io.FileNotFoundException: /usr/share/pki/CS_SERVER_VERSION (No such file or directory)

Which gives us a lead, but nothing came of that error in reading and searching.  Then paid more attention and see this:

Request ID '20180122053643':
        status: CA_UNREACHABLE
        stuck: no
        expires: 2022-08-27 17:23:10 UTC        

Cert's expired.  Time to roll back the clock:

# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Failed to restart httpd Service
Shutting down
Hint: You can use –ignore-service-failure option for forced start in case that a non-critical service failed
Aborting ipactl

but nope this fails.  Let's try the fix command:

# ipa-cert-fix

                          WARNING

ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA.  It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.

The following certificates will be renewed:

Dogtag sslserver certificate:
  Subject: CN=idmipa02.nix.mds.xyz,O=NIX.MDS.XYZ
  Serial:  268369924
  Expires: 2022-08-27 17:23:10

Enter "yes" to proceed: yes
Proceeding.
Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver' returned non-zero exit status 1
The ipa-cert-fix command failed.

Getting the typical CSR error:

# pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver
INFO: Loading password config: /etc/pki/pki-tomcat/password.conf
INFO: Fixing the following system certs: [‘sslserver’]
INFO: Renewing the following additional certs: []
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Stopping the instance to proceed with system cert renewal
INFO: Configuring LDAP password authentication
INFO: Setting pkidbuser password via ldappasswd
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Selftests disabled for subsystems: ca
INFO: Resetting password for uid=ipara,ou=people,o=ipaca
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Creating a temporary sslserver cert
INFO: Getting sslserver cert info for ca
INFO: Trying to create a new temp cert for sslserver.
INFO: Generate temp SSL certificate
INFO: Getting sslserver cert info for ca
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Unable to find CSR for sslserver cert

Let's get the CSR:

# grep -Ei "csr=" -A19 /var/lib/certmonger/requests/20180122053643 | grep -Evi "CATE REQ" | tr -d '[:space:]'
MIIDNzCCA…………………………………………….4gpgJAb+hM=

Check that you added the entry correctly:

# ca.sslserver.certreq=MIIDNzCCAh8………………………………………………………………………….gpgJAb+hM=

Try the IPA fix once more.  This time we have success:

# ipa-cert-fix

                          WARNING

ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA.  It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.

The following certificates will be renewed:

Dogtag sslserver certificate:
  Subject: CN=idmipa02.nix.mds.xyz,O=NIX.MDS.XYZ
  Serial:  268369924
  Expires: 2022-08-27 17:23:10

Enter "yes" to proceed: yes
Proceeding.
Renewed Dogtag sslserver certificate:
  Subject: CN=idmipa02.nix.mds.xyz,O=NIX.MDS.XYZ
  Serial:  268369929
  Expires: 2024-09-17 05:32:43

The ipa-cert-fix command was successful, apparently.  Restarting services to confirm:

[idmipa02] # ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[idmipa02] # getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
        status: MONITORING
        stuck: no
        expires: 2042-09-11 09:07:22 UTC
Request ID '20180122053642':
        status: MONITORING
        stuck: no
        expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
        status: MONITORING
        stuck: no
        expires: 2024-09-15 05:46:41 UTC
Request ID '20180122053644':
        status: MONITORING
        stuck: no
        expires: 2024-09-27 23:42:10 UTC
Request ID '20180122053649':
        status: MONITORING
        stuck: no
        expires: 2024-09-27 23:41:58 UTC
Request ID '20180122053742':
        status: MONITORING
        stuck: no
        expires: 2023-09-26 00:54:54 UTC

And just to be sure:

[idmipa01]
ipa config-show | grep 'IPA CA renewal master'
  IPA CA renewal master: idmipa01.nix.mds.xyz
  
[idmipa02]
ipa config-show | grep 'IPA CA renewal master'
  IPA CA renewal master: idmipa01.nix.mds.xyz 

Hope this helps someone!

Cheers,
Tom

REFERENCES:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/JYQU7PJGY4QV7C6S34Q7VOAAGU7FGLWF/
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/
https://access.redhat.com/solutions/3081821
https://access.redhat.com/articles/4062581
https://access.redhat.com/solutions/3357261
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authenticated-with-given-ca-certificates/
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/XSMWWPJU2VRUIGE6SRAHYAJF7BYBCNOE/
https://frasertweedale.github.io/blog-redhat/posts/2019-05-24-ipa-cert-fix.html
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/P73XKHFUJ75VHOJWK2A6ZTLZQ7I2IYE6/
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
https://access.redhat.com/solutions/4908451
https://access.redhat.com/solutions/4852721
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
https://serverfault.com/questions/709470/date-and-hwclock-not-in-sync-why