Patroni and ETCD: Upgrade from PostgreSQL 10 to PostgreSQL 14
Begin by installing PostgreSQL 14. In this case we're installing all PostgreSQL 14 packages.
yum install postgresql14*
Check version after installation:
[root@psql04 ~]# /usr/pgsql-14/bin/psql –version
psql (PostgreSQL) 14.5
[root@psql04 ~]#
[root@psql04 ~]# grep -Ei bin_dir /etc/patroni.yml
bin_dir: /usr/pgsql-14/bin
[root@psql04 ~]#
Restore VM's from snapshot. Yes, this is a new attempt at restoring some FreeIPA hosts that have been, ahem, neglected slightly to the point where things expired and don't work. A few unexpected reboots and FS corruption didn't help the matter either. Regardless, the recovery will in many ways show off the restoration capabilities of.FreeIPA which have certinly grew with the product. Once again we see the following in the debug logs:
# tail -f /var/log/pki/pki-tomcat/ca/debug -n 200
Could not connect to LDAP server host idmipa01.nix.mds.xyz port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
Internal Database Error encountered: Could not connect to LDAP server host idmipa01.nix.mds.xyz port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)
Use idmipa01 to fix certificates. Set idmipa01 as renewal master, if not already:
# ipa config-mod \
–ca-renewal-master-server idmipa01.nix.mds.xyz \
| grep 'CA renewal master'
Set idmipa02 as following the renewal master (idmipa01 is designated / defacto master in the cluster)
[ idmipa01 ]
# ipa config-show | grep 'IPA CA renewal master'
IPA CA renewal master: idmipa02.nix.mds.xyz
[ idmipa02 ]
# ipa config-show | grep 'IPA CA renewal master'
IPA CA renewal master: idmipa02.nix.mds.xyz
[ idmipa02 ]
# ipa config-mod \
–ca-renewal-master-server idmipa01.nix.mds.xyz \
| grep 'CA renewal master'
Once this is done, certs appear with expiration dates as below:
# getcert list|grep -Ei expire
expires: 2022-09-12 03:14:57 UTC
expires: 2020-10-03 20:04:58 UTC
expires: 2022-09-12 03:13:58 UTC
expires: 2036-11-21 07:32:02 UTC
expires: 2022-09-12 03:13:48 UTC
expires: 2022-09-12 03:13:47 UTC
expires: 2022-10-05 23:00:29 UTC
expires: 2022-10-05 23:00:59 UTC
expires: 2023-09-26 00:54:45 UTC
Start the IPA service ignoring failures:
# ipactl restart –ignore-service-failure
Follow steps on this RH blog:
https://access.redhat.com/solutions/3357261
# systemctl stop ntpd
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
do
certdate=$(date -d "`certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after | cut -d: -f2-`" +%s )
echo "$nickname – $certdate"
[[ ${newdate:-99999999999} -gt “${certdate}” ]] && newdate=$certdate
done
date –set="`date –date=@$[newdate – 86400]`"
# systemctl restart certmonger
We are greeted with the following since the site certificate is valid only in the future:
# getcert list|grep -Ei expire
expires: 2020-10-03 20:05:47 UTC
expires: 2020-10-03 20:04:58 UTC
expires: 2022-09-12 03:13:58 UTC
expires: 2036-11-21 07:32:02 UTC
expires: 2022-09-12 03:13:48 UTC
expires: 2022-09-12 03:13:47 UTC
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.).
expires: 2022-10-05 23:00:29 UTC
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.).
expires: 2022-10-05 23:00:59 UTC
expires: 2023-09-26 00:54:45 UTC
# openssl s_client -showcerts -connect idmipa01.nix.mds.xyz:443
CONNECTED(00000003)
depth=1 O = NIX.MDS.XYZ, CN = Certificate Authority
verify return:1
depth=0 O = NIX.MDS.XYZ, CN = idmipa01.nix.mds.xyz
verify error:num=9:certificate is not yet valid
notBefore=Oct 4 23:00:59 2020 GMT
verify return:1
depth=0 O = NIX.MDS.XYZ, CN = idmipa01.nix.mds.xyz
notBefore=Oct 4 23:00:59 2020 GMT
verify return:1
—
Certificate chain
0 s:/O=NIX.MDS.XYZ/CN=idmipa01.nix.mds.xyz
i:/O=NIX.MDS.XYZ/CN=Certificate Authority
[ …. ]
We notice that the date on the host was set to:
Fri Oct 2 20:12:43 EDT 2020
which is pior to the earliest date in the certificates:
# getcert list|grep -Ei expire
expires: 2020-10-03 20:05:47 UTC
expires: 2020-10-03 20:04:58 UTC
However, the Apache / HTTPD SSL Certificate is only valid after:
notBefore=Oct 4 23:00:59 2020 GMT
So we either need to update the HTTPD certificate or move the date past Oct 4th 2020. Let's set the date to Oct 4th:
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
do
certdate=$(date -d "`certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after | cut -d: -f2-`" +%s )
echo "$nickname – $certdate"
[[ ${newdate:-99999999999} -gt “${certdate}” ]] && newdate=$certdate
done
date –set="`date –date=@$[newdate + 172800]`"
Restart certmonger and check status:
# systemctl restart certmonger
check status:
# getcert list
Still expired. Did some reading:
https://frasertweedale.github.io/blog-redhat/posts/2019-05-24-ipa-cert-fix.html
and decided to try:
# ipa-cert-fix
WARNING
ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA. It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
Dogtag ca_ocsp_signing certificate:
Subject: CN=OCSP Subsystem,O=NIX.MDS.XYZ
Serial: 17
Expires: 2020-10-03 20:04:58
Dogtag ca_audit_signing certificate:
Subject: CN=CA Audit,O=NIX.MDS.XYZ
Serial: 15
Expires: 2020-10-03 20:05:47
Enter "yes" to proceed: yes
Proceeding.
Renewed Dogtag ca_ocsp_signing certificate:
Subject: CN=OCSP Subsystem,O=NIX.MDS.XYZ
Serial: 31
Expires: 2022-09-26 00:11:14
Renewed Dogtag ca_audit_signing certificate:
Subject: CN=CA Audit,O=NIX.MDS.XYZ
Serial: 32
Expires: 2022-09-26 00:12:16
Becoming renewal master.
The ipa-cert-fix command was successful
Which was apparently successful though failed to renew things:
# getcert list|grep -Ei expire
expires: 2020-10-03 20:05:47 UTC *
expires: 2020-10-03 20:04:58 UTC *
expires: 2022-09-12 03:13:58 UTC
expires: 2036-11-21 07:32:02 UTC
expires: 2022-09-12 03:13:48 UTC
expires: 2022-09-12 03:13:47 UTC
expires: 2022-10-05 23:00:29 UTC
expires: 2022-10-05 23:00:59 UTC
expires: 2023-09-26 00:54:45 UTC
Restart certmonger which now captures correct dates:
# systemctl restart certmonger
# getcert list|grep -Ei expire
expires: 2022-09-26 00:12:16 UTC *
expires: 2022-09-26 00:11:14 UTC *
expires: 2022-09-12 03:13:58 UTC
expires: 2036-11-21 07:32:02 UTC
expires: 2022-09-12 03:13:48 UTC
expires: 2022-09-12 03:13:47 UTC
expires: 2022-10-05 23:00:29 UTC
expires: 2022-10-05 23:00:59 UTC
expires: 2023-09-26 00:54:45 UTC
Restart IPA services ignoring failures in the process, while still maintaining the reset date of Oct 5th 2020 (Today is Sep 25 2022)
# ipactl restart –ignore-service-failure
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
This is the part where I realize the system was using the hwclcok date not the 'date' date (facepalm):
# date
Mon Oct 5 20:18:46 EDT 2020
# hwclock
Sun 25 Sep 2022 09:35:45 PM EDT -0.321095 seconds
Hence why cert dates came back with 2022. Whatever, let's set the date back. It may work but let's check the UI certs. Copy the openssl output certificate portions from above into a file and run the following:
# cat site-cert.pem
—–BEGIN CERTIFICATE—–
MIIEmzCCA4OgAwI…………………………………………………HIFvjW5pjp58mflhQ==
—–END CERTIFICATE—–
1 s:/O=NIX.MDS.XYZ/CN=Certificate Authority
i:/O=NIX.MDS.XYZ/CN=Certificate Authority
—–BEGIN CERTIFICATE—–
MIIDkDCC……………………………………………………………..w0T37yu7pbxM
LGclqw==
—–END CERTIFICATE—–
Check the site cert extracted from the above command:
# openssl x509 -enddate -startdate -noout -in site-cert.pem
notAfter=Oct 5 23:00:59 2022 GMT
notBefore=Oct 4 23:00:59 2020 GMT
Cert appears good until 2022 Oct 5th which we are not yet in. Let's set the date forwards a tad:
# hwclock –set –date "Fri Sep 25 21:49:00 EDT 2022"; date -s "Fri Sep 25 21:49:00 EDT 2022"
# systemctl restart ntpd
# ntpdate -s 192.168.0.12 # My NTP host.
Now try a status and a restart as well:
# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
But checking the certs again, seeing two more that are older then Sep 25 2022:
# getcert list|grep -Ei expire
expires: 2022-09-26 00:12:16 UTC
expires: 2022-09-26 00:11:14 UTC
expires: 2022-09-12 03:13:58 UTC *
expires: 2036-11-21 07:32:02 UTC
expires: 2022-09-12 03:13:48 UTC *
expires: 2022-09-12 03:13:47 UTC *
expires: 2022-10-05 23:00:29 UTC
expires: 2022-10-05 23:00:59 UTC
expires: 2023-09-26 00:54:45 UTC
Need to move the dates back again to a day prior and renew again:
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
do
certdate=$(date -d "`certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after | cut -d: -f2-`" +%s )
echo "$nickname – $certdate"
[[ ${newdate:-99999999999} -gt “${certdate}” ]] && newdate=$certdate
done
date –set="`date –date=@$[newdate + 86400]`"
Well that above command failed:
Sun Oct 4 20:04:58 EDT 2020
There is no certs with that date:
# getcert list|grep -Ei expire
expires: 2022-09-26 00:12:16 UTC
expires: 2022-09-26 00:11:14 UTC
expires: 2022-09-12 03:13:58 UTC
expires: 2036-11-21 07:32:02 UTC
expires: 2022-09-12 03:13:48 UTC
expires: 2022-09-12 03:13:47 UTC
expires: 2022-10-05 23:00:29 UTC
expires: 2022-10-05 23:00:59 UTC
expires: 2023-09-26 00:54:45 UTC
So let's try a modified copy:
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
do
certdate=$(certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after)
echo $certdate;
done
Somehow this script is basing this off of the current date? I won't reverse engineer it and set the date manually instead:
# hwclock –set –date "Fri Sep 12 01:00:00 EDT 2022"; date -s "Fri Sep 12 01:00:00 EDT 2022"
You should see certificates in submitting status now:
# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20180122053031':
status: SUBMITTING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=CA Audit,O=NIX.MDS.XYZ
expires: 2022-09-26 00:12:16 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180122053032':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=OCSP Subsystem,O=NIX.MDS.XYZ
expires: 2022-09-26 00:11:14 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180122053033':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=CA Subsystem,O=NIX.MDS.XYZ
expires: 2022-09-12 03:13:58 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180122053034':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=Certificate Authority,O=NIX.MDS.XYZ
expires: 2036-11-21 07:32:02 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180122053035':
status: SUBMITTING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=IPA RA,O=NIX.MDS.XYZ
expires: 2022-09-12 03:13:48 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20180122053036':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2022-09-12 03:13:47 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180122053037':
status: SUBMITTING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2022-10-05 23:00:29 UTC
principal name: ldap/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv NIX-MDS-XYZ
track: yes
auto-renew: yes
Request ID '20180122053042':
status: SUBMITTING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2022-10-05 23:00:59 UTC
principal name: HTTP/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20180122053135':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2023-09-26 00:54:45 UTC
principal name: krbtgt/NIX.MDS.XYZ@NIX.MDS.XYZ
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Wait a bit and check again if they were successfully processed. Or not:
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://idmipa01.nix.mds.xyz:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)).
Let's try the following command again:
# ipa-cert-fix -v
But we get this instead:
INFO: Restoring previous LDAP configuration
ERROR: Unable to find CSR for sslserver cert
ipapython.admintool: DEBUG: File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 128, in run
replicate_dogtag_certs(subject_base, ca_subject_dn, certs)
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 251, in replicate_dogtag_certs
cert = x509.load_certificate_from_file(cert_path)
File "/usr/lib/python2.7/site-packages/ipalib/x509.py", line 425, in load_certificate_from_file
with open(filename, mode='rb') as f:
ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: IOError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt'
ipapython.admintool: ERROR: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt'
ipapython.admintool: ERROR: The ipa-cert-fix command failed.
And we fix with this article:
https://access.redhat.com/solutions/4852721
Following the document steps, convert the cert accordingly:
# grep -A 19 csr /var/lib/certmonger/requests/20180122053033
csr=—–BEGIN NEW CERTIFICATE REQUEST—–
MIIDJTCCAg………………………………………………X1cWBn+CU=
—–END NEW CERTIFICATE REQUEST—–
spkac=MIICQDCCASgwgg…………….p78JfKV2/VHxXJTULg==
# vi 1.txt
# cat 1.txt
ca.subsystem.certreq=MIIDJTCCAg0CAQA…………….45oAX1cWBn+CU=
Make backups of anything you modify, whether or not you have snapshots. (Snapshots will cause you to restart from the beginning of this article 😛 )
# cp -ip /etc/pki/pki-tomcat/ca/CS.cfg /etc/pki/pki-tomcat/ca/CS.cfg-backup01
Confirm if the line exists:
# grep -Ei ca.subsystem.certreq /etc/pki/pki-tomcat/ca/CS.cfg
#
It should not otherwise you wouldn't get the above error:
Add the ca.subsystem.certreq= below the ca.subsystem.cert= line in /etc/pki/pki-tomcat/ca/CS.cfg:
# grep -Ei ca.subsystem.certreq /etc/pki/pki-tomcat/ca/CS.cfg
ca.subsystem.certreq=MIIDJTCCAg0CAQ………………………..X1cWBn+CU=
Let's try the command again:
# ipa-cert-fix
But no luck:
# ipa-cert-fix
[ ….. ]
Enter "yes" to proceed: yes
Proceeding.
[Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.crt'
The ipa-cert-fix command failed.
Let's move the dates back again, manually:
# hwclock –set –date "Fri Sep 12 01:00:00 EDT 2022"; date -s "Fri Sep 12 01:00:00 EDT 2022"
# ipa-cert-fix
[ ….. ]
Enter "yes" to proceed: yes
Proceeding.
Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –cert ca_ocsp_signing –cert ca_audit_signing –extra-cert 25' returned non-zero exit status 1
The ipa-cert-fix command failed.
did not work. So moving slightly ahead:
# hwclock –set –date "Fri Sep 13 01:00:00 EDT 2022"; date -s "Fri Sep 13 01:00:00 EDT 2022"
results in absolutely nothing. So trying with a different date:
# hwclock –set –date "Fri Sep 11 04:00:00 EDT 2022"; date -s "Fri Sep 11 04:00:00 EDT 2022"
Sun Sep 11 04:00:00 EDT 2022
Resulted in a successfully started host:
# ipactl restart –ignore-service-failure
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
Hmm, ok we're on to something here. Now that the services started fully, let's use the following:
# ipa-cert-fix
WARNING
ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA. It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
Dogtag sslserver certificate:
Subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
Serial: 22
Expires: 2022-09-12 03:13:47
Dogtag subsystem certificate:
Subject: CN=CA Subsystem,O=NIX.MDS.XYZ
Serial: 26
Expires: 2022-09-12 03:13:58
IPA IPA RA certificate:
Subject: CN=IPA RA,O=NIX.MDS.XYZ
Serial: 25
Expires: 2022-09-12 03:13:48
Enter "yes" to proceed: yes
Proceeding.
Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –extra-cert 25' returned non-zero exit status 1
The ipa-cert-fix command failed.
# pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –extra-cert 25
INFO: Loading password config: /etc/pki/pki-tomcat/password.conf
INFO: Fixing the following system certs: [‘sslserver’, ‘subsystem’]
INFO: Renewing the following additional certs: [’25’]
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Stopping the instance to proceed with system cert renewal
INFO: Configuring LDAP password authentication
INFO: Setting pkidbuser password via ldappasswd
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Selftests disabled for subsystems: ca
INFO: Resetting password for uid=ipara,ou=people,o=ipaca
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Creating a temporary sslserver cert
INFO: Getting sslserver cert info for ca
INFO: Trying to create a new temp cert for sslserver.
INFO: Generate temp SSL certificate
INFO: Getting sslserver cert info for ca
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Unable to find CSR for sslserver cert
Looks like it expects more CSR's. In this case:
Request ID '20180122053036':
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
expires: 2022-09-12 03:13:47 UTC
Doesn't have a CSR, so we add one`:
# cat /var/lib/certmonger/requests/20180122053036
csr=—–BEGIN NEW CERTIFICATE REQUEST—–
MIIDIzCCAgsCA……………………………………xQ/FFfh
Then convert it to the following one liner in the scrxipt with nickname ca.sslserver.certreq in this case:
ca.sslserver.certreq=MIIDIzCCAgsCA…………………………………………………………5XsHg07A8
But, alas, I had a copy in another cert:
csr=—–BEGIN NEW CERTIFICATE REQUEST—–
MIIDIzCC……………………………………………..kxQ/FFfh
—–END NEW CERTIFICATE REQUEST—–
# grep certreq /etc/pki/pki-tomcat/ca/CS.cfg
ca.sslserver.certreq=MIIDIzCCAgsCA……………………………………………………………………..V5XsHg07A8
NOTE the missing FFfh characters in the CSR vs what I typed in the CS.cfg. Hence got this:
# ipa-cert-fix -v
[ …………… ]
SASL SSF: 0
INFO: Creating a temporary sslserver cert
INFO: Getting sslserver cert info for ca
INFO: Trying to create a new temp cert for sslserver.
INFO: Generate temp SSL certificate
INFO: Getting sslserver cert info for ca
INFO: CSR for sslserver has been written to /tmp/tmpYQSMJk/sslserver.csr
INFO: Getting signing cert info for ca
INFO: CA cert written to /tmp/tmpYQSMJk/ca_certificate.crt
INFO: AKI: 0x1F737CF691BC6D8F93ACA3599FB6DBAB35AED71D
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Failed to generate CA-signed temp SSL certificate. RC: 255
ipapython.admintool: DEBUG: File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 117, in run
run_cert_fix(certs, extra_certs)
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 245, in run_cert_fix
ipautil.run(cmd, raiseonerr=True)
File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 563, in run
raise CalledProcessError(p.returncode, arg_string, str(output))
ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: CalledProcessError: Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –extra-cert 25' returned non-zero exit status 1
ipapython.admintool: ERROR: Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver –cert subsystem –extra-cert 25' returned non-zero exit status 1
ipapython.admintool: ERROR: The ipa-cert-fix command failed.
Editing and ensuring it's correct this time, using a one liner to properly set it up:
# grep -A 19 csr /var/lib/certmonger/requests/20180122053036|grep -v spkac|grep -v "-"|tr '\n' ' '|sed -e "s/ //g"
MIIDIzCCAgs…………………………………………………………….zHkxQ/FFfh
seams to have allowed IPA to restart properly:
# ipa-cert-fix -v
[ ………………….. ]
INFO: Starting the instance with renewed certs
Renewed Dogtag sslserver certificate:
Subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
Serial: 34
Expires: 2024-08-31 09:03:43
Renewed Dogtag subsystem certificate:
Subject: CN=CA Subsystem,O=NIX.MDS.XYZ
Serial: 35
Expires: 2024-08-31 09:03:43
Renewed IPA IPA RA certificate:
Subject: CN=IPA RA,O=NIX.MDS.XYZ
Serial: 36
Expires: 2024-08-31 09:03:44
ipalib.backend: DEBUG: Created connection context.ldap2_139668384537744
ipalib.backend: DEBUG: Destroyed connection context.ldap2_139668384537744
Becoming renewal master.
ipalib.install.sysrestore: DEBUG: Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
ipalib.install.sysrestore: DEBUG: Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=ipactl restart
ipapython.ipautil: DEBUG: Process finished, return code=0
ipapython.ipautil: DEBUG: stdout=Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipapython.ipautil: DEBUG: stderr=ipa: INFO: The ipactl command was successful
ipapython.admintool: INFO: The ipa-cert-fix command was successful
yet no change to the certs above. Trying the renew option now:
# ipa-cacert-manage renew
Following this page:
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
ran the following:
# getcert modify-ca -c dogtag-ipa-ca-renew-agent -e '/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -vv'
java.lang.Exception: Certutils.verifySystemCertValidityByNickname: faliled: nickname: ocspSigningCert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname: failed: nickname: ocspSigningCert cert-pki-ca
[11/Sep/2022:23:10:10][localhost-startStop-1]: SignedAuditLogger: event SELFTESTS_EXECUTION
[11/Sep/2022:23:10:10][localhost-startStop-1]: SelfTestSubsystem: Shutting down server due to selftest failure: Certutils.verifySystemCertValidityByNickname: faliled: nickname: ocspSigningCert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname: failed: nickname: ocspSigningCert cert-pki-ca
java.lang.Exception: Certutils.verifySystemCertValidityByNickname: faliled: nickname: ocspSigningCert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname: failed: nickname: ocspSigningCert cert-pki-ca
The below errors could have been when IPA services were stopped while the ipactl restart command was executed:
Request ID '20180122053035':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to http://idmipa01.nix.mds.xyz:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=IPA RA,O=NIX.MDS.XYZ
expires: 2022-09-12 03:13:48 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20180122053036':
status: CA_UNREACHABLE
ca-error: Error 7 connecting to http://idmipa01.nix.mds.xyz:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2022-09-12 03:13:47 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Tried resubmitting manually, perhaps the service was offline when it attempted upon ipactl restart execution:
# getcert resubmit -i 20180122053036
Resubmitting "20180122053036" to "dogtag-ipa-ca-renew-agent".
# getcert resubmit -i 20180122053035
Resubmitting "20180122053035" to "dogtag-ipa-ca-renew-agent".
And this time those two certs are ok:
Request ID '20180122053035':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=IPA RA,O=NIX.MDS.XYZ
expires: 2024-08-31 09:03:44 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20180122053036':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2024-08-31 09:03:43 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
This moves us forward to the last two:
Request ID '20180122053037':
status: CA_UNREACHABLE
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://idmipa01.nix.mds.xyz:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)).
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2022-10-05 23:00:29 UTC
principal name: ldap/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv NIX-MDS-XYZ
track: yes
auto-renew: yes
Request ID '20180122053042':
status: CA_UNREACHABLE
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://idmipa01.nix.mds.xyz:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)).
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2022-10-05 23:00:59 UTC
principal name: HTTP/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20180122053135':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2023-09-26 00:54:45 UTC
principal name: krbtgt/NIX.MDS.XYZ@NIX.MDS.XYZ
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Let's repeat the resubmission for these 2 as well. This time the error changed:
Request ID '20180122053037':
status: CA_UNREACHABLE
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 4016 (RPC failed at server. Failed to authenticate to CA REST API).
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2022-10-05 23:00:29 UTC
principal name: ldap/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv NIX-MDS-XYZ
track: yes
auto-renew: yes
Request ID '20180122053042':
status: CA_UNREACHABLE
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 4016 (RPC failed at server. Failed to authenticate to CA REST API).
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2022-10-05 23:00:59 UTC
principal name: HTTP/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Reading, this could help:
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
# ipa-certupdate
trying https://idmipa01.nix.mds.xyz/ipa/session/json
[try 1]: Forwarding 'ca_is_enabled/1' to json server 'https://idmipa01.nix.mds.xyz/ipa/session/json'
[try 1]: Forwarding 'ca_find/1' to json server 'https://idmipa01.nix.mds.xyz/ipa/session/json'
Systemwide CA database updated.
Systemwide CA database updated.
The ipa-certupdate command was successful
It appears to have done something. Let's check what that is:
Request ID '20180122053037':
status: CA_UNREACHABLE
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 4016 (RPC failed at server. Failed to authenticate to CA REST API).
Request ID '20180122053042':
status: CA_UNREACHABLE
ca-error: Server at https://idmipa01.nix.mds.xyz/ipa/xml failed request, will retry: 4016 (RPC failed at server. Failed to authenticate to CA REST API).
not much. Hmm. Running a manual resubmit appears to have done something:
# getcert resubmit -i 20180122053042
Resubmitting "20180122053042" to "IPA".
# getcert resubmit -i 20180122053037
Resubmitting "20180122053037" to "IPA".
New dates are posted for the certs, which looks promising:
Request ID '20180122053037':
status: POST_SAVED_CERT
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-NIX-MDS-XYZ/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-NIX-MDS-XYZ',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2024-09-26 05:16:52 UTC
principal name: ldap/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv NIX-MDS-XYZ
track: yes
auto-renew: yes
Request ID '20180122053042':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=NIX.MDS.XYZ
subject: CN=idmipa01.nix.mds.xyz,O=NIX.MDS.XYZ
expires: 2024-09-26 05:16:38 UTC
principal name: HTTP/idmipa01.nix.mds.xyz@NIX.MDS.XYZ
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Let's check the final result but before that, let's check the date. It seems odd that it picked 09-26 above but I don't care as long as it works properly:
# date
Mon Sep 26 01:18:19 EDT 2022
Seems ipactl restart or start did the date change hence the date of 09-26. Let's check the certs now:
# getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053031':
status: MONITORING
stuck: no
expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053032':
status: MONITORING
stuck: no
expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053033':
status: MONITORING
stuck: no
expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053034':
status: MONITORING
stuck: no
expires: 2042-09-11 09:07:22 UTC
Request ID '20180122053035':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053036':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:43 UTC
Request ID '20180122053037':
status: MONITORING
stuck: no
expires: 2024-09-26 05:16:52 UTC
Request ID '20180122053042':
status: MONITORING
stuck: no
expires: 2024-09-26 05:16:38 UTC
Request ID '20180122053135':
status: MONITORING
stuck: no
expires: 2023-09-26 00:54:45 UTC
And we are done. Seems our certs are all renewed now and our IDMIPA host is back to a working state. At least idmipa01 is! Let's fix the replica:
[idmipa01] # ipa-replica-manage list -v
idmipa01.nix.mds.xyz: master
idmipa02.nix.mds.xyz: master
[idmipa02 ] # ipa-replica-manage list -v
idmipa02.nix.mds.xyz
idmipa01.nix.mds.xyz: replica
last update status: Error (18) Replication error acquiring replica: Incremental update transient warning. Backing off, will retry update later. (transient warning)
last update ended: 1970-01-01 00:00:00+00:00
using this command:
# ipa-replica-manage re-initialize –from idmipa01.nix.mds.xyz
But alas, no, it's master/master setup:
# ipa-replica-manage re-initialize –from idmipa02.nix.mds.xyz
'idmipa02.nix.mds.xyz' has no replication agreement for 'idmipa02.nix.mds.xyz'
[idmipa01]
# sha256sum /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg
14553b94d6fad6350ce1cf2896c757657d638c8a08b250d13eac6bbacf5d3383 /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg
[ idmipa02 ]
sha256sum /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg
14553b94d6fad6350ce1cf2896c757657d638c8a08b250d13eac6bbacf5d3383 /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg
Reissue the following:
# ipa-replica-install –setup-ca –setup-dns –forwarder=192.168.0.224 /var/lib/ipa/replica-info-idmipa02.nix.mds.xyz.gpg
Your system may be partly configured.
Run /usr/sbin/ipa-server-install –uninstall to clean up.
ipapython.admintool: ERROR IPA server is already configured on this system.
If you want to reinstall the IPA server, please uninstall it first using 'ipa-server-install –uninstall'.
ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
However, it complained. Let's try to find another way. So rebooted instead to see if that will work. Nothing happened, apparently, though I did not fully check. However, running the following worked well:
[ idmipa02 ]
# ipa-replica-manage re-initialize –from idmipa01.nix.mds.xyz
Directory Manager password:
Update in progress, 4 seconds elapsed
Update succeeded
# getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
status: MONITORING
stuck: no
expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
status: MONITORING
stuck: no
expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
status: MONITORING
stuck: no
expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
status: MONITORING
stuck: no
expires: 2036-11-21 07:32:02 UTC
Request ID '20180122053642':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
status: NEED_TO_SUBMIT
stuck: no
expires: 2022-08-27 17:23:10 UTC
Request ID '20180122053644':
status: CA_UNREACHABLE
stuck: no
expires: 2022-09-29 17:22:58 UTC
Request ID '20180122053649':
status: CA_UNREACHABLE
stuck: no
expires: 2022-09-29 17:22:45 UTC
Request ID '20180122053742':
status: MONITORING
stuck: no
expires: 2023-09-26 00:54:54 UTC
And a full restart went perfectly well:
# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
But giving the above a few moments, certs still didn't update after some time. Trying to run the following:
# ipa-certupdate
This got me further but one is still unreachable with error:
# getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
status: MONITORING
stuck: no
expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
status: MONITORING
stuck: no
expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
status: MONITORING
stuck: no
expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
status: MONITORING
stuck: no
expires: 2042-09-11 09:07:22 UTC
Request ID '20180122053642':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
status: CA_UNREACHABLE
stuck: no
expires: 2022-08-27 17:23:10 UTC
Request ID '20180122053644':
status: MONITORING
stuck: no
expires: 2024-09-27 23:42:10 UTC
Request ID '20180122053649':
status: MONITORING
stuck: no
expires: 2024-09-27 23:41:58 UTC
Request ID '20180122053742':
status: MONITORING
stuck: no
expires: 2023-09-26 00:54:54 UTC
A more detailed look from getcert list :
Request ID '20180122053643':
status: CA_UNREACHABLE
ca-error: Error 60 connecting to https://idmipa02.nix.mds.xyz:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
stuck: no
Resubmit did nothing:
# getcert resubmit -i 20180122053643
Checking the CA we receive:
# ipa ca-show ipa -v
Usage: ipa [global-options] ca-show NAME [options]
# ipa ca-show ipa
ipa: ERROR: Failed to authenticate to CA REST API
Digging into the getcert list and /var/log/pki/pki-tomcat/ca/debug logs further, we get the following messages:
# getcert list
ca-error: Error 60 connecting to https://idmipa02.nix.mds.xyz:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
# /var/log/pki/pki-tomcat/ca/debug
[28/Sep/2022:00:59:32][localhost-startStop-1]: CertUtils: verifySystemCertsByTag() failed: java.lang.Exception: Certutils.verifySystemCertValidityByNickname: faliled: nickname: Server-Cert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname: failed: nickname: Server-Cert cert-pki-ca
[28/Sep/2022:00:59:32][localhost-startStop-1]: SignedAuditLogger: event CIMC_CERT_VERIFICATION
[28/Sep/2022:00:59:32][localhost-startStop-1]: SignedAuditLogger: event CIMC_CERT_VERIFICATION
java.lang.Exception: Certutils.verifySystemCertValidityByNickname: faliled: nickname: Server-Cert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname: failed: nickname: Server-Cert cert-pki-ca
[28/Sep/2022:00:59:32][localhost-startStop-1]: SelfTestSubsystem: Shutting down server due to selftest failure: Certutils.verifySystemCertValidityByNickname: faliled: nickname: Server-Cert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname: failed: nickname: Server-Cert cert-pki-ca
java.lang.Exception: Certutils.verifySystemCertValidityByNickname: faliled: nickname: Server-Cert cert-pki-cacause: java.lang.Exception: Certutils.verifySystemCertValidityByNickname: failed: nickname: Server-Cert cert-pki-ca
[28/Sep/2022:00:59:33][http-bio-8080-exec-1]: Failed to read product version String. java.io.FileNotFoundException: /usr/share/pki/CS_SERVER_VERSION (No such file or directory)
Which gives us a lead, but nothing came of that error in reading and searching. Then paid more attention and see this:
Request ID '20180122053643':
status: CA_UNREACHABLE
stuck: no
expires: 2022-08-27 17:23:10 UTC
Cert's expired. Time to roll back the clock:
# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Failed to restart httpd Service
Shutting down
Hint: You can use –ignore-service-failure option for forced start in case that a non-critical service failed
Aborting ipactl
but nope this fails. Let's try the fix command:
# ipa-cert-fix
WARNING
ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA. It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
Dogtag sslserver certificate:
Subject: CN=idmipa02.nix.mds.xyz,O=NIX.MDS.XYZ
Serial: 268369924
Expires: 2022-08-27 17:23:10
Enter "yes" to proceed: yes
Proceeding.
Command 'pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver' returned non-zero exit status 1
The ipa-cert-fix command failed.
Getting the typical CSR error:
# pki-server cert-fix –ldapi-socket /var/run/slapd-NIX-MDS-XYZ.socket –agent-uid ipara –cert sslserver
INFO: Loading password config: /etc/pki/pki-tomcat/password.conf
INFO: Fixing the following system certs: [‘sslserver’]
INFO: Renewing the following additional certs: []
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Stopping the instance to proceed with system cert renewal
INFO: Configuring LDAP password authentication
INFO: Setting pkidbuser password via ldappasswd
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Selftests disabled for subsystems: ca
INFO: Resetting password for uid=ipara,ou=people,o=ipaca
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Creating a temporary sslserver cert
INFO: Getting sslserver cert info for ca
INFO: Trying to create a new temp cert for sslserver.
INFO: Generate temp SSL certificate
INFO: Getting sslserver cert info for ca
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Unable to find CSR for sslserver cert
Let's get the CSR:
# grep -Ei "csr=" -A19 /var/lib/certmonger/requests/20180122053643 | grep -Evi "CATE REQ" | tr -d '[:space:]'
MIIDNzCCA…………………………………………….4gpgJAb+hM=
Check that you added the entry correctly:
# ca.sslserver.certreq=MIIDNzCCAh8………………………………………………………………………….gpgJAb+hM=
Try the IPA fix once more. This time we have success:
# ipa-cert-fix
WARNING
ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA. It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
Dogtag sslserver certificate:
Subject: CN=idmipa02.nix.mds.xyz,O=NIX.MDS.XYZ
Serial: 268369924
Expires: 2022-08-27 17:23:10
Enter "yes" to proceed: yes
Proceeding.
Renewed Dogtag sslserver certificate:
Subject: CN=idmipa02.nix.mds.xyz,O=NIX.MDS.XYZ
Serial: 268369929
Expires: 2024-09-17 05:32:43
The ipa-cert-fix command was successful, apparently. Restarting services to confirm:
[idmipa02] # ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[idmipa02] # getcert list|grep -Ei "Request ID|status:|stuck:|expires"
Request ID '20180122053638':
status: MONITORING
stuck: no
expires: 2024-09-15 05:15:58 UTC
Request ID '20180122053639':
status: MONITORING
stuck: no
expires: 2024-09-15 05:09:34 UTC
Request ID '20180122053640':
status: MONITORING
stuck: no
expires: 2024-09-15 05:14:47 UTC
Request ID '20180122053641':
status: MONITORING
stuck: no
expires: 2042-09-11 09:07:22 UTC
Request ID '20180122053642':
status: MONITORING
stuck: no
expires: 2024-08-31 09:03:44 UTC
Request ID '20180122053643':
status: MONITORING
stuck: no
expires: 2024-09-15 05:46:41 UTC
Request ID '20180122053644':
status: MONITORING
stuck: no
expires: 2024-09-27 23:42:10 UTC
Request ID '20180122053649':
status: MONITORING
stuck: no
expires: 2024-09-27 23:41:58 UTC
Request ID '20180122053742':
status: MONITORING
stuck: no
expires: 2023-09-26 00:54:54 UTC
And just to be sure:
[idmipa01]
ipa config-show | grep 'IPA CA renewal master'
IPA CA renewal master: idmipa01.nix.mds.xyz
[idmipa02]
ipa config-show | grep 'IPA CA renewal master'
IPA CA renewal master: idmipa01.nix.mds.xyz
Hope this helps someone!
Cheers,
Tom
REFERENCES:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/JYQU7PJGY4QV7C6S34Q7VOAAGU7FGLWF/
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/
https://access.redhat.com/solutions/3081821
https://access.redhat.com/articles/4062581
https://access.redhat.com/solutions/3357261
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authenticated-with-given-ca-certificates/
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/XSMWWPJU2VRUIGE6SRAHYAJF7BYBCNOE/
https://frasertweedale.github.io/blog-redhat/posts/2019-05-24-ipa-cert-fix.html
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/P73XKHFUJ75VHOJWK2A6ZTLZQ7I2IYE6/
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
https://access.redhat.com/solutions/4908451
https://access.redhat.com/solutions/4852721
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
https://serverfault.com/questions/709470/date-and-hwclock-not-in-sync-why
After a hard reset, a greeting of:
no healthy upstream
pops up from the vSphere Client. Login as root and issue:
service-control –start vmware-vpxd
to see if there's any additional information in regards to this error. You may or may not receive more info. Check the time and NTP settings. There's a good chance time is not synced up.
date
Login to the management console. For example:
https://vcsa01.nix.mds.xyz:5480/#/login
If it fails to login with:
Unable to login
check space with:
# df -h |grep 100
/dev/mapper/log_vg-log 9.8G 9.5G 0 100% /storage/log
Clear space by removing old log files, for example:
root@vcsa01 [ ~ ]# df -h |grep 100
/dev/mapper/log_vg-log 9.8G 9.5G 0 100% /storage/log
root@vcsa01 [ ~ ]# cd /storage/log
root@vcsa01 [ /storage/log ]# du -sh *|grep G
9.4G vmware
root@vcsa01 [ /storage/log ]# cd vmware/
root@vcsa01 [ /storage/log/vmware ]# du -sh *|grep G
1.6G eam
2.7G lookupsvc
root@vcsa01 [ /storage/log/vmware ]# cd lookupsvc/
root@vcsa01 [ /storage/log/vmware/lookupsvc ]# du -sh *|grep G
2.6G tomcat
root@vcsa01 [ /storage/log/vmware/lookupsvc ]# cd tomcat/
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]#
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]# rm -rf localhost_access.2021*
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/log_vg-log 9.8G 8.0G 1.4G 86% /storage/log
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]#
After cleaning up the space in a few more folders, reboot the appliance:
reboot
You should now be able to login after the space is freed. Try to loign again to the management console (Port 5480) to disable and enable Time Synchronization if UI still doesn't show up. If not, issue:
root@vcsa01 [ ~ ]# service-control –start applmgmt
From the vSphere Client SSH session. If you get a certificate expiration failure:
Exception in invoking authentication handler [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
Renew the self signed cert, or official certificate. Use this page:
https://kb.vmware.com/s/article/76719
Example output:
root@vcsa01 [ /tmp ]# ./fixsts.sh
NOTE: This works on external and embedded PSCs
This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain
==================================
Resetting STS certificate for vcsa01.nix.mds.xyz started on Mon Aug 1 04:23:46 UTC 2022
Detected DN: cn=vcsa01.nix.mds.xyz,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: vcsa01.nix.mds.xyz
Detected PSC: vcsa01.nix.mds.xyz
Detected SSO domain name: vsphere.local
Detected Machine ID: 310ae9cb-82a9-4fa4-bcd4-d34b054d0090
Detected IP Address: 192.168.0.33
Domain CN: dc=vsphere,dc=local
==================================
==================================
Detected Root's certificate expiration date: 2030 Jun 3
Detected today's date: 2022 Aug 1
==================================
Exporting and generating STS certificate
Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success
Enter password for administrator@vsphere.local:
Highest tenant credentials index : 1
Exporting tenant 1 to /tmp/vmware-fixsts
Deleting tenant 1
Highest trusted cert chains index: 1
Exporting trustedcertchain 1 to /tmp/vmware-fixsts
Deleting trustedcertchain 1
Applying newly generated STS certificate to SSO domain
adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
Replacement finished – Please restart services on all vCenters and PSCs in your SSO domain
==================================
IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
==================================
==================================
root@vcsa01 [ /tmp ]#
Try to login again.
Cheers,
Tom
Recent power outages and ISP outages left my network without a proper internal NTP server which was, coincidently, installed on an ESXi host. Having to revert to an external NTP server for the time being, a recent outage with my ISP highlighted the fact that even that isn't enough. The ISP outage () made it clear I need a solution that is:
So went with an OpenWRT and a Raspberry Pi device for just this very thing. This is super simple:
opkg update
opkg install ntpd
/etc/init.d/sysntpd disable
/etc/init.d/ntpd enable
/etc/init.d/ntpd start
netstat -l | grep ntp
Configure the external NTP servers to use ( NOTE: See additional information below in the EDIT section for extra iptions or the configuration will not work. ):
root@OWRT01:~# cat /etc/config/system
config system
option ttylogin '0'
option log_size '64'
option urandom_seed '0'
option hostname 'OWRT01'
option log_proto 'udp'
option conloglevel '8'
option cronloglevel '5'
option timezone 'EST5EDT,M3.2.0,M11.1.0'
option zonename 'America/Toronto'
option log_ip 192.168.0.14
option log_port 514
option log_proto udp
config timeserver 'ntp'
list server '0.ca.pool.ntp.org'
list server '1.ca.pool.ntp.org'
list server '2.ca.pool.ntp.org'
root@OWRT01:~#
Set the time manually, in the event the system can't sync it's time with an external server:
# date
# date -k
So our brand new NTP server is sitting on:
192.168.0.12
Let's now set the Date / Time to sync from this NTP server. For Cisco switches:
mdscisco01#conf t
Enter configuration commands, one per line. End with CNTL/Z.
mdscisco01(config)#ntp server 192.168.0.12
mdscisco01(config)#end
mdscisco01#show run
mdscisco01#show running-config
Ensure local time is also set correctly:
mdscisco02#
mdscisco02#conf t
Enter configuration commands, one per line. End with CNTL/Z.
mdscisco02(config)#clock timezone EST -5
mdscisco02(config)#end
mdscisco02#clock set 11:52:00 July 10 2022
mdscisco02#copy run
mdscisco02#copy running-config startup-config
Destination filename [startup-config]?
Building configuration…
Compressed configuration from 7043 bytes to 2639 bytes[OK]
mdscisco02#
For Linux Servers:
[root@mbpc-pc ~]# cat /etc/ntp.conf|grep -Eiv "^#"
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
server 192.168.0.12
server idmipa01.nix.mds.xyz prefer
server idmipa02.nix.mds.xyz prefer
server 0.rhel.pool.ntp.org iburst
server 1.rhel.pool.ntp.org iburst
server 2.rhel.pool.ntp.org iburst
server 3.rhel.pool.ntp.org iburst
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
For DD-WRT, configure via the basic config to use the OpenWRT NTP server:
Configure ESXi hosts you may have:
REF: https://openwrt.org/docs/guide-user/services/ntp/client-server
REF: https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750x_3560x/software/release/12-2_55_se/configuration/guide/3750xscg/swadmin.html
REF: https://wiki.dd-wrt.com/wiki/index.php/Network_Time_Protocol#:~:text=You%20cannot%20set%20your%20time,to%20match%20your%20local%20time.
EDIT: Aug 17 2022
An additional option is required to make this a server:
root@OWRT01:/tmp/log# cat /etc/config/system
config system
option ttylogin '0'
option log_size '64'
option urandom_seed '0'
option hostname 'OWRT01'
option log_proto 'udp'
option conloglevel '8'
option cronloglevel '5'
option timezone 'EST5EDT,M3.2.0,M11.1.0'
option zonename 'America/Toronto'
option log_ip 192.168.0.14
option log_port 514
option log_proto udp
config timeserver 'ntp'
list server '0.ca.pool.ntp.org'
list server '1.ca.pool.ntp.org'
list server '2.ca.pool.ntp.org'
option enable_server '1'
root@OWRT01:/tmp/log#
Restart the service:
root@OWRT01:/tmp/log# service ntpd restart
The above results in:
root@OWRT01:/tmp/log# cat /var/etc/ntpd.conf
driftfile /var/lib/ntp/ntp.drift
restrict default limited kod nomodify notrap nopeer
restrict -6 default limited kod nomodify notrap nopeer
restrict source noquery
# No limits for local monitoring
restrict 127.0.0.1
restrict -6 ::1
server 0.ca.pool.ntp.org iburst
server 1.ca.pool.ntp.org iburst
server 2.ca.pool.ntp.org iburst
root@OWRT01:/tmp/log#
Without the above, errors such as these will be shown:
Jan 1 03:51:00 DD-WRT-BEESWAX user.info : [ntpclient] : Network Time Protocol client trying to stop
Jan 1 03:51:00 DD-WRT-BEESWAX daemon.info process_monitor[1824]: _evalpid:ntpclient 192.168.0.12
Jan 1 03:51:00 DD-WRT-BEESWAX daemon.debug ntpclient[7036]: Connecting to 192.168.0.12 [192.168.0.12] …
Jan 1 03:51:03 DD-WRT-BEESWAX daemon.debug ntpclient[7036]: Timed out waiting for 192.168.0.12 [192.168.0.12].
Jan 1 03:51:03 DD-WRT-BEESWAX daemon.err process_monitor[1824]: cyclic NTP Update failed (servers 192.168.0.12)
Jan 1 03:51:33 DD-WRT-BEESWAX daemon.info process_monitor[1824]: _evalpid:stopservice ntpc -f
Jan 1 03:51:33 DD-WRT-BEESWAX user.info : _evalpid:/sbin/service ntpc stop
Without the server line, the NTP server will receive the request but will refuse to send the client time. A few typical messages:
root@OWRT01:~# strace -s 256 -f -p 701
[pid 701] _newselect(59, [48 49 50 51 53 54 55 56 57 58], NULL, NULL, NULL) = 1 (in [54])
[pid 701] clock_gettime(CLOCK_REALTIME, {tv_sec=1660757832, tv_nsec=325460418}) = 0
[pid 701] recvmsg(54, {msg_name={sa_family=AF_INET, sin_port=htons(34957), sin_addr=inet_addr(“192.168.0.21″)}, msg_namelen=28->16, msg_iov=[{iov_base=”\33\0\4\372\0\1\0 \0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\203\252\360=\246>\215\323”, iov_len=2120}], msg_iovlen=1, msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_TIMESTAMPNS, cmsg_data={tv_sec=1660757832, tv_nsec=324808641}}], msg_controllen=20, msg_flags=0}, 0) = 48
[pid 701] recvmsg(54, {msg_namelen=28}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 701] _newselect(59, [48 49 50 51 53 54 55 56 57 58], NULL, NULL, NULL) = 1 (in [54])
[pid 701] clock_gettime(CLOCK_REALTIME, {tv_sec=1660757833, tv_nsec=286244526}) = 0
[pid 701] recvmsg(54, {msg_name={sa_family=AF_INET, sin_port=htons(44107), sin_addr=inet_addr(“192.168.0.19″)}, msg_namelen=28->16, msg_iov=[{iov_base=”\33\0\4\372\0\1\0 \0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\203\252\202\264\200\n\215\34”, iov_len=2120}], msg_iovlen=1, msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_TIMESTAMPNS, cmsg_data={tv_sec=1660757833, tv_nsec=285534936}}], msg_controllen=20, msg_flags=0}, 0) = 48
[pid 701] recvmsg(54, {msg_namelen=28}, 0) = -1 EAGAIN (Resource temporarily unavailable)
REF: https://oldwiki.archive.openwrt.org/doc/howto/ntp.client
REF: https://forum.openwrt.org/t/ntp-server-through-busybox-ntp/91344/6
HTH
Not getting a DHCP IP? Instead getting:
169.254
169.254.149.164
The tcpdump of the traffic from the affected DD-WRT router shows the DHCP sending over a valid IP yet said IP address is not assigned to the 5Ghz WIFI band?
23:54:16.735841 bb:bb:bb:bb:bb:bb > aa:aa:aa:aa:aa:aa, ethertype 802.1Q (0x8100), length 366: vlan 1, p 0, ethertype IPv4 (0x0800), (tos 0xc0, ttl 64, id 27979, offset 0, flags [none], proto UDP (17), length 348)
192.168.0.6.67 > 192.168.0.144.68: [udp sum ok] BOOTP/DHCP, Reply, length 320, xid 0xaf0ee6f3, secs 25, Flags [none] (0x0000)
Your-IP 192.168.0.144
Server-IP 192.168.0.6
Client-Ethernet-Address aa:aa:aa:aa:aa:aa
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Server-ID (54), length 4: 192.168.0.6
Lease-Time (51), length 4: 86400
RN (58), length 4: 43200
RB (59), length 4: 75600
Subnet-Mask (1), length 4: 255.255.255.0
BR (28), length 4: 192.168.0.255
Domain-Name (15), length 7: "dom.abc"
Unknown (252), length 1: 10
Domain-Name-Server (6), length 20: 192.168.0.30,192.168.0.10,192.168.0.20,123.87.80.1,123.87.81.1
Default-Gateway (3), length 4: 192.168.0.6
23:54:17.580935 cc:cc:cc:cc:cc:cc > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 594: vlan 1, p 0, ethertype IPv4 (0x0800), (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 576)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from cc:cc:cc:cc:cc:cc, length 548, xid 0x9035706b, Flags [none] (0x0000)
Client-IP 192.168.0.210
Client-Ethernet-Address cc:cc:cc:cc:cc:cc
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Request
Parameter-Request (55), length 5:
Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), Domain-Name (15)
Hostname (12)
Hostname (12), length 10: "RokuPlayer"
23:54:25.476020 bb:bb:bb:bb:bb:bb > aa:aa:aa:aa:aa:aa, ethertype 802.1Q (0x8100), length 366: vlan 1, p 0, ethertype IPv4 (0x0800), (tos 0xc0, ttl 64, id 28795, offset 0, flags [none], proto UDP (17), length 348)
192.168.0.6.67 > 192.168.0.144.68: [udp sum ok] BOOTP/DHCP, Reply, length 320, xid 0xaf0ee6f3, secs 34, Flags [none] (0x0000)
Your-IP 192.168.0.144
Server-IP 192.168.0.6
Client-Ethernet-Address aa:aa:aa:aa:aa:aa
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Server-ID (54), length 4: 192.168.0.6
Lease-Time (51), length 4: 86400
RN (58), length 4: 43200
RB (59), length 4: 75600
Subnet-Mask (1), length 4: 255.255.255.0
BR (28), length 4: 192.168.0.255
Domain-Name (15), length 7: "dom.abc"
Unknown (252), length 1: 10
Domain-Name-Server (6), length 20: 192.168.0.30,192.168.0.10,192.168.0.20,123.87.80.1,123.87.81.1
Default-Gateway (3), length 4: 192.168.0.6
Particularly on the 5Ghz signal. In this case, leaving TurboQAM and NitroQAM enable however disabling Implicit Beamforming and Explicit Beamforming fixed the problem. Apparently a self inflicted wound. Had this disabled earlier and just recently reenabled.
Some handy commands:
tcpdump '(port 67 or port 68) and ether host aa:bb:cc:dd:ee:ff' -e -n -vv
HTH
If the Wireless setup page is missing the 5GHz or any GHz settings, or other pages are missing this info:it could be a sign of exhausted nvram. A clue to this is when resetting the nvram to defaults, all options become available again. But what about this message?
kern.warn kernel: wl driver 7.14.164.18 (r692288) failed with code 1
The above was really a red herring. The module can be checke using:
# lsmod|grep -Ei wl
wl 4420666 0
And if not loaded, use rmmod wl then modprobe wl to re add the module. These pop up and look like they could be related but they are not, in reality connected to the Wireless page not showing settings. This is also interesting since these settings don't show up even with plenty of NVRAM left. Hmm:
# nvram show >/dev/null
size: 68727 bytes (62345 left)
The culprit for the 5Ghz was too many F/W rules in DD-WRT causing (apparent) nvram exhaustion. To fix this, relocate the iptables rules to /jffs where there's plenty of storage:
To save nvram space, encode the F/W commands as follows (First command below compresses directly from the current F/W settings):
# nvram set pH_fw="$(nvram get rc_firewall | gzip | uuencode -m /dev/stdout)"
# nvram set rc_firewall="nvram get pH_fw | uudecode -o /tmp/pH_fw.gz;gunzip /tmp/pH_fw.gz;chmod +x /tmp/pH_fw;/tmp/pH_fw"
# nvram get pH_fw
begin-base64 644 /dev/stdout
H4sIAAAAAAAAA9VabXeiRhT+vr/inqbnbHJORBgxoh/aYwzZuFVC1WTbTz1E
JkqjQAE3zuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuumb7AwLId
k0Jf/+jjtQxjyw2MpwX1odaFvqY/TKDmgyIKSlNQZEGsEwI1F4KpC7U/YeHM
.
.
.
tUsYOsv6sssssssssssssssssssssssssssssssssssssssssssTUbMPl9Nh
xTHyY5gcggggggggggggggggggggggggggggggggggggggg+2NcYRZ4RnwDl
F/L7XKtQKbbbbbbbbbbbbbbbbbbbbbbbbbbbbbQrPhhhhhhhhhhJc5MVx+S3
xdpjWoFIzzzzzzzzzzzzzzzzzzzzzzkkkkkkkkzzhBDu2+ddVF7b51S2lciU
Gb50hBMJ/YPkKgb/h38AsEVI4rQ2AAA=
====
The command that will show in the UI is:
# nvram get pH_fw | uudecode -o /tmp/pH_fw.gz;gunzip /tmp/pH_fw.gz;chmod +x /tmp/pH_fw;/tmp/pH_fw
To edit the rules use:
# nvram get pH_fw|uudecode | gunzip | tee -a $(nvram get router_name)-firewall.conf
Edit the F/W rules conf file:
# vi $(nvram get router_name)-firewall.conf
Then compress once more:
# nvram set pH_fw="$(cat $(nvram get router_name)-firewall.conf | gzip | uuencode -m /dev/stdout)"
And verify:
# nvram get pH_fw
Verify in the UI that the .rc_firewall command has the compress line above. An alternative to the above, is to always compress the firewall from a file stored on /jffs. This will allow for far more storage available to firewall rules. Create a file on the /jffs/firewall folder or another folder, perhaps on your USB:
# vi /jffs/firewall/$(nvram get router_name)-firewall.run
Copy in or enter the firewall rules. Once done, set the rc_firewall nvram setting to:
# nvram set rc_firewall="time /bin/sh /jffs/firewall/$(nvram get router_name)-firewall.run"
Verify:
# nvram get rc_firewall
# time /bin/sh /jffs/firewall/$(nvram get router_name)-firewall.run
UI verification:
ISSUE
Some symptoms of low nvram are messages such as these:
dd-wrt-heimdall.the.abyss.log-20220607.gz:–
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: 2231 total pagecache pages
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: 0 pages in swap cache
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: Swap cache stats: add 0, delete 0, find 0/0
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: Free swap = 0kB
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: Total swap = 0kB
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: 131072 pages RAM
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: 98304 pages HighMem/MovableOnly
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: 2645 pages reserved
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 668] 0 668 195 39 4 0 0 0 hotplug2
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 672] 0 672 206 64 3 0 0 0 mstpd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 676] 0 676 245 139 3 0 0 0 irqbalance
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 736] 0 736 394 202 3 0 0 0 watchdog
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1060] 0 1060 363 194 3 0 0 0 syslogd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1062] 0 1062 363 170 4 0 0 0 klogd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1071] 0 1071 187 13 3 0 0 0 p910nd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1122] 0 1122 307 59 3 0 0 0 dropbear
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1501] 0 1501 424 114 5 0 0 0 ttraff
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1650] 0 1650 321 41 3 0 0 0 dhcp6c
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1676] 0 1676 367 191 4 0 0 0 watchquagga
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1678] 0 1678 379 110 4 0 0 0 wland
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1690] 0 1690 425 171 3 0 0 0 dnsmasq
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1700] 0 1700 317 173 3 0 0 0 radvd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 1701] 0 1701 317 40 3 0 0 0 radvd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2143] 0 2143 394 70 3 0 0 0 process_monitor
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2146] 0 2146 286 115 4 0 0 0 scc
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2147] 0 2147 363 88 3 0 0 0 udhcpc
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2155] 0 2155 392 223 5 0 0 0 nas
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2156] 0 2156 392 118 3 0 0 0 nas
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2157] 0 2157 392 212 4 0 0 0 nas
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2158] 0 2158 392 117 5 0 0 0 nas
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2164] 0 2164 1140 154 5 0 0 0 httpd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2165] 0 2165 329 141 3 0 0 0 resetbutton
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2265] 0 2265 248 57 4 0 0 0 usbipd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2338] 0 2338 368 72 3 0 0 0 zabbix_agentd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2340] 0 2340 368 219 3 0 0 0 zabbix_agentd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2341] 0 2341 370 207 3 0 0 0 zabbix_agentd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2342] 0 2342 370 108 3 0 0 0 zabbix_agentd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2343] 0 2343 370 207 3 0 0 0 zabbix_agentd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2344] 0 2344 370 109 3 0 0 0 zabbix_agentd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2345] 0 2345 370 108 3 0 0 0 zabbix_agentd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2410] 0 2410 188 104 4 0 0 0 cron
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [ 2461] 0 2461 454 316 3 0 0 0 ripd
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: Out of memory: Kill process 2461 (ripd) score 2 or sacrifice child
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: Killed process 2461 (ripd) total-vm:1816kB, anon-rss:392kB, file-rss:872kB
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: zabbix_agentd invoked oom-killer: gfp_mask=0x26040d0, order=0, oom_score_adj=0
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: zabbix_agentd cpuset=/ mems_allowed=0
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: CPU: 1 PID: 2340 Comm: zabbix_agentd Tainted: P 4.4.302 #5717
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: Hardware name: Northstar Prototype
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: Backtrace:
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [<80012988>] (dump_backtrace) from [<80012c0c>] (show_stack+0x18/0x1c)
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: r7:87265388 r6:20000013 r5:00000000 r4:8055e9e4
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [<80012bf4>] (show_stack) from [<80014b84>] (dump_stack+0x94/0xa8)
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: [<80014af0>] (dump_stack) from [<800140e4>] (dump_header+0x54/0x17c)
dd-wrt-heimdall.the.abyss.log-20220607.gz:Jun 5 14:52:34 dd-wrt-heimdall.the.abyss kernel: r7:87265388 r6:00000000 r5:8374bbec r4:87265380
Cheers,
REF: https://wiki.dd-wrt.com/wiki/index.php/Useful_Scripts#Compress_the_Firewall_Script_.28to_reduce_nvram_usage.29
REF: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327261
After regenerating the initramfs using dracut -f, we end up seeing the following, rather disappointing message:
r8169 0000:09:00.0: no dedicated PHY driver found for PHY ID 0x001cc912, maybe realtek.ko needs to be added to initramfs?
More specifically, in order to fix a QLogic card firmware issue on the following Linux kernel level, which foced the regeneration of the initramfs to take in the ql2400_fw.bin file once more with the updated 08.07 firmware:
Linux mbpc-pc 5.11.13 #1 SMP Sun Apr 11 21:31:14 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
On reboot however, though the FW driver worked like a charm, the network card didn't come up. Digging deeper:
[root@mbpc-pc firmware]#
Apr 10 17:17:45 mbpc-pc kernel: r8169 0000:09:00.0: no dedicated PHY driver found for PHY ID 0x001cc912, maybe realtek.ko needs to be added to initramfs?
Apr 10 17:17:45 mbpc-pc kernel: r8169: probe of 0000:09:00.0 failed with error -49
[root@mbpc-pc ~]# grep -Ei r8169 /var/log/messages
Apr 10 16:46:15 mbpc-pc kernel: libphy: r8169: probed
Apr 10 16:46:15 mbpc-pc kernel: r8169 0000:09:00.0 eth0: RTL8168d/8111d, aa:bb:cc:dd:ee:ff, XID 283, IRQ 18
Apr 10 16:46:15 mbpc-pc kernel: r8169 0000:09:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
Apr 10 16:46:15 mbpc-pc kernel: RTL8211B Gigabit Ethernet r8169-900:00: attached PHY driver (mii_bus:phy_addr=r8169-900:00, irq=IGNORE)
Apr 10 16:46:15 mbpc-pc kernel: r8169 0000:09:00.0 eth0: Link is Down
Apr 10 16:46:15 mbpc-pc kernel: r8169 0000:09:00.0 eth0: Link is Up – 1Gbps/Full – flow control off
Apr 10 17:17:45 mbpc-pc kernel: libphy: r8169: probed
Apr 10 17:17:45 mbpc-pc kernel: r8169 0000:09:00.0: no dedicated PHY driver found for PHY ID 0x001cc912, maybe realtek.ko needs to be added to initramfs?
Apr 10 17:17:45 mbpc-pc kernel: r8169: probe of 0000:09:00.0 failed with error -49
Apr 10 18:06:04 mbpc-pc kernel: libphy: r8169: probed
Apr 10 18:06:04 mbpc-pc kernel: r8169 0000:09:00.0 eth0: RTL8168d/8111d, aa:bb:cc:dd:ee:ff, XID 283, IRQ 18
Apr 10 18:06:04 mbpc-pc kernel: r8169 0000:09:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
Apr 10 18:06:04 mbpc-pc kernel: RTL8211B Gigabit Ethernet r8169-900:00: attached PHY driver (mii_bus:phy_addr=r8169-900:00, irq=IGNORE)
Apr 10 18:06:04 mbpc-pc kernel: r8169 0000:09:00.0 eth0: Link is Down
Apr 10 18:06:06 mbpc-pc kernel: r8169 0000:09:00.0 eth0: Link is Up – 1Gbps/Full – flow control off
This is a horrible way to fix things but given the fact that this is a Scientific Linux 6.X OS w/ a 5.X kernel in it, slated for a future upgrade, so be it:
[root@mbpc-pc ~]# vi /etc/rc.local
# Sometimes the realtek driver doesn't load. It's between realtek.ko and r8169. In this case, if eth0 doesn't show up, reinsert the module and log the attempt.
#
# Attempts to fix:
#
# mbpc-pc kernel: r8169 0000:09:00.0: no dedicated PHY driver found for PHY ID 0x001cc912, maybe realtek.ko needs to be added to initramfs?
# kernel: r8169: probe of 0000:09:00.0 failed with error -49
#
# This fix is only temporary for buggy kernels until a better kernel shows up that fixes this. There are alot of assumptions here.
#
if ip a|grep -Ei eth0 >/dev/null >/dev/null 2>&1; then
echo "ERROR: eth0 was not present. Removing r8169 and reinserting it again to 'fix' the problem." | tee -a /var/log/messages;
rmmod r8169;
modprobe r8169;
else
echo "GOOD: eth0 was present. Not necessary to reinsert the r8169 kernel module." | tee -a /var/log/messages;
fi
[root@mbpc-pc 5.11.13]#
Hope it works for you too!
Gnome session issues:
Apr 9 17:02:55 rfc1178-01 systemd[1]: Cannot add dependency job for unit sys-kernel-security.mount, ignoring: Unit sys-kernel-security.mount failed to load: Invalid argument. See system logs and 'systemctl status sys-kernel-security.mount' for details.
Apr 9 17:02:55 rfc1178-01 systemd[1]: Stopping Remote desktop service (VNC)…
Apr 9 17:02:55 rfc1178-01 gnome-session[2885]: gnome-session[2885]: Gdk-WARNING: gnome-session: Fatal IO error 11 (Resource temporarily unavailable) on X server :1.
Apr 9 17:02:55 rfc1178-01 gnome-session[2885]: Gdk-WARNING: gnome-session: Fatal IO error 11 (Resource temporarily unavailable) on X server :1.
Apr 9 17:02:55 rfc1178-01 gnome-session[2885]: (gnome-session-failed:2915): Gdk-WARNING **: gnome-session-failed: Fatal IO error 11 (Resource temporarily unavailable) on X server :1.
Apr 9 17:02:55 rfc1178-01 systemd[1]: Starting Remote desktop service (VNC)…
Apr 9 17:02:55 rfc1178-01 runuser[2941]: Set Bell volume to 'mute' (0).
Apr 9 17:02:55 rfc1178-01 runuser[2941]: xset: unable to open display ""
Apr 9 17:02:58 rfc1178-01 runuser[2941]: New 'rfc1178-01:1 (root)' desktop is rfc1178-01:1
Apr 9 17:02:58 rfc1178-01 runuser[2941]: Starting applications specified in /root/.vnc/xstartup
Apr 9 17:02:58 rfc1178-01 runuser[2941]: Log file is /root/.vnc/rfc1178-01:1.log
Apr 9 17:02:58 rfc1178-01 systemd[1]: Started Remote desktop service (VNC).
Apr 9 17:02:58 rfc1178-01 gnome-session[2995]: gnome-session[2995]: WARNING: Unable to find required component 'gnome-shell'
Apr 9 17:02:59 rfc1178-01 gnome-session[2995]: WARNING: Unable to find required component 'gnome-shell'
Apr 9 17:02:59 rfc1178-01 gnome-session[2995]: Entering running state
Apr 9 17:02:59 rfc1178-01 gnome-session[2995]: Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 14: reading configurations from ~/.fonts.conf is deprecated.
Fix by installing missing components:
# yum install gnome-shell.i686
Restart the VNC server:
# systemctl restart vncserver@:1
HTH,
Getting this?
[root@rhcpm03 ~]# crictl images
FATA[0000] listing images: rpc error: code = Unknown desc = layer not known
[root@rhcpm03 ~]#
Solve it with this:
mv /var/lib/containers/storage/overlay-images /root/
and reboot. Result:
[root@rhcpm03 ~]# crictl images
IMAGE TAG IMAGE ID SIZE
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> efffc0c0c9d49 272MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> cb6a3de089555 706MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> 6b15709214373 320MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> 91965ed632cd9 346MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> cf9a3590794c6 324MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> f93d86f4137d2 342MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> 1ca3399715bd7 325MB
[root@rhcpm03 ~]#
And all of a sudden, containers are listed as well:
[root@rhcpm03 member]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
3b15c78e27081 91965ed632cd91f053532cefc7d8b853a2bb5b8b6d1768c8ad473fe404b9338d 4 seconds ago Running etcd 3 173e01bafa268
da9cd9852082d 91965ed632cd91f053532cefc7d8b853a2bb5b8b6d1768c8ad473fe404b9338d 45 seconds ago Exited etcd 2 173e01bafa268
3690e2c6d4559 1ca3399715bd78d327040cdb7ab42b9a327f4d5fb051a82803530b0fad881688 55 seconds ago Exited kube-apiserver-check-endpoints 1 3acefe2582471
9622b9edb4c41 cb6a3de0895557add3a8c06853be26361d7986aa58c04c451d5f28cd606ea13a 56 seconds ago Exited kube-apiserver 1 3acefe2582471
45ec2a0aaa2b9 f93d86f4137d2ad194edecb73424a4ce373c6a1bd11e6b4588b5afe601c1553d About a minute ago Running kube-scheduler-recovery-controller 0 59483f323a340
9a4fa1095ef6c f93d86f4137d2ad194edecb73424a4ce373c6a1bd11e6b4588b5afe601c1553d About a minute ago Running kube-scheduler-cert-syncer 0 59483f323a340
cfa4fc1624ee4 cb6a3de0895557add3a8c06853be26361d7986aa58c04c451d5f28cd606ea13a About a minute ago Running kube-scheduler 0 59483f323a340
04dd78dbf1de3 91965ed632cd91f053532cefc7d8b853a2bb5b8b6d1768c8ad473fe404b9338d 2 minutes ago Running etcd-metrics 0 173e01bafa268
8c49f34fa26da 1ca3399715bd78d327040cdb7ab42b9a327f4d5fb051a82803530b0fad881688 2 minutes ago Running kube-apiserver-insecure-readyz 0 3acefe2582471
2d01ad7011aad 91965ed632cd91f053532cefc7d8b853a2bb5b8b6d1768c8ad473fe404b9338d 2 minutes ago Running etcdctl 0 173e01bafa268
2f6264de26076 1ca3399715bd78d327040cdb7ab42b9a327f4d5fb051a82803530b0fad881688 2 minutes ago Running kube-apiserver-cert-regeneration-controller 0 3acefe2582471
a632ca35c1e47 1ca3399715bd78d327040cdb7ab42b9a327f4d5fb051a82803530b0fad881688 2 minutes ago Running kube-apiserver-cert-syncer 0 3acefe2582471
6038ee56e440d cb6a3de0895557add3a8c06853be26361d7986aa58c04c451d5f28cd606ea13a 2 minutes ago Exited wait-for-host-port 0 59483f323a340
2f5afb3599e93 91965ed632cd91f053532cefc7d8b853a2bb5b8b6d1768c8ad473fe404b9338d 2 minutes ago Exited etcd-resources-copy 0 173e01bafa268
48b921c551021 cf9a3590794c69413a2d77f1c5fc3392f069412b310d522c27ad64a2fc53f2f7 2 minutes ago Running kube-controller-manager-recovery-controller 0 77ee601e357e5
ba1d1ccd4eac5 cf9a3590794c69413a2d77f1c5fc3392f069412b310d522c27ad64a2fc53f2f7 2 minutes ago Running kube-controller-manager-cert-syncer 0 77ee601e357e5
796bc713f02df 6b1570921437331098e754bec919e2682c1a2aff7aa54a25b56d1f9d3a282957 2 minutes ago Running cluster-policy-controller 0 77ee601e357e5
f1ea6c341b86c cb6a3de0895557add3a8c06853be26361d7986aa58c04c451d5f28cd606ea13a 2 minutes ago Running kube-controller-manager 0 77ee601e357e5
a4af04ce65cea cb6a3de0895557add3a8c06853be26361d7986aa58c04c451d5f28cd606ea13a 2 minutes ago Exited setup 0 3acefe2582471
3d3a45e580fc1 91965ed632cd91f053532cefc7d8b853a2bb5b8b6d1768c8ad473fe404b9338d 2 minutes ago Exited etcd-ensure-env-vars 0 173e01bafa268
[root@rhcpm03 member]#
Cheers,
Seeing this?
github: fatal: Authentication failed for
Renew the github token: Github Login -> Settings -> Developer Settings -> Personal Access Tokens
Expired today.
Use the new token!
Cheers,
Recent Entries|
Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.
|
