Header Shadow Image


vmware: no healthy upstream

After a hard reset, a greeting of:

no healthy upstream

pops up from the vSphere Client.  Login as root and issue:

service-control –start vmware-vpxd

to see if there's any  additional information in regards to this error.  You may  or may not receive more info.  Check the time and NTP settings. There's a good chance time is not synced up.

date

Login to the management console.  For example:

https://vcsa01.nix.mds.xyz:5480/#/login

If it fails to login with:

Unable to login

check space with:

# df -h |grep 100
/dev/mapper/log_vg-log                    9.8G  9.5G     0 100% /storage/log

Clear space by removing old log files, for example:

root@vcsa01 [ ~ ]# df -h |grep 100
/dev/mapper/log_vg-log                    9.8G  9.5G     0 100% /storage/log
root@vcsa01 [ ~ ]# cd /storage/log
root@vcsa01 [ /storage/log ]# du -sh *|grep G
9.4G    vmware
root@vcsa01 [ /storage/log ]# cd vmware/
root@vcsa01 [ /storage/log/vmware ]# du -sh *|grep G
1.6G    eam
2.7G    lookupsvc
root@vcsa01 [ /storage/log/vmware ]# cd lookupsvc/
root@vcsa01 [ /storage/log/vmware/lookupsvc ]# du -sh *|grep G
2.6G    tomcat
root@vcsa01 [ /storage/log/vmware/lookupsvc ]# cd tomcat/
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]#
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]# rm -rf localhost_access.2021*
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]# df -h .
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/log_vg-log  9.8G  8.0G  1.4G  86% /storage/log
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]#

After cleaning up the space in a few more folders, reboot the appliance:

reboot

You should now be able to login after the space is freed.  Try to loign again to the management console (Port 5480) to disable and enable Time Synchronization if UI still doesn't show up.  If not, issue:

root@vcsa01 [ ~ ]# service-control –start applmgmt

From the vSphere Client SSH session.  If you get a certificate expiration failure:

Exception in invoking authentication handler [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)

Renew the self signed cert, or official certificate.  Use this page:

https://kb.vmware.com/s/article/76719

Example output:

root@vcsa01 [ /tmp ]# ./fixsts.sh
NOTE: This works on external and embedded PSCs
This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain
==================================
Resetting STS certificate for vcsa01.nix.mds.xyz started on Mon Aug  1 04:23:46 UTC 2022


Detected DN: cn=vcsa01.nix.mds.xyz,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: vcsa01.nix.mds.xyz
Detected PSC: vcsa01.nix.mds.xyz
Detected SSO domain name: vsphere.local
Detected Machine ID: 310ae9cb-82a9-4fa4-bcd4-d34b054d0090
Detected IP Address: 192.168.0.33
Domain CN: dc=vsphere,dc=local
==================================
==================================

Detected Root's certificate expiration date: 2030 Jun 3
Detected today's date: 2022 Aug 1
==================================

Exporting and generating STS certificate

Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success


Enter password for administrator@vsphere.local:
Highest tenant credentials index : 1
Exporting tenant 1 to /tmp/vmware-fixsts

Deleting tenant 1

Highest trusted cert chains index: 1
Exporting trustedcertchain 1 to /tmp/vmware-fixsts

Deleting trustedcertchain 1

Applying newly generated STS certificate to SSO domain
adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"

adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"


Replacement finished – Please restart services on all vCenters and PSCs in your SSO domain
==================================
IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
==================================
==================================
root@vcsa01 [ /tmp ]#

Try to login again.  

Cheers,
Tom


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License