Header Shadow Image

vmware: no healthy upstream

After a hard reset, a greeting of:

no healthy upstream

pops up from the vSphere Client.  Login as root and issue:

service-control –start vmware-vpxd

to see if there's any  additional information in regards to this error.  You may  or may not receive more info.  Check the time and NTP settings. There's a good chance time is not synced up.


Login to the management console.  For example:


If it fails to login with:

Unable to login

check space with:

# df -h |grep 100
/dev/mapper/log_vg-log                    9.8G  9.5G     0 100% /storage/log

Clear space by removing old log files, for example:

root@vcsa01 [ ~ ]# df -h |grep 100
/dev/mapper/log_vg-log                    9.8G  9.5G     0 100% /storage/log
root@vcsa01 [ ~ ]# cd /storage/log
root@vcsa01 [ /storage/log ]# du -sh *|grep G
9.4G    vmware
root@vcsa01 [ /storage/log ]# cd vmware/
root@vcsa01 [ /storage/log/vmware ]# du -sh *|grep G
1.6G    eam
2.7G    lookupsvc
root@vcsa01 [ /storage/log/vmware ]# cd lookupsvc/
root@vcsa01 [ /storage/log/vmware/lookupsvc ]# du -sh *|grep G
2.6G    tomcat
root@vcsa01 [ /storage/log/vmware/lookupsvc ]# cd tomcat/
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]#
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]# rm -rf localhost_access.2021*
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]# df -h .
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/log_vg-log  9.8G  8.0G  1.4G  86% /storage/log
root@vcsa01 [ /storage/log/vmware/lookupsvc/tomcat ]#

After cleaning up the space in a few more folders, reboot the appliance:


You should now be able to login after the space is freed.  Try to loign again to the management console (Port 5480) to disable and enable Time Synchronization if UI still doesn't show up.  If not, issue:

root@vcsa01 [ ~ ]# service-control –start applmgmt

From the vSphere Client SSH session.  If you get a certificate expiration failure:

Exception in invoking authentication handler [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)

Renew the self signed cert, or official certificate.  Use this page:


Example output:

root@vcsa01 [ /tmp ]# ./fixsts.sh
NOTE: This works on external and embedded PSCs
This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain
Resetting STS certificate for vcsa01.nix.mds.xyz started on Mon Aug  1 04:23:46 UTC 2022

Detected DN: cn=vcsa01.nix.mds.xyz,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: vcsa01.nix.mds.xyz
Detected PSC: vcsa01.nix.mds.xyz
Detected SSO domain name: vsphere.local
Detected Machine ID: 310ae9cb-82a9-4fa4-bcd4-d34b054d0090
Detected IP Address:
Domain CN: dc=vsphere,dc=local

Detected Root's certificate expiration date: 2030 Jun 3
Detected today's date: 2022 Aug 1

Exporting and generating STS certificate

Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success

Enter password for administrator@vsphere.local:
Highest tenant credentials index : 1
Exporting tenant 1 to /tmp/vmware-fixsts

Deleting tenant 1

Highest trusted cert chains index: 1
Exporting trustedcertchain 1 to /tmp/vmware-fixsts

Deleting trustedcertchain 1

Applying newly generated STS certificate to SSO domain
adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"

adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"

Replacement finished – Please restart services on all vCenters and PSCs in your SSO domain
IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
root@vcsa01 [ /tmp ]#

Try to login again.  


Leave a Reply

You must be logged in to post a comment.

  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License