vSphere Web Client doesn’t answer instead 503 errors

Hi there,

I was getting vSphere Web Client error when I tried to enabled BASH on my vCenter – so I could get the HTML5 client working.  So I decided to restart the VCSA.  After the restart, and a delay, I see this error message.

error

The C# client works fine.  My backups are working fine too so the API is fine, but it is specifically a vSphere Web Client issue.

So I think to restart the vSphere Web Client to see if I can get a useful error message.

running

So it seems to start but it is not.

running2

The wrapper and Java are starting.  And doesn’t really get past there before it expires.

running3

delay

I think to check the vSphere Client logs and they are found in:

/var/logs/vmware/vpshere-client

logs

The current log is empty which I think is due to vSphere Web Client not running.  The wrapper log has a bunch of Java issues but nothing I can action.

So now to Google.  I find a lot of stuff.  But most of it is useless.  But one is interesting.

In this article it suggests that if I regen all the certs it will solve this issue.  Interesting. It points me to this article.

Here is another article that is only a little help but it helps around commands to use.  It also suggests to check the vpxd.log file. I do, and it is busy indeed, but I see nothing like error, timeout or the like.

But from the article that said they had the same problem and fixed it I know the command I need is:

Reset All Certificates

But I need to find an example of the command line for it to be used in.  This appears to work:

./certool reset all certificates

As seen below

cert

Not sure what needs to be restarted so I will try first just starting vSphere Web Client and see what happens.  Not sure how regenerated cert can solve this issue.

I tried connecting to vSphere Web Client and I see that I still have a green HTTPS connection. If the reset all certificates had worked it should be red.  I wonder if I need to restart things?  Going to do a shutdown and startup.  I do like the Host Client for this!

So after a restart, and using Chrome to connect to the vSphere Web Client it still show green so it has not regenerated the certs otherwise it would be red and I would have to download a cert to make it all green – see how here – this will mean you have less clicks to access vC.

On the console, I see that vSphere Web Client is not running.  But after it bits it starts.

AfterREboot

As I mentioned, the vSphere Web Client log is empty.  The wrapper log file has really lots in it.  Here is a shot of it where things are still in the starting state.

wrapper

I see in it that Java is having an issue accessing the jks-based keystore.  I can conceivably see that causing a delay in starting the vSphere Web Client so that it times out

So I am stuck.  The reset all certs did not work, and not sure if that would help the Java keystore issue.  But I am going to publish this article without a solution in case someone has an idea for me to try, or if I figure out the recert thing.  I will update this article as I learn more! But dinnertime is here!

Update

  • 8/24/16 – I was not able to fix this, nor get any help for it, so I did the stupid and painful thing – which was delete my vC and install it again. Lovely. But this matter is closed and I am very sorry I was not able to work with VMware support and figure it out so that if you hit it you can fix it rather than what I am doing.
  • 8/22/16 – spent a bunch of time on this yesterday and made no positive progress.  I think I need to install vC again.  Really sucks when you think of dVS I will need to redo, and how I will need to reconnect things like Dell, Veeam, and View to it. But what choice do I have?  I agree it looks like some sort of a Java issue, but nothing on the internet helps, and vExperts don’t get support. I also discovered the cert change I did also hurt the Support Assistant.  Easy fix was to reconnect.
  • 8/21/16 – above I wonder if regen of all certs in the VCSA is going to hurt.  And there was a little hurt.  All my Veeam jobs failed last night.  So of them did show remote cert errors but many did not.  All that was necessary to fix was to connect Veeam to vC again.  It immediately notes the new cert, and you can accept and you are good.
  • 8/21/16 – I confirmed that I have enough storage on the VCSA as it was suggested in comments that if I was running out of storage that could cause an issue for Java.  I have also tried running the VCSA on a different host in case this is connected to bad host RAM.  I do note that after I regenerated the certificates – which I thought might have caused Java to not run, that I get a very slightly different error message.  Essentially a port change I think.  Not sure if it is a clue or not.

DifferentButSameError

  • 8/20/16 – Found this article to help with redo of the cert regen. BTW, I used Choice 4, and used all their defaults save one – I used the actual FQDN of vC.  Took a little while. Looks like 26 services were updated.  I wonder if this is going to hurt somehow? It seems to have spent some time at 85% Complete [starting services…].  Wait.  100% complete successfully. Well.  The cert is replaced as I see red and a line through the HTTPS.  But, I still have the same error message.  So the cert is not the solution.  Crap.  Really going to dinner now.

Any comments or suggestions gratefully welcome!

Michael

Tagged with: , ,
Posted in Home Lab, How To
4 comments on “vSphere Web Client doesn’t answer instead 503 errors
  1. Chip Zoller says:

    Hi Michael, I see you have encountered some Java-related crashes as evident by the residual .hprof files in the vsphere-client directory. Do a “df -h” at root and check the filesystem usage to make sure none of the LVMs are at or near 100%. I’ve seen this error caused by inability to write to a full or near-full filesystem.

    • Thanks Chip, none of my filesystems are full. the log_vg-log is at 86% but it has not been changing much. Everything else is much smaller.

      Thanks for the idea! Much appreciated.

      Michael

  2. Liandri says:

    Last time i got this kind of 503 on esxi 6, it was dead ram.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: