My vSphere 6.7 Upgrade - WOW!

Hi all,

This article covers off the process of my upgrade to vSphere 6.7 in my lab.  It is later than you may have suspected due to dependencies in my lab I had to wait to be dealt with.

So this process starts with me getting the docs, and the bits (vCenter and ESXi). I read through the release notes (vCenter and ESXi) carefully.

Next is compatibility.  This is very important, even in my lab as I have a bunch of things here. Hardware compatibility - meaning can your ESXi hosts run this new version - can be checked in this web site. Now VMware compatibility needs to be checked.  I am using VMware View for example and I need to know if it works with 6.7.  This web site can help with that.

I also find a KB article about the upgrade that has more good info. So I go through all of this and what do I find?

Results of research?

I find a number of things that are important.

• My SuperMicro hardware is supported.  There was 24 pages of SuperMicro hardware - amazing.
• My Horizon View 7.3.1 is not supported.  Earliest limited support is 7.4 and full support is in 7.5.
• My Backup software - Veeam 9.5 Update 3 is not yet supported - but any day it should be.
• My version of vRealize Automation (7.3.1) is supported by 6.7.
• My version of vRealize Log Insight (4.6.0) is supported by 6.7.
• The latest version of VMware Support Assistant appears to be supported only as of 6.5.

So with this info I know what to do - at least to plan. My lab is simple compared to some, I know people that have other components like NSX in the lab so that means more checking to determine the order of things!

Order of Operation

I should follow the following overall order.

• Remove Support Assistant (un-register and delete).
• Upgrade View to 7.5
• Upgrade Log Insight to 4.6.1.
• Upgrade Veeam to 9.5 Update 3a
• Then do the 6.7 upgrade.

Remember that there are sometimes components that are part of the upgrade that are not order dependent.  Log Insight could be the first / second / third to be updated for example. The key one is that View and Veeam need to be updated before vSphere 6.7 since I will need to do backups, or connect remotely, after the upgrade and so they need to work.

Horizon View 7.5 Upgrade

Bits, docs, release notes.

And of course read the release notes.  And skim through the upgrade guide too.

I should confirm that I can upgrade from 7.3.1 to 7.5 - yes, it appears to be supported.  As well, my Mac View Client recently updated itself so it too is current and will work fine - both with 7.3.1 and 7.5. I need to confirm that 7.5 will work on the embarrassing old version of Win2K8 that is my OS on View servers is still supported - and it does.

I also need to confirm that things work so I access my desktops, and the admin interface and make sure all looks good.

I also take advantage of this outage and update for any Microsoft updates or AV or whatever.

I do a backup of my connection servers first - just in case! And I check the backup to make sure it worked good too! And it has.

Upgrade the Connection Server(s) first, followed by Security servers, followed by the agent.

Next you need to test.  Connect to desktops, use the Admin UI, check out the updated Help Desk and make sure it all works.

BTW, after I updated my connection server I could not connect to the admin UI.  I had to restart the connection server before I could connect to my admin UI. If you want to access the Help Desk, first use the Horizon Console button in the admin UI, than access a user in the Console and you will have the Help Desk.  The old URL doesn’t work. So I also restarted the Security Server after it was updated.

Also, still true, you cannot upgrade the View Agent if you have Viewed into that desktop.

So my View infrastructure is updated to 7.5 with no issues.  All good.

Log Insight 4.6.1 Upgrade

This was easy - and quick too. Release Notes. I just love how easy it is to update LI.  There is not much in this release but it is current now so that is good.

Veeam Backup & Replication 9.5 Update 3a

This is an easy update - like most Veeam ones in fact.  Find the bits (bottom of the page for release notes) and release notes. In the notes you can find out the amazing range of updates in this Update so that is quite good.  It is not just vSphere 6.7 support after all.

Execute the update on your VBR server, and it will update VBR, your console, and your proxy’s.  Best to do this when no jobs are running.  If you have issues, you can disable the Veeam Backup service. Also don’t forget to enable the remote backups switch.

This will help you do a quicker and easier update. Applying the update in my lab, with slowish storage took around 35 minutes.

After you restart, access the VBR console and confirm things work as well as the new build - 9.5.0.1922.

So we are now ready for the big upgrade - vSphere 6.7.

vSphere 6.7 Upgrade

I am using 6.5 build 8024368 and have not done a Update 2 update so I should be good to upgrade. My hosts are 6.5.0 build 7967591.

So this process starts with me getting the docs, and the bits (vCenter and ESXi). I read through the release notes (vCenter and ESXi) carefully. I read through the release notes at the start of this, but, I have seen release notes updated so I check them again. BTW, I am upgrading to vCenter (VCSA) 6.7 build 8546234 which is the recent release of 6.7a.

Here is an article ongoing the upgrade with nice screenshots. There is quite a few articles on the upgrade process so I am not going to detail it here but will mention things I think worth sharing as I go through it.

Oops.  I forgot to uninstall the Veeam plug-in from the vSphere Web client before I updated. That would have saved a little extra work.  At least I know what that little work is (KB2677).  Also, during the pre-upgrade check that the upgrade does, Veeam extensions is highlighted as not upgraded so that explains why the problems, but also why I meant to uninstall it. Runecast was included too but I may have an old version.

After the deploy, and on step 3 I hit an issue.

I could not get past that screen without filling in the Export directory. This was quite a surprise to me.  Google did not find me anything to help. So I reached out to VMware Support. I had the idea to expand the size of the destination disk which is in fact not too hard (thanks to this and this). Fortunately, the VMware support guy said bad idea as it would mean in the next update I would have the same issue. I also realized I could cancel here and the new VCSA would still be deployed but I could figure things out and then start again by accessing the https://fqdn_vcsa:5480 and continuing on.

So I also went into the Veeam Enterprise Manager and uninstalled the Veeam plug-in to vCenter as it would help a smooth upgrade.

Then I removed the ISO I had in the Update Manager thinking that would save at least 1 GB. I have a lot of extra patches due to being connected to Dell repo’s at one time but no idea on how to delete those which might save some room.

During phase 2 I select host where the source VCSA was, so I needed to change the DRS of that cluster to off.

But the compressed logs, and ISO I removed did not help much.  I did figure out where the Dell patches, and Cisco patches were and was able to delete them. I talked with VMware to see what else I can remove.  In the /storage/sso folder there was a lot of compressed log files so I deleted them too.  VMware did get back to me with a screenshot of a default Update Manager folder structure showing the default folders and files.  Anything other that what is in the screenshot can be deleted.  Sounds tricky but useful.

So I hear from VMware, after I mention that no matter how much I delete the same size is seen - same as in the screenshot, and they say that I need to delete the new / destination VCSA and deploy again and I will see the size difference.

Of course, I take some time this morning and start things over.  And you can imagine what happens?  Yes, I get the same damn number of 4.79 GB. So I will remove the destination VCSA and confirm all I deleted before is gone, and then clean up VUM as they suggested and see what happens.

So this time I use the du -sch * to see what the numbers are like, and /storage is really full as is the /storage/log so I use SecureCRT to connect and clean up Update Manger as support suggested.  But then I use SecureFX to connect in a graphical UI and start checking each folder in /storage/log and deleting all the *.gz files, and the *.zip files but also any old dump files or old logs. This drops me to 3.4 from 3.9 GiB so we will see if this helps at all.

I must admit I wonder if I am having this trouble due to me picking the Tiny size of appliance?

So after my second cleaning, I now am successful as I see only 2.79 GiB and an estimated 39 minutes. But I chose instead the Configuration, historical, and even performance metrics.  So it is 2.89 GiB.  Took maybe 1.5 hours - more or less and we are done.

Once we have a VCSA at 6.7, we need to do a few things:

• Confirm the new version - 6.7.0.11000
• Enable DRS again.
• Update the VCSA - for me there was an update to 6.7.0b outstanding. This will get you to 6.7.0.120000
• Configure VUM to help with the host upgrades.
• Host upgrades. I had some issues updating my hosts, but they were typical issues, and I could deal with them.  For example, in one cluster of two hosts I put one in maintenance mode, and upgraded it.  Then took it out of MM and put the other in MM and upgraded it.  This was not a vMotion issue so not sure what it was.  Actually, as an update I had more troubles than I should have - for example i had one upgrade fail in a way I had to update the host over again using an ISO and KVM.  Definitely connected to 6.7. Odd. Also odd is the VUM Scan action always seems to have issues.  Will take some time on this.

And finally we need to do some tests.

• Can we get a backup, and a restore? There may be a need to reconnect to vC and accept  certificate but in fact there wasn’t for me in Veeam.
• Can we connect to a View desktop?
• Do we see new data in Log Insight?
• I also like to visit a few hosts and confirm the new build number.

Links

• Supermicro install info - and no need to load the Intel network drivers in 6.7!
• No longer necessary to fix Hardware health issues on Supermicro with 6.7
• My co-worker Anthony has an issue with an upgrade - different issue than mine but here it is in case it helps you.
• Here is a nice walk through of a VCSA upgrade with no issues.  It has nice screenshots too. Sort of how I wished my upgrades went - but I have a busy and hardworking lab so I guess that changes things.
• If you are doing the upgrade you should already be familiar with using VUM to upgrade your hosts. But if not, you can find help with that in Clint’s article.

This took a lot more work than I expected, and certainly more time.  I did lose some time on it, but no data or VMs were harmed in this process.  It is done and I do love the HTML5 client. I should also mention that I worked with both the 6.6 and 6.7 beta and was very happy with them, but this is my first upgrade.

Updates:

• 9/4/18 - so finally had some time, and good luck.  Support provided this KB article which did not work on 6.7 but should be updated soon.  BTW, the problem was in step 4, where it says updatemgr-util reset-db.  It should say updatemgr-utility.py reset-db.  But all good now!
• 8/7/18 - I do have ESXi 6.7 patches now, in fact a handful.  And I still get scan errors.  So the support guy who said II would not was missing something, or wrong.  I need to chase him. VMware mentioned that a patch in vCEnter is needed to fix this and while it was supposed to be out recently, it wasn’t.
• 7/23/18 - I noticed that if I did a Scan with either the Critical or non-critical baselines it would fail with errors and not do the scan.  I just found out that is a known issue, and will be fixed in the near future. There was a hint of this in the release notes but it was not clear enough for me I guess.

Questions or comments are always welcome!

Michael

=== END ===