My vSphere 6.5 Upgrade Checklist – painful

Hello all,

Yes, this has taken me a while to get too.  And it took me a while to do as well. Now that I work at Veeam, and while I have used Veeam products for many years, now I need to work my lab like a Veeam customer. vSphere 6.5 is a big deal for me, and it has a lot of new stuff in it that I am looking forward to so I have been waiting anxiously for Update 1 for Veeam that supports 6.5. It is here now but there is still one more tool of mine that is not 6.5 capable but I am tired of waiting for it.  It is the VMware Infrastructure Navigator (VIN) and I find it most useful.  BTW, I do this article, and then with it as a plan I do the upgrade.  After that I update the article with anything I learned and then finally I publish. I will keep in this article the problems I had in case it helps someone.

What do I have to start with?

  • VCSA is 6.0 U2 at build 4541948 as seen in the vSphere Web Client
  • ESXi is 6.0 U2 at build 4600944
  • I have one VCSA and 5 hosts in two clusters.
  • I have Veeam ONE at v9.5 U1
  • Veeam Backup & Replication is v9.5 U1.
  • VMware vCenter Support Assistant at 6.0.2.1 build 4156172
  • Log Insight 4.0
  • View 7.0.3
  • Dell OpenMange Integration for vCenter – 3. 1.1.150 – in fact there was an update outstanding on this to I did that update before continuing.

I like where possible to have everything at a build level where 6.5 support is not an issue. There is no update yet for VIN – current is 5.8.5, so it is out.  The rest is good.

I have also recently restarted my hosts – likely that is not needed any longer but in the past that helped avoid an issue when updating the hosts. Everything is patched or updated to current. Also before we start I make sure that everything is working – in particular, I suggest you make sure your vMotion is working. You don’t want to work on a problem that you think started with the upgrade when it fact it has been around for a long time.

Preparation

It is a very good idea to check the following docs during preparation for the upgrade.

  • Release notes – vC and ESXi – found here
  • Release notes – Update Manager – found here
  • Update Sequence for vSphere 6.5 – kb article – this is important if you have a complex install and are using many different VMware products.  I am not and so it is pretty simple.
  • Important information before upgrading to vSphere 6.5 – kb article  – this is important as it talks about what is not yet compatible with 6.5 in the VMware products.
  • Best practices for upgrading to VCSA 6.5 – kb article

As well we need to get the bits:

VUM prep

One we are ready to start, we need to prepare VUM.  And we need to do this before the upgrade / migration process.

So on your VUM server you need to access the vSphere 6.5 bits.

a1vum

We need to copy the migration-assistant folder – indicated by the red arrow above – to the local hard drive of the VUM server.

a2vumexe

We need to double-click on the VMware-Migration-Assistant.exe as seen above.  We will need to enter the password to our service account that is used by VUM.

a3run

Once it completes it looks a little different.

a4ready

We leave it running.

Misc Prep

Since our old VCSA is in the cluster we are migrating our VCSA in, we need to put DRS in manual, and use a host name rather than vC – I will point that out later.  The host name is the name of the host that is running the original VCSA. We also need to enable SSH in that host too and on the old VCSA.

But, before we start, we make sure we have a backup of vCenter.  If I was using vCenter on Windows, I would backup up both it and the database server.

So, it looks like we are good to go.  We have read what is necessary, and updated where necessary, and confirmed everything is working fine.  We have all the bits, and we have a backup.

Process

We start the process off with an update of vCenter.  Sometimes – often in fact – customers will halt there and wait a bit to see how it goes before they upgrade the hosts.

VCSA

This is not really an upgrade but rather a migration.

We need to be able to access the ISO.  So I have expanded / extracted it in my SMB storage.

1folder

We will need to use the UI installer.  Not doing a CLI migration.  Unless I had a bunch of vC’s to upgrade – and bunch would likely start at 2 or 3.

So change into the UI installer folder, and then change into the appropriate folder – I am using a Mac but that should not impact what you do.

Double-click on the installer file and you should see the following screen.

2choices

We are doing an upgrade (yes, I know I said it was – and is – a migration) but we have to go with the flow.

After we select Upgrade we get some info on what is going to happen.

3info

Next up is to agree with the EULA.  And we always do that.  Although, I did some work at one customer where every EULA was provided to a lawyer to review before we could agree to them.  Did slow things down.

On the next screen we need to fill in some details that are quite important.

fix1

Note the SSO user name, and the appliance root password fields on the source.  As this is a migration we have a source that is our original and we will migrate the info out of it to the new appliance that will become our new VCSA. Also very important to note in the bottom of the screen you need to put the ESXi host name that your 6.0 VCSA is running on.

5certs

Note below that we use a host name and not the vC name since we are upgrading the vC and it will not be available.

fix2

We decide next where to put the new appliance.

9reay

Now we decide what compute.

10compute

Now we need to name the new appliance.

11newname

Now we chose our deployment size.  I need 100 VMs or less and 10 hosts or less to the Tiny choice is good for me.

12size

Next is the storage location.

13storafe

And next is the temporary network information.  Temporary as network info is going to be migrated too. I am going to use DHCP. Make sure the right network is displayed.

14network

We see the typical summary.

fixfinish

And after we use the Finish button we see the start of things.

15initial

16deply

After after a bit – maybe 20 minutes you see this.

17success

So Stage 1 done and on to Stage 2.

20continue

And next we see ….

21continue

And next we see some issues.  Again.

pupcheck

In this case, there is nothing for me to worry about about so I close.

stage21

I would like to have configuration, events and tasks, but see no reason to have performance metrics. So I make that choice.

On the next screen I leave the CEIP program selected as I believe in that sort of stuff – it really can help.

stage22

Now select Next to continue.

stage23

Check the information on this screen, and select that you have a backup and then Finish.

stage24

It certainly is, as it would not be good to have both running – IP conflict for example.

stage25

Now we wait.

stage26

So after maybe 20 minutes I see this.

stage2complete

So done with my vC upgrade.  Time to test and make sure it worked! Test section is below and can provide some ideas on what to test.

ESXi

This is easy as the Update Manager does most of the work.  I love how that is now part of the VCSA. We need to start in the Update Manager.  You can get there from under the main menu, or by selecting the app on the home screen.

homeum

So we need to select the vCenter / VUM server, then Manage, and ESXi Images.

import

We use the Import ESXi Image button to load the vSphere ESXi 6.5 image we want to use to update the hosts.  This can be the ISO we downloaded from My VMware, or from a partner,  or this could be the image you have updated  yourself. Several of my hosts have hardware that is not part of the base ESXi 6.5 image and I deal with that using VUM after the server is running.

Select the appropriate ISO and you can watch it upload.

upload

Once it is done you see something different.

uploaded3

Now we have an image to work with. If we highlight it, we see some additional buttons to work with.

baseline

We can see what is in this image, but we also now have a Create baseline button. If we use it, we can create a baseline.

baseline1

After we OK we can see the baseline we create if we change to the Host Baselines tab.

baseline2

Now that we have a baseline we can use to upgrade hosts, lets change to the Hosts & Clusters view and select the Update Manager tab.

hostscluster

We now use the Attach button to attach the baseline we just created.  Note how we are working at the cluster level?  This is easier than doing this three times – once for each host, but it also will cover any new hosts that added to the cluster in the future.

attach

Now I use the Scan button to see what the status is in the cluster with respect to our baselines.

status1

We can see that everything is compliant – except with the new baseline we created.  So that is good.  I like to have my hosts patched to current before they are updated.

Now we use the Remediate button to start things off. I just love this functionality.

remediate1

So a wizard starts, and we select Upgrade and vSphere6.5.

On our next screen we select all of our hosts – they all need to be upgraded after all.

remediate2

The next screen is the EULA – so we just accept it and move on. After that there are Advanced options such as scheduling.

remediate3

I do not normally use the schedule option.

Next up is Host Remediation options.  My screen may look a little different than yours as I have made some of these choices default.  Which you can do too in the Update Manager Settings area, or in the bottom right corner – which is new and handy.

remediate4

Again, on the next screen – Cluster remediation options it may look different for you.  Again you can change the options to be default and the checkbox in bottom right can enforce that for you.

remediate5

I really like the option for parallel remediation.  If you are patching or updating a new cluster this can be a very good way to speed things up – do 2 or 3 or more hosts at the same time.

On the next screen we see the summary.

remediate6

Once started you can watch the Recent Tasks for updates – as seen below.

remediate7-status
If you need to monitor more closely you can look at cluster level, Monitoring, and Tasks / Events.  You can also look at Monitoring on Update Manager and I find the Events tab most useful.

VMware Tools

There is a number of ways to do this.

You can trigger an automatic update from the VM summary screen.

tools1

You can even configure your VMs to upgrade the tools at the next restrart.

tools2

You can even use PowerCLI to do the updates (more info here).  There are also baselines for Virtual Hardware upgrades as well as VMware Tools.

vm

I did virtual hardware upgrades first using the baselines, and then VMware Tools.  I attached the baselines to a folder structure that contained most of my virtual machines.  There is a reboot often involved with the update of VMware Tools so watch out for that.

Problems I had

  • No SSH access available on the host that held my VCSA. Easy to fix.
  • My DRS was still fully automatic rather than manual. Easy to fix.

22moreerrors

  • No Migration Assistant running on the VUM. Easy to fix.

7nextopps

  • Using old VCSA as destination when I should have used host. Below in the second set of fields I should have had the ESXi host. Easy to fix.

4filin

22moreerrors

  • Bad account name entered – I used copy and paste when I shouldn’t have. The adminstrator@ in the field cannot be removed – I thought I pasted over it but look what happened.  I did not notice at first. Easy to fix.

4filin

  • Extensive  internal errors like this one seen below when logging into the Web Client on Mac.  But the error did not occur on Win10.  This problem was solved by one or more of the following actions – deleted the 6.x Client Integration Plug-in, restart of the Mac, successful login on Windows.

wtf3

  • VUM would not upgrade hosts.  Remediation pre check said HA was enabled. Nothing in Events to help. The cluster option to disable HA admission control was enabled. Where is a log file I can check? So I manually disabled HA so things could proceed. Turns out HA is not the issue.  Events at the cluster level show that it was a conflict in VIB.  Which is interesting since I use stock ISO. Unlike in the past when I have see this it doesn’t show what VIBs are in conflict. By looking at the VUM tab on each host I was able to confirm which host had the issue.  My lone Dell R710 in this cluster was the issue. Again on the host in question, and in VUM you can get more info on the issue. So we are good to go now. Mostly easy to fix – this can help.

error

No fix found for the issue in red below.

hasettings

conflictingvib

wtf

  • I do not think it was part of the upgrade, but vMotion doesn’t work any longer.

vmoissue

Clean up

We now need to clean up stuff.  For example, if the 6.5 VCSA is working properly we need to delete the old one.

  • Is it time to delete the old VCSA?
  • Can we delete the old VUM yet?  If we delete the old VCSA we can.  Except in my case with MSDN licensing I am going to keep it and re-purpose it. So I power it off.
  • Are we going to do Auto Deploy?  That UI is not seen until you enable it.  There is a link below to help with this.
  • We should disable SSH on the host(s) we enabled it on.
  • We should enable fully automated DRS now too.
  • Check out VUM to make sure the settings and baselines are what you expect.
  • Once we have cleaned up – deleted the old VCSA we could rename the new one to the old name.  After all it is answering DNS or API to the old name.
  • To get the new vSphere 6.5 VMFS functionality like faster re-signaturing (great for SRM customers) and the automatic unmap support you need to format new VMFS partitions.  No upgrade.  But a good thing to do and Storage vMotion is your friend.

Test

Likely this will go well but we still need to confirm things work.  For example, sometimes View Desktops don’t work after the VMware Tools is updated – and we will need to do that.  So it is always good to test.

  • We should see the new vSphere build for vCenter – 6.5 4601587 – Yes
  • We should see the new ESXi build – 6.5 4564106 – Yes
  • Can we log into the vSphere Web Client with our normal credentials? Yes
  • Can we do a on-demand backup and have it complete with no errors? Tried it with vC 6.5 and it works. Tried it with vC 6.5 and ESXi 6.5 and it works.
  • Can we access our View desktop as we did before? Yes. Upgraded VMware Tools to current and still it works.
  • Does the new authentication plug-in work in Windows – Yes, but I removed the old plug-in first. Tested on Win10.
  • I did not test before the upgrade, but after the upgrade vMotion does not work any longer.  I am going to guess that it is not upgrade related and deal with it.  The symptom is no network.

Things I noticed

  • The authentication plug-in is actually two installs in one.  Make sure to get them both.
  • There were a few places where you had to pay particular attention to – when it asks for host or vCenter name and you are upgrading the vCenter don’t use it but rather the host.
  • I did not have to do anything with Support Assistant – it worked fine after upgrade.  So it was migrated nicely.
  • It looked like Dell OpenManage was still functional but I can confirm that with more testing and re-register with vCenter. Confirmed – it works fine even though the message said it would not
  • The Veeam plug-in for vCenter appears to be working as well. Interesting!  I could have re-registered it as necessary but it is not necessary as it is working.
  • I have worked with VMware from before vCenter even existed.  So it is pretty cool I did this all on a Mac.
  • While the Client Integration Plug-in is no longer required, and in Help About it even confirms that, when I hit the upload button on a datastore, it downloaded the CIP for me – when working on a Mac. On a PC, that that has the authentication plug-in installed it works as it should – prompts you for a fie. In both cases I was using Chrome.

Links

  • Release notes – vC and ESXi – found here
  • Release notes – Update Manager – found here
  • vSphere docs – hereupgrade guide
  • What’s New in vSphere 6.5 – updated doc
  • 6.5 Configuration Maximums Guide – here
  • Update Sequence for vSphere 6.5 – kb article
  • Important information before upgrading to vSphere 6.5 – kb article
  • Best practices for upgrading to VCSA 6.5 – kb article
  • William’s great collection of links – here
  • Jon Kensy – VCSA upgrade – this guy upgrades a VCSA that had no connection with VUM.  How realistic is that?
  • Image Builder 6.5 UI is not seen in the UI after an upgrade – here
  • Failure to upgrade to 6.5 – VUM migration error – very odd
  • Doc on VUM migration as part of 6.5 upgrade – docs
  • Good upgrade article – here
  • Use PowerCLI to update VMware Tools – here

This was one of my more difficult upgrades.  But glad it is done.  Now I need to fix vMotion. I hope that the problems I had, and documented, help you. I hope your upgrade (migration) goes better then mine! I should mention that other than vMotion everything is good.  All my services and apps are running and there was no downtime. So it was painful but ended OK.  Except for vMotion.  Still haven’t figured that out.  But weekend is over and need to do some stuff.

Updates:

  • 2/11/17 – thanks to a reader, I found out that Dell OpenManage Integration for vCenter v4.0 is out in – maybe Feb – and that it adds 6.5 support.  Which is odd as it seems to sort of work in 6.5. Find out more about 4.0 in this article.
  • 1/31/17 – added the note about about VMFS in vSphere 6.5.  In my lab, after the upgrade, I had 5.60 and 5.54 versions of VMFS and I could not upgrade them.  But a new VMFS partition is 6.81 and supports the cool new functionality of faster re-signature and automatic unmap. Also added the point about testing your vMotion before you start.
  • 1/30/17 – solved the vMotion issue.  I first made sure there was no vMotion stack on my servers.  There was on one so I removed it.  BTW, I find it with this command esxcli network ip netstack list , and then remove it with esxcli network ip netstack remove -N=”vmotion” .  Then, on each of the vmkernal ports (on each server) that were enabled for vMotion I removed it.  Then I added it back.  So did that on two hosts and vMotion was working fine again. Amazing. Then when I added a new 10 GB card to a third host, I had to remove the vMotion stack, and remove and add back the vMotion parameter and then everything worked.  This is in fact connected to vSphere 6.5.
  • 1/30/17 – the incompatible VIB was the driver for a very odd 10 GB card.  I thought it was incompatible due to a newer drive in 6.5 but that is silly.  It is incompatible with 6.5.  When I removed it upgrade proceeded but the 10 GB card did not work.  So no vMotion – and no storage.  That host is now in maintenance mode until I can put a nice Intel 10 GB card into it but vMotion still doesn’t work.  I can do a vmkping on the vMotion network so I know that works.  Must look more!

Questions and comments are always welcome.

Michael

=== END ===

Tagged with: , , , ,
Posted in Home Lab, How To
13 comments on “My vSphere 6.5 Upgrade Checklist – painful
  1. I ran into the same thing today with my upgrade I completed today.

    • I figured out mine. I was unable to vMotion as I had a dedicated vMotion TCP/IP stack created on one host but not the other. I was able to identify it by running the following on each host from the command line:

      esxcli network ip netstack list

      In my case, the vMotion netstack was listed as “vmotion”. From there, I ran the following to get rid of it:

      esxcli network ip netstack remove -n=”vmotion”

      Soon as that command finished, life was great. I was able to get to this point by trying to remove and re-add the vMotion vmkernel and it failing and directing me towards the IP netstack. Hopefully this helps.

      • Thanks for this Zack, I tried it an only one host had the vmotion netstack. I removed it and thing broke not at or around 21% but at 0%. So instance fail. But it gave me the idea to remove vMotion from each servers vmkernal and than add it back and that solved the issue!

        Thanks again,

        Michael

      • Very good info. I think it helped out.

        Michael

  2. Bob Plankers says:

    “I have also recently restarted my hosts” — I do this as well. Perhaps something people who worked in support organizations at one point just instinctively do.

  3. Patrick says:

    Sorry for a little offtopic question, but where did you find Dell OpenManage 3.1.1.150? I only see “Dell_OpenManage_Integration_3.1.0.134_A00.zip” 😦

  4. […] White over at Notes from MWhite has a very good description of how to perform the upgrade. I suggest reading it and following it, […]

  5. We’re planning a vSphere 5.5 to 6.5 migration with Cisco UCS hosts. I will review your guide. Thank you for putting this all together!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: