This is something that is important to me. I believe in it, and actually practiced it, before I knew what it was. About 2 years ago we were talking about our new group at VMware – Integration Engineering and what it would do. One of the several bosses I had at the time – and both very smart and experienced, suggested the term to represent what we would do. AdamZ suggested Customer and I agreed, and understood right away.
We would have three data-centers, an operations team, and in theory we would do things like customers would. We had architects like me that know a lot about that sort of thing. We would use alpha and beta builds and do things like customers to find customer bugs before customers did. Like customers do. That is pretty general, and yet to me, it is also specific. But it means, in theory, things like:
- Networking that is like customers – so storage networks, management networks, vMotion networks, etc.
- Security – no one uses the administrator account, and complex passwords are used, roles too. Services accounts used as well.
- Workloads – we would have a variety of workloads, to exercise things in a realistic manner
- Storage – we would have two kinds of storage, like many customers, and we would install / configure / and consume them like customers and use the tools that they come with like vSphere vC plug-ins that many storage companies have now.
- We would do backups, and implement DR using SRM when we could. In the beginning we did not have SRM in our mandate as we could only work with so many products when we started.
- We would have architects that had no admin access, and operations that did and ran things.
- Architecture – we would design, not like a engineering organization, but like a customer environment – management cluster for example, and operations using the design would build.
- Another generic type example, but important is upgrades – we would not do clean installs, but upgrades. So we would end up over time having the same amount of cruft that customers did.
- Anti – virus should be used as well, and preferably we would use the off – host scanning like this.
Most of all, the attitude is key. Meaning looking at things, and working at things like a customer.
In my own lab, which you saw some of here, I do in fact do all of this that I can. I do backups, patching, upgrading, no administrator account (yes, the password is complex and in an envelope), complex passwords, service accounts, realistic workloads (email, SQL, backup, web, View), separate networks, and more. This means I get to find interesting things. Like that complex passwords are not supported by an extension (VUM – fixed), or that when you regenerate an SSL certificate (on SSO VA) – I mean the Regenerate button was right there – that vCAC cannot connect to the SSO virtual appliance (Not Fixed). Another fun one was in an early version of SRM you could not have a password longer than maybe 13 characters, or a domain longer than 31 characters – I think 31 anyway (yes fixed).
Why does this matter? I see bugs in Software R&D in two categories – in this conversation anyway. Engineering bugs and Customer bugs. Engineering bugs are found by engineers and QE types. They are not always something that a customer would experience. Customer bugs are something that customers will generally find. Or said another way, that most but maybe not all customers will find. Understanding the two kinds of bugs mean you can prioritize the fixing of customer ones, and that is important. It can also means sometimes that Engineering bugs are not fixed as fast as Customer ones. Think about it, a bug that most customers will see, and one that most customers, maybe all, will not see. Which one would you like fixed first? Before Customer VMware did not always know the difference.
I had someone ask me why I have Dell R710’s and not Mac Mini’s in my lab. Customer is why. Specifically I cannot get enough networks in Mini’s as an example of why no Minis. And I need my lab as much like customers as possible. Years ago it was because I was a Professional Services guy who learned in his lab before I did things at customer site. Lately it has been because I really believe in Customer.
I should add that it turns out Customer is hard. It is easy at one level with guys like me, but it becomes political, and getting it started really needs help from the top down. But it is worth it!
Questions and comments are always welcome!