Microsoft Testing has Changed!
About the Change
In 2007 I wrote a blog post on how to test better than Microsoft. It was based on information as an outsider pre-Vista and back as far as the Windows 2000 development lifecycle. At the time, I was convinced that Microsoft was determined to trail blaze a path to software hell and that no one was going to stop them. The unbalanced approach where "the code is all that matters" really concerned me for the user experience. Well, then something happened. Vista happened. And Microsoft changed! I don't mean changed slightly. I mean started taking good advice from thought leaders. The distance between reality and presented metrics became known and smart people did something to fix it! I am not sure if there was a test leadership change, or if it was due to layoffs, or what exactly changed there, but since Vista there is something new going on in Redmond. Maybe it was always there, but it wasn't visible to outsiders.
My opinion of testing at Microsoft has changed, but it isn't because I think I was wrong. It is because testing at Microsoft is different than I thought it was. Maybe I hadn't talked to enough testers on enough different teams, or maybe they are more open about sharing ideas than they were. The company culture has changed, or at least my view of it has changed. The arrogance has diminished. The attitude of "We have the right answer and everyone else in testing is to be mocked" no longer applies. They are sharing with the community and not just to bestow their wisdom from their pristine tower on the peasants, but they are opening a two-sided conversation.
And now? What happened to the Microsoft of old? Other companies are COPYING the failure in the old trail they have moved on from. I thought Microsoft was crazy three years ago, but who is crazy now? The companies who are following 3-5 year old ideas that have failed in this economy? I think that Microsoft, with the huge number of testers it has, really does influence testing overall. However, it seems that as the testers and higher up managers moved on from Microsoft some years ago they took with them ideas from the company at that time without the advantages of the experience since. Those companies who think Microsoft is no longer much of a threat? I think they might want to take another look. I think they might start to deliver some innovative software again. I am not saying they will, but they might. Do I know for sure? Nope. This is just my impression from what I've seen as an outsider right now. In 3 years my opinion might change again. The company might change again. Who knows?
At PNSQC the first Keynote speaker from Intel talked about quality mascots. First it was the Salmon--"Moving quality upstream", then it was the Hedgehog based on "Good to Great", until the companies declared "great" didn't turn out to last, then it was "Shift Left" which is to the speaker a repeat of "Move quality upstream" and he feels like the true mascot of quality is the old farting dog who no one notices unless it makes a stink. Hearing him talk about this rhetoric that is sold to quality groups industry wide made me realize that we're being tricked. The same cycle of ideas and mascots aren't working in this economy. Quality isn't in a groove, that's a RUT we're stuck in. We are chasing our own tail. Guess what? Microsoft isn't over simplifying the problem anymore. They have multiple thought leaders working on important things, like training! Really. Other companies have stopped going to conferences and training their QE people all together. They have deprioritized teaching their so called "Quality" teams who really are no more than baby programmers who have no testing experience anything at all about testing as a profession. Now who is determined to follow the rut straight to quality purgatory? The same companies who Microsoft based their course correction on to get better quality?
What company has the guts to stay their own course? It seems right now that only Apple has that courage.
So, I'm sure that some people reading this think I am wrong. Let me just say I hope so. Time will tell. I hope very much that I am wrong and that I've missed a critical aspect among the jargon and buzzwords. I hope that there is some listening going on with the talking and that the feedback is a loop and not a funnel. I really hope that quality software will continue to be produced from multiple companies because the competition is good for those of us who use computer software.
How much failure is needed before a company will course correct? In the case of Microsoft, it appears Vista was enough. How stubborn are other software companies prepared to be? Stubborn enough to not exist anymore? Stubborn enough to alienate most of their user base? I read this weekend about "The Quark Debacle" and wondered if in the future we'll be reading about that with other companies names as well. I very much hope not.
About Yesterday
Microsoft testers were very good to me! I was also thrilled to ask questions and find out that Leo Lee who helped with the arrangements and set up the meeting was a volunteer to do these tasks. When people in QE take time out of testing to educate themselves and their fellow testers that is something of value to the company and to see that Microsoft realizes it? Good for them.
I did go into more detail on each of the bloat reducing ideas, but I should have cut some from the start of the presentation so I could have summarized. This is for those who attended.
The summary I wish I'd given:
Reducing test case bloat is a relatively new activity to add to your testing tasks. During test planning I find it useful to specify when I'm going to "maintain" and "define" my test case sets. Balancing the intentional risk with the time savings can allow you to focus on the highest priority test cases, giving your software an advantage when compared against an overly bloated test case database diverting time and attention from the highest impact tests. In addition to intentionally cutting test cases, organizing your tests into tiered structures can help you test in order of stakeholder priority making your approach more pragmatic and efficient. If bloat is not yet a problem for your team, and you are still working hard to create enough tests to get to code coverage and automation percentage goals, keep this idea tucked away for the future. If your legacy test cases have become unmanageable and too expensive to maintain and they are impeding you from moving forward, consider this survival guide a resource to help you create a better balance so that you can focus on the future while still dealing with the past.
In 2007 I wrote a blog post on how to test better than Microsoft. It was based on information as an outsider pre-Vista and back as far as the Windows 2000 development lifecycle. At the time, I was convinced that Microsoft was determined to trail blaze a path to software hell and that no one was going to stop them. The unbalanced approach where "the code is all that matters" really concerned me for the user experience. Well, then something happened. Vista happened. And Microsoft changed! I don't mean changed slightly. I mean started taking good advice from thought leaders. The distance between reality and presented metrics became known and smart people did something to fix it! I am not sure if there was a test leadership change, or if it was due to layoffs, or what exactly changed there, but since Vista there is something new going on in Redmond. Maybe it was always there, but it wasn't visible to outsiders.
My opinion of testing at Microsoft has changed, but it isn't because I think I was wrong. It is because testing at Microsoft is different than I thought it was. Maybe I hadn't talked to enough testers on enough different teams, or maybe they are more open about sharing ideas than they were. The company culture has changed, or at least my view of it has changed. The arrogance has diminished. The attitude of "We have the right answer and everyone else in testing is to be mocked" no longer applies. They are sharing with the community and not just to bestow their wisdom from their pristine tower on the peasants, but they are opening a two-sided conversation.
And now? What happened to the Microsoft of old? Other companies are COPYING the failure in the old trail they have moved on from. I thought Microsoft was crazy three years ago, but who is crazy now? The companies who are following 3-5 year old ideas that have failed in this economy? I think that Microsoft, with the huge number of testers it has, really does influence testing overall. However, it seems that as the testers and higher up managers moved on from Microsoft some years ago they took with them ideas from the company at that time without the advantages of the experience since. Those companies who think Microsoft is no longer much of a threat? I think they might want to take another look. I think they might start to deliver some innovative software again. I am not saying they will, but they might. Do I know for sure? Nope. This is just my impression from what I've seen as an outsider right now. In 3 years my opinion might change again. The company might change again. Who knows?
At PNSQC the first Keynote speaker from Intel talked about quality mascots. First it was the Salmon--"Moving quality upstream", then it was the Hedgehog based on "Good to Great", until the companies declared "great" didn't turn out to last, then it was "Shift Left" which is to the speaker a repeat of "Move quality upstream" and he feels like the true mascot of quality is the old farting dog who no one notices unless it makes a stink. Hearing him talk about this rhetoric that is sold to quality groups industry wide made me realize that we're being tricked. The same cycle of ideas and mascots aren't working in this economy. Quality isn't in a groove, that's a RUT we're stuck in. We are chasing our own tail. Guess what? Microsoft isn't over simplifying the problem anymore. They have multiple thought leaders working on important things, like training! Really. Other companies have stopped going to conferences and training their QE people all together. They have deprioritized teaching their so called "Quality" teams who really are no more than baby programmers who have no testing experience anything at all about testing as a profession. Now who is determined to follow the rut straight to quality purgatory? The same companies who Microsoft based their course correction on to get better quality?
What company has the guts to stay their own course? It seems right now that only Apple has that courage.
So, I'm sure that some people reading this think I am wrong. Let me just say I hope so. Time will tell. I hope very much that I am wrong and that I've missed a critical aspect among the jargon and buzzwords. I hope that there is some listening going on with the talking and that the feedback is a loop and not a funnel. I really hope that quality software will continue to be produced from multiple companies because the competition is good for those of us who use computer software.
How much failure is needed before a company will course correct? In the case of Microsoft, it appears Vista was enough. How stubborn are other software companies prepared to be? Stubborn enough to not exist anymore? Stubborn enough to alienate most of their user base? I read this weekend about "The Quark Debacle" and wondered if in the future we'll be reading about that with other companies names as well. I very much hope not.
About Yesterday
Microsoft testers were very good to me! I was also thrilled to ask questions and find out that Leo Lee who helped with the arrangements and set up the meeting was a volunteer to do these tasks. When people in QE take time out of testing to educate themselves and their fellow testers that is something of value to the company and to see that Microsoft realizes it? Good for them.
I did go into more detail on each of the bloat reducing ideas, but I should have cut some from the start of the presentation so I could have summarized. This is for those who attended.
The summary I wish I'd given:
Reducing test case bloat is a relatively new activity to add to your testing tasks. During test planning I find it useful to specify when I'm going to "maintain" and "define" my test case sets. Balancing the intentional risk with the time savings can allow you to focus on the highest priority test cases, giving your software an advantage when compared against an overly bloated test case database diverting time and attention from the highest impact tests. In addition to intentionally cutting test cases, organizing your tests into tiered structures can help you test in order of stakeholder priority making your approach more pragmatic and efficient. If bloat is not yet a problem for your team, and you are still working hard to create enough tests to get to code coverage and automation percentage goals, keep this idea tucked away for the future. If your legacy test cases have become unmanageable and too expensive to maintain and they are impeding you from moving forward, consider this survival guide a resource to help you create a better balance so that you can focus on the future while still dealing with the past.
great observation, Lanette.
Reply to this