How to Test Better Than Microsoft
Summary: Make usability and the customer's human experience using the software more important than creation and use of any particular method or tool in testing. You can't beat them in tools, number of people, amount of money, automation, or willingness to be competitive (well, I would argue that dog fighting rings could beat them in being competitive, but that would just be snarky). They have one huge weak spot, and that is in the area of creativity and seeing important things that can't be measured.
Judging from my views on Automation, I'm guessing most of you can deduce that I'm not a Microsoft Employee. Many people feel that Microsoft is a huge leader in software, and for that reason, they must have the best quality principles and methodologies in use, and every other company should follow them. In my blog today, I'd like to share why I completely disagree with that notion.
Having talked with many people, both Microsofties and Microserfs former and current, there are many positive things about testing at Microsoft. The availability of tools, the ease of finding them, the constant process improvements being driven by the competitive culture, and even the perception that aggression is "effective" in some groups have been sighted as strengths. I've also heard that the culture encourages people to take on side-projects which impact as many people as possible in order to get raises and bonuses. I am not suggesting that good testing isn't happening there, I'm only suggesting that it is possible to do better. I think that Google IS doing better. I think that our company IS doing better. If companies take the lead from Microsoft while they are growing, they will not be doing better for long. I haven't seen anyone have luck at imitating them and still produce better software than they do.
1. Currently having the most money is not a good measure of testing success.
2. Saving the most money is not a good measure of testing success.
3. Many of their users perceive their quality as very low and hate using their products.
4. Their products have very little "soul" and feel bland, generic, and slightly boring and inhuman in general. From this I assume that usability bugs are not taken very seriously, or, even worse, aren't being found and reported because it isn't easily tested using automation. How can something both be boring and "needlessly complicated?" Use Outlook and find out.
5. Who can automate the most and who can pass code coverage are also not a good measures of testing success. Covering the code is NOT equal to testing the product.
6. Who can cover their butts the most while testing is also not a good measure of testing success.
7. Having highest percentage of testers who can code does not mean you have the best test team.
8. The lengths someone will go to, cutting out their personal life balance is not a good measure of employee loyalty and quality.
How can I comment on this having never tested at Microsoft? First, a confession. I was married for 5 years (as previously indicated in my blog) to someone who was a tester at Microsoft. I not only got quite an interesting glimpse of the culture on that particular test team, but got to meet many of the testers. I can assure you that coding skill was one of the most valued assets as was "not being wrong in email". I'm also quite clear that since this was during the XP time that there is now a "new school" style of working at Microsoft, much as there is in any company five years later. Testing has changed, and I'm not the most up to date or informed on the current goings on there. I still think my old knowledge applies, because many smaller companies are now catching up in terms of momentum and size.
At my job, if you sent me a scathing email broadly distributed, stating what an idiot I was, (trust me, it has been done once), you could expect to be kindly reminded that we don't have a culture of killing our young and we aren't in ancient Sparta, so knock it off! If you continued to do it, it will be at your own peril. We humans are allowed mistakes, wrong assumptions, and the ability to make it right. Being civil isn't optional and only to those in a position above you. I can only hope that this misguided spirit of competition within teams isn't as extreme as it was many years ago. I take my role as a mentor to new testers very seriously and the lack of direct competition within our team is a huge part of what allows us to collaborate effectively in my opinion. If I am less able to get a raise if I help my co-worker, where is the motivation for us to work together? Team goals with team rewards. Call me a communist, but I prefer it. It may not be as motivating as the outright competition that other companies use, but each time I snag a smart person from their test team, it brings me joy to try to "de-Microsoft" them. Some people have had so much Kool-Aid that they can't have the rabid dog taken out of them. They continue to be very stressed and burn themselves out even though it isn't required anymore. They want to know where all of our "process" is, and why we don't have more "tools" and see us as quite disorganized.
What happens when we get everything organized? Do our products then become soulless too? Does thinking and human testing stop? Count me as one person on the train asking where it is headed. If we haven't experienced the product like a user would, do we even know what we are shipping?
I ran into someone from my old team today and asked him, "Which collaborative testing techniques did you use this release? Did you still use any of (listed 3)?" He said no. That the push was on automation and that was most of the testing that took place. They spent more time troubleshooting false alarms than ever before. I asked him, "Was it fun?" Of course, like every other tester I've asked, he said no. I asked him if he thought they would have found more bugs with the human testing versus spending more time on the automation. He gave me the same answer about how automated tests are an investment in the future because we can cover behind us, blah blah. He failed to mention the time it takes to maintain those tests. He said that finding those bugs up front wasn't that important anymore as we are constantly releasing updates on the web, so the cost of a missed bug is much lower.
What? We now put the burden of testing our products on the user who paid for it? How fair is that?
Just as fair as going to a place where you serve yourself and still being expected to put a tip in the tip jar. It works just until consumers say no. Maybe that will never happen. I still think that testing must be about protecting the user from low quality software and a bad experience to the best of our ability.
Being that I'm not of the Microsoft culture, I don't feel that we need to beat them and kill all other companies. I hope that we all can do well. The places I most want to beat them are in creativity, kindness, and quality. I think we have a great chance so long as we don't try to imitate them.
Judging from my views on Automation, I'm guessing most of you can deduce that I'm not a Microsoft Employee. Many people feel that Microsoft is a huge leader in software, and for that reason, they must have the best quality principles and methodologies in use, and every other company should follow them. In my blog today, I'd like to share why I completely disagree with that notion.
Having talked with many people, both Microsofties and Microserfs former and current, there are many positive things about testing at Microsoft. The availability of tools, the ease of finding them, the constant process improvements being driven by the competitive culture, and even the perception that aggression is "effective" in some groups have been sighted as strengths. I've also heard that the culture encourages people to take on side-projects which impact as many people as possible in order to get raises and bonuses. I am not suggesting that good testing isn't happening there, I'm only suggesting that it is possible to do better. I think that Google IS doing better. I think that our company IS doing better. If companies take the lead from Microsoft while they are growing, they will not be doing better for long. I haven't seen anyone have luck at imitating them and still produce better software than they do.
1. Currently having the most money is not a good measure of testing success.
2. Saving the most money is not a good measure of testing success.
3. Many of their users perceive their quality as very low and hate using their products.
4. Their products have very little "soul" and feel bland, generic, and slightly boring and inhuman in general. From this I assume that usability bugs are not taken very seriously, or, even worse, aren't being found and reported because it isn't easily tested using automation. How can something both be boring and "needlessly complicated?" Use Outlook and find out.
5. Who can automate the most and who can pass code coverage are also not a good measures of testing success. Covering the code is NOT equal to testing the product.
6. Who can cover their butts the most while testing is also not a good measure of testing success.
7. Having highest percentage of testers who can code does not mean you have the best test team.
8. The lengths someone will go to, cutting out their personal life balance is not a good measure of employee loyalty and quality.
How can I comment on this having never tested at Microsoft? First, a confession. I was married for 5 years (as previously indicated in my blog) to someone who was a tester at Microsoft. I not only got quite an interesting glimpse of the culture on that particular test team, but got to meet many of the testers. I can assure you that coding skill was one of the most valued assets as was "not being wrong in email". I'm also quite clear that since this was during the XP time that there is now a "new school" style of working at Microsoft, much as there is in any company five years later. Testing has changed, and I'm not the most up to date or informed on the current goings on there. I still think my old knowledge applies, because many smaller companies are now catching up in terms of momentum and size.
At my job, if you sent me a scathing email broadly distributed, stating what an idiot I was, (trust me, it has been done once), you could expect to be kindly reminded that we don't have a culture of killing our young and we aren't in ancient Sparta, so knock it off! If you continued to do it, it will be at your own peril. We humans are allowed mistakes, wrong assumptions, and the ability to make it right. Being civil isn't optional and only to those in a position above you. I can only hope that this misguided spirit of competition within teams isn't as extreme as it was many years ago. I take my role as a mentor to new testers very seriously and the lack of direct competition within our team is a huge part of what allows us to collaborate effectively in my opinion. If I am less able to get a raise if I help my co-worker, where is the motivation for us to work together? Team goals with team rewards. Call me a communist, but I prefer it. It may not be as motivating as the outright competition that other companies use, but each time I snag a smart person from their test team, it brings me joy to try to "de-Microsoft" them. Some people have had so much Kool-Aid that they can't have the rabid dog taken out of them. They continue to be very stressed and burn themselves out even though it isn't required anymore. They want to know where all of our "process" is, and why we don't have more "tools" and see us as quite disorganized.
What happens when we get everything organized? Do our products then become soulless too? Does thinking and human testing stop? Count me as one person on the train asking where it is headed. If we haven't experienced the product like a user would, do we even know what we are shipping?
I ran into someone from my old team today and asked him, "Which collaborative testing techniques did you use this release? Did you still use any of (listed 3)?" He said no. That the push was on automation and that was most of the testing that took place. They spent more time troubleshooting false alarms than ever before. I asked him, "Was it fun?" Of course, like every other tester I've asked, he said no. I asked him if he thought they would have found more bugs with the human testing versus spending more time on the automation. He gave me the same answer about how automated tests are an investment in the future because we can cover behind us, blah blah. He failed to mention the time it takes to maintain those tests. He said that finding those bugs up front wasn't that important anymore as we are constantly releasing updates on the web, so the cost of a missed bug is much lower.
What? We now put the burden of testing our products on the user who paid for it? How fair is that?
Just as fair as going to a place where you serve yourself and still being expected to put a tip in the tip jar. It works just until consumers say no. Maybe that will never happen. I still think that testing must be about protecting the user from low quality software and a bad experience to the best of our ability.
Being that I'm not of the Microsoft culture, I don't feel that we need to beat them and kill all other companies. I hope that we all can do well. The places I most want to beat them are in creativity, kindness, and quality. I think we have a great chance so long as we don't try to imitate them.
An interesting perspective, especially since you knew someone working there. Since it's about 2.5 years since you wrote this post, have you changed your mind on any of these points?
Reply to this
Yes I sure have changed my view! Mostly since I've learned more about what is on at the company.
I believe that the lessons learned from an unbalanced diet of Automation (all checking with little testing) are being applied in interesting ways. Hearing from some of the brilliant testers like Jean and Alan has totally changed my arrow minded view and now I see that while some of the data presented doesn't present quite as full and rigorous a view as ideally I'd like to see in metrics, their strategy has evolved significantly in the last 3 years, let alone the insight I had that is now over 5 years old.
Reply to this
We want to learn and cultivate underdog spirit, never too sure of ourselves, always aware that all of our processes can be improved. Test better than Microsoft? Who works better than Microsoft? On planet earth? In the Universe?
Reply to this