Exploratory Testing-Still effective despite the polish being off the apple.
Last night I told Craig about a bug I was glad to have found doing exploratory testing and how per the usual it makes me sad that it is undervalued as a test methodology. I told him I would try again to at least tell my boss that we are still finding quite important bugs with it.
Craig asked, "So, are these different people you are trying to convince that it is important, or the same ones?"
Me, "Oh. Yeah. Still the same ones."
I proceeded to think about it with a disappointed look. I've tried to not care about it, and it isn't that I disagree with using a variety of good quality practices at all, including well planned automation, using existing tools, and even creating tools sometimes. In fact, I'm involved in many parts of quality planning and execution. I just don't understand why having talent for exploratory testing isn't valued as one skill among many. It makes me sad because I'm not mediocre at it. I'm really fantastic at it and I love doing it sometimes.In fact, I quite enjoy doing other things as well. I just don't understand why the ROI on exploratory testing is so often ignored, considered too costly, or just skipped as being "the past", just as the ROI is inflated on automated testing when it is still missing so many bugs of critical importance and it is still so often sucking in the areas of sustainability and cost of adapting the automation to a changing product under test.
Reality. How much does it change when your perception of it does? According to "the Secret" all of it.
I'm planning to head over to see Harry Robinson speak tonight if my pain level is conducive to me sitting for that much of the day. There is no reservation status for "Living in a cage of pain. If walking with only a slight limp I'll be there, if grimacing and walking with severe limp signs point to no."
UPDATE: I drove over there and was so confused about why no one was there. Ummm DUH--It's September 9th!! Guess I hope that's a decent pain day because I'm excited about it.
I'd like to thank the good Dr. K and my wonderful and patient boss that I'm still employed at this time. I care very much about the success of the products I work on and the overall quality they produce. I put my talents to work in any way I can to reach the overall goals.
Talking about having good results with exploratory testing is sort of like talking about the importance of still testing what customers are using. It may be true, but it isn't an interesting story that people want to invest in. It doesn't sound good and no one wants to talk about it who has any power to finance more of it. It is something just taken for granted, like dinner being ready when you get home, where you don't comment on it or really notice until one day the suitcase is packed and you are eating takeout on the floor wondering why you didn't notice sooner that you had it pretty good. Because "it is being tested" no longer means anyone human is looking at it. It doesn't mean anyone has USED it or even evaluated it in a real scenario. I met an engineer I really respect in the lunch room yesterday who told me off the record about his concern that test coverage is so thin in terms of anyone looking at it with testing intent or even using the software anymore and he was commiserating with me mostly just worried about the end result. He isn't a lazy developer like some who just check in crappy code. He's one of the guys who even in the early days wasn't the cause of breaking the build even with complex changes. So, why is he sharing this concern with me in the lunchroom in near silence to all peers and higherups?
Because, exploratory testing needs to get back in the kitchen where it belongs so we can ignore it and take it for granted. We've got new exciting and unproven test methodologies to go on tumultuous first dates with. Only in desperation at the last minute when our new methods have failed to show up for us will we come back to the old and unexciting exploratory testing which has served us well in the past. There it is, our unstated backup plan. We call it, what worked before. It's our sure thing. Break out in case of emergency. There are a few of us here who are really good at that sort of thing and also can do other things. What about the day when there is no backup left? Perhaps I am no good because I don't move forward with wild abandon and easily forget about what worked in the past. I have an uncommon and strange sense of loyalty to my product, my customers, my team, and also to what has worked in the past. Moving forward for me means learning and adapting but only dropping the past entirely when it is no longer needed, not before. It has to be annoying though for a person who's already moved on to hear again from Testy Redhead, "Hello? This is still needed. Let's give Old Faithful another look?"
Guess I'll head to the kitchen now.
Craig asked, "So, are these different people you are trying to convince that it is important, or the same ones?"
Me, "Oh. Yeah. Still the same ones."
I proceeded to think about it with a disappointed look. I've tried to not care about it, and it isn't that I disagree with using a variety of good quality practices at all, including well planned automation, using existing tools, and even creating tools sometimes. In fact, I'm involved in many parts of quality planning and execution. I just don't understand why having talent for exploratory testing isn't valued as one skill among many. It makes me sad because I'm not mediocre at it. I'm really fantastic at it and I love doing it sometimes.In fact, I quite enjoy doing other things as well. I just don't understand why the ROI on exploratory testing is so often ignored, considered too costly, or just skipped as being "the past", just as the ROI is inflated on automated testing when it is still missing so many bugs of critical importance and it is still so often sucking in the areas of sustainability and cost of adapting the automation to a changing product under test.
Reality. How much does it change when your perception of it does? According to "the Secret" all of it.
I'm planning to head over to see Harry Robinson speak tonight if my pain level is conducive to me sitting for that much of the day. There is no reservation status for "Living in a cage of pain. If walking with only a slight limp I'll be there, if grimacing and walking with severe limp signs point to no."
UPDATE: I drove over there and was so confused about why no one was there. Ummm DUH--It's September 9th!! Guess I hope that's a decent pain day because I'm excited about it.
I'd like to thank the good Dr. K and my wonderful and patient boss that I'm still employed at this time. I care very much about the success of the products I work on and the overall quality they produce. I put my talents to work in any way I can to reach the overall goals.
Talking about having good results with exploratory testing is sort of like talking about the importance of still testing what customers are using. It may be true, but it isn't an interesting story that people want to invest in. It doesn't sound good and no one wants to talk about it who has any power to finance more of it. It is something just taken for granted, like dinner being ready when you get home, where you don't comment on it or really notice until one day the suitcase is packed and you are eating takeout on the floor wondering why you didn't notice sooner that you had it pretty good. Because "it is being tested" no longer means anyone human is looking at it. It doesn't mean anyone has USED it or even evaluated it in a real scenario. I met an engineer I really respect in the lunch room yesterday who told me off the record about his concern that test coverage is so thin in terms of anyone looking at it with testing intent or even using the software anymore and he was commiserating with me mostly just worried about the end result. He isn't a lazy developer like some who just check in crappy code. He's one of the guys who even in the early days wasn't the cause of breaking the build even with complex changes. So, why is he sharing this concern with me in the lunchroom in near silence to all peers and higherups?
Because, exploratory testing needs to get back in the kitchen where it belongs so we can ignore it and take it for granted. We've got new exciting and unproven test methodologies to go on tumultuous first dates with. Only in desperation at the last minute when our new methods have failed to show up for us will we come back to the old and unexciting exploratory testing which has served us well in the past. There it is, our unstated backup plan. We call it, what worked before. It's our sure thing. Break out in case of emergency. There are a few of us here who are really good at that sort of thing and also can do other things. What about the day when there is no backup left? Perhaps I am no good because I don't move forward with wild abandon and easily forget about what worked in the past. I have an uncommon and strange sense of loyalty to my product, my customers, my team, and also to what has worked in the past. Moving forward for me means learning and adapting but only dropping the past entirely when it is no longer needed, not before. It has to be annoying though for a person who's already moved on to hear again from Testy Redhead, "Hello? This is still needed. Let's give Old Faithful another look?"
Guess I'll head to the kitchen now.


I'm still in shock that there isn't a huge Usability department there. I mean, last I checked, Microsoft has dedicated THREE WHOLE BUILDINGS to Usability, complete with custom video labs and such. It's not a new thing there either - they started their Usability program about 10 years ago. And their products aren't even designed for the presentation-focused users like yours are. So how has your upper management justified *not* keeping up with the industry? *boggle*
Reply to this
Oh dear. Now I've gone and mislead you. We have multiple groups, both an industry leading user experience and design team so amazing that the empire has tried to brain drain it, and then we've fought back and well,..so yes! Usability is quite important. There is also user research with major studies done, so I didn't mean to indicate that there wasn't in any ways.
What I meant to say is that within quality we are still doing exploratory testing and I find it quite effective still. In keeping up with the industry, of course, we have huge automation efforts. Because of industry changes more testing moves upstream, to whitebox, and less and less time is spent with exploratory testing DURING testing phases (Not less attention on the user experience company wide at all mind you, two different things!).
Now, when the automation and upfront design is so good that we no longer uncover many important bugs I could concede that less exploratory testing is needed. However, until such a time happens what I see instead is a dangerous testing gap and I rush to plug it much like the little Dutch boy. You see, we testers can not resist. Well, then Craig asks me what in the heck I'm doing on my laptop at 10pm while we are trying to watch a movie and I get in trouble and tell him I'm doing unimportant testing that wasn't resourced.
Reply to this
Hi Lanette,
You make some great points. Thanks for sharing them.
Best wishes,
Laura
Reply to this
Hi,
I think next year you need to go to CAST and hang out with people who have been successful with exploratory testing in all sorts of environments.
For now, you might want to check out http://www.associationforsoftwaretesting.org/drupal/CAST2009/Sessions#Andersson
which was a talk about introducing exploratory testing into an environment that was initially hostile to it, but showed great results (bugs found that that "holy automation" system missed).
Have fun and don't give up.
Reply to this
To quote Michael Bolton (from memory), could be slightly different in the original.
"Would you be interested in a bug even if I discovered it in a way that you didn't expect?"
That question might help a bit to drive the point home to the exploratory sceptics that as long as you find bugs you're doing a good job. And if your method of finding bugs is exploratory, does it matter? Is there a reason why a successful method shouldn't be more widely used?
You might have gone down that road already but it's a good start to get a good discussion rolling with the most important argument on your side - you're finding the bugs.
Good post
Reply to this