The Ever Popular Pairwise Testing
Each time I review the "testing basics" to make sure I haven't forgotten anything (I do this each year or so, I recommend a refresher for anyone), the topic of Pairwise testing comes up again.
In trying to get into more depth on the topic, I came across an article from 2004 by James Bach. What I like about this article is the research he did.
Best practices only save time and money and improve efficiency if applied correctly and with context.
In trying to get into more depth on the topic, I came across an article from 2004 by James Bach. What I like about this article is the research he did.
Best practices only save time and money and improve efficiency if applied correctly and with context.
The typical pairwise testing story goes like this:
1) Pairwise testing protects against pairwise bugs
2)
while dramatically reducing the number tests to perform,3)
which is especially cool because pairwise bugs represent the majority of combinatoricbugs,
4)
and such bugs are a lot more likely to happen than ones that only happen with morevariables.
5)
Plus, you no longer need to create these tests by hand.Critical thinking and empirical analysis requires us to change the story:
1) Pairwise testing
might find some pairwise bugs2)
while dramatically reducing the number tests to perform, compared to testing allcombinations, but not necessarily compared to testing just the combinations that
matter.
3)
which is especially cool because pairwise bugs might represent the majority ofcombinatoric bugs,
or might not, depending on the actual dependencies amongvariables in the product.
4)
and some such bugs are more likely to happen than ones that only happen with morevariables,
or less likely to happen, because user inputs are not randomly distributed.5)
Plus, you no longer need to create these tests by hand, except for the work of analyzingthe product, selecting variables and values, actually configuring and performing the
test, and analyzing the results.
Just as an aside, no guilt. Didn't comment on a post of yours yesterday. Why? I worked untill 2:00 AM, set 6 alarm clocks for 3:00 AM to catch the plane from Munich to Marseille at 6:30 AM
and woke up at 7:15 AM this morning.Had to book a new flight for Monday morning and will work all night to not fall asleep and miss the second flight. My money is running out. Made some good contacts
at the Visual Studio ONE Conference in Munich and hope to be back soon and start work.99th day of job searching. I've kept busy in session based work à la James Bach Style
and increased my knowledge base. Well aware that XP style, agile, pair programming would have boosted even more my performance.Wow another article from James Bach; Important message, last sentence on page 1: "...warning testers against blindly accepting best practices." Heard it again at the
conference #VSOne: Never turn your brain off and stop thinking. James Bach also mentioned the keywords: "Critical Thinking" on a recent tweet. Okay, I was in a session
with the title "Coding Dojo for .NET Developers" and the presenter put us at ease saying to relax, which I did, so relaxed that my brain turned off and then I immediately fell
into the trap of the TDD exercise. That reminded me when I got the error message from my compiler "Relax the Data Set constraints". Now that pissed me off, when I'm debugging and
bug hunting I don't want to be relaxed, but in an itchy, adrenaline pumped up state. Careful about not establishing and following best practices in testing. That could
be said by someone who wants to get paid for doing nothing. We must measure all activities, processes and generate detailed reports for ourselves. Don't measure it, won't improve
and to inform management of the state of affairs. Pairwise testing or pairwise coding creates confidence, that is a more angled sword, it could generate a dynamic
where each individual contributor learns new stuff and wants to impress the other. Let's just say that you and me pair test, you are about 100 times more productive then
I am. I will slow you down and will feel guilty, so I will move my ass very fast to get up to par and learn some new stuff from ... so that I can show you something new too
and make your time worthwile. That could be a configuration, right? It's already 12:44 AM and I need to get some sleep. I'll continue on with James Bach's article tomorrow, procrastination.
I know, just reached MS Word and feel that this could excite me to the extent to keep me up all night.Fully agree that all best practices save time and money and improve efficieny if applied correctly and within context. Resistence to change: I observe that in my work patterns.
It's difficult to try something new, a new technique, keyboard shortcut, pattern, new chunk of code, copy and paste from the internet but that's an old moral issue I must rid myself of.
1) Pairwise testing: 2 brains have the double brain power and each brain is further stimulated by t
Reply to this
Read the article from JB several times as well as your post. What comes to mind is that if we were to engage in pairwise testing, it must be a form of competition and not fusion to please one another or else we will find less bugs. Got to set the context straight and probably a good idea to regularly exchange the pairing partners to avoid a less bug finding productive fusional state that all couples reach sooner or later.
Reply to this