Bug Harvest Idea
PLEASE COMMENT-This is the first time I've taken this idea public and I'm really looking for honest, constructive feedback. Even if think it isn't at all useful, please explain why. It will help me.
Intent
A bug harvest is an organized and focused blackbox group testing exercise with a twist. It is called a "harvest" because just as those who pick strawberries may not be the fastest at picking apples, you want to select testers who already work on similar areas or technologies for effective and quick group testing in this context. The word harvest also implies that the fruit must be ripe. It is called a harvest because of the notion that bugs are introduced into the software with code changes, making certain times very "ripe" for the picking of bugs. Most experienced testers know exactly when that time is, and are used to using code changes to prioritize their testing. In some companies there is a handoff process when certain features are ready for test. There may be multiple times in the project where fast coverage by multiple people is very beneficial, especially if the development is staged. It is not meant to replace any other test methods, such as an open bug bash or bug hunt, but instead is an additional method that for use when bugs must be found under tight time constraints. It is intended to cover new code changes as quickly and efficiently as possible to compress the time it takes to get feedback and bugs to developers. It can be used in basically any software development model so long as your testing department has more than one tester, but I have only personally used it in a waterfall development cycle and one staged development model.
The other time I use this method is when fire drill requests come in. Wait, a new operating system and we have 3 days to test on it before they lock down the code? Oops, we forgot to tell your team about our new dot release, and it could impact compatability with your software. In this case, why not throw every tester in the world at the problem? Well, because it leads to duplicate bugs, takes everyone off of what they are working on and stalls your project, and because you can do just as well, especially in early stages with a small focused swarm of testers attacking the area as with a large misdirected frantic swarm. I love Bug Harvest as a method for dealing with firedrills and have gone to it more than 5 times in my last product cycle with great results.
Who, When, Where
One of the keys to holding a successful bug harvest is to invite the right testers. This is not a bug hunt or an open call for all people to beat on a new feature. This will most likely overwhelm the developer with bugs, some "not bugs" and create a large pile of defects to be sifted through which may or may not be about the feature you wanted information on. Only testers with relevant experience should be invited. This is your chance to try to assemble a dream team of testers who know the related technology and have some experience testing in a similar area. The best harvests include between 2 and 7 testers for a time period of 1-4 days depending on the size of the area under test. This can be done in one location, such as a lab, or in many locations, but the instant communication is also helpful, so on my team we use instant messaging if we are in multiple locations. I have tried this over multiple time zones, and while you can do it, to make a long story short, it kind of sucks and try to avoid this if you can. The instant contact helps prevent us from writing duplicate bugs. I also try to get at least one developer to agree to be online to answer our questions. I've been very fortunate and have never been turned down by a developer. In fact, many times they will come watch us test as a group and will be interested in the feedback.
I usually try to include one tester who isn't the best choice for bug finding, but has shown interest in learning the technology. This isn't required, but if you care about mentoring new testers, and the future testing at your company, I believe it is important. I had some wonderful mentors as a new tester and I feel I owe it to them to pass on the things that they taught me. If you select random people who have no experience with the software or with the technology, most of the time spent will not be productive because the ramp up time is too long. Remember, this method is intended to be the most efficient first, and that is a higher priority than spreading product knowledge across groups. It will spread out the knowledge across your test team, but that is not the main intent.
How
Many times a bug harvest will be most effective before test documentation has been written. I have spent months writing out perfect step by step test cases only to have them become completely useless within one version of the product. I was beyond smug when I handed that document off for review and got feedback that it was the most complete test script that the person had seen before. It hurt me deeply to see my 22 pages of meticulously hand crafted test cases including many matrices disappear into nothing useful. I'm very glad to have had that experience 3 years into my testing career because it changed the way I think about testing. When I first introduced this idea at my company, I devised a new way to create quick test documentation which can be followed by any tester who knows the technology to some extent. I quickly realized that not only was my idea not a good fit in every situation, but that it really doesn't matter how you get people to start in different areas and track the test coverage, only that you use a method that works for you and that the setup overhead is low, the coverage is spread out for minimal overlap, and that you can report back easily what was covered. If you can't assign out the testing in one hour based on your method, I gently suggest you use one of these other methods.
Note: When I say "assign" I really mean distribute in whatever way works for your test group. Some people like a lottery, pull out of a hat thing, others like to select like picking teams, and on my team we just assign them out because it is quicker and really, it's only a place to start, not a definative assignment. As long as everyone isn't starting in the same area, it doesn't matter how you split up the testing.
Some of the methods that would work:
-Create and assign charters using Session Based Test Management
-Create a model and assign out areas of the model using Model Based Testing
-A basic text outline with assignments
-Actions
-Feature sub-areas from a database, bug or test case
-Split up sections of a design spec for starting points
-In case of a firedrill, such as the OS example, split out the new OS changes and areas of high risk into areas and assign them out. Not to name any names, but I cannot confirm or deny that I've used the information on a website such as this to do a bug harvest in light of the lack of information provided.
-Roadmap
Note: This is my idea that I referenced above as having mixed results and I'll make a whole blog about it with examples. It is sort of an unholy mixture of model based testing for dummies with action words.
Do not do the unthinkable evil, please:
This method is not intended to be used with step-by-step test cases. This ruins the bug harvest. If you do this, imagine me crying somewhere in a closet. It is meant to be a creative and exploratory process while still getting efficient test coverage. If any monkey who can read can follow it, this micromanaging guidance is way too detailed to be used for a Bug Harvest. Based on previous experience, I don't have a very high opinion of the effectiveness of test cases, but even if you do, please take a break from them if you want to try this method. Once again, this method doesn't replace anything you are already doing, it is just something to try in addition.
When is it Over and What Then?
The harvest ends either when you are blocked and can test no more, when time is up, or when the feature changes are covered pretty well and new bugs have slowed to a trace, much like the non-popping watched microwave popcorn bag.
If it ended because you were out of time, you create a small summary and send it out. I like to include what areas were covered, what remains to be covered, what bugs were found, and general comments from the testers at the bottom regarding the intangible data, such as, what was your gut feeling about the stability, which areas are of higher risk, how did it feel to use the feature/area, and that sort of feedback. I'm sure that touchy feely kind of information wouldn't fly at all companies, but so far only a few non-testers have really been annoyed at me for it.
If it ended because you were blocked, explain where you were blocked and plan for another harvest if appropriate. Send out the same summary as you would normally.
If it ended because of lack of action, send out the summary, explain what was found and that you believe the harvest time is over for the feature unless more extreme changes are introduced. I'll explain how you can go back using some of the tracking methods to quickly do regression on a covered area in another blog. I call this the "tree top" method. In short, you run the most complicated case for each branch in the model as most code is built on other functionality which has to pass for the complicated case to work. This is the opposite of most automated systems which try to do the simplest task in the most isolated fashion so that it's easy to maintain, reuse, and isolate defects.
Why You Should Try It
-It's cheap. Other people give up a few days for you, you give up a few days for them. You lose nothing.
-It builds relationships and spreads out knowledge.
-It is actually fun!
-You will improve your speed in the areas of test planning, risk management, and test coverage reporting by doing it more often in smaller doses. You will feel more comfortable when emergencies come up having a plan that you have tried and that you always have in your back pocket.
-Developers get a load of bugs, and then we "go away" to focus on something else so that they can fix them for the next round.
-New eyes on the feature.
-We don't have to wait until the documentation is fully done and the feature is fully ready to start testing. Less wasted time with an out of date spec.
-Collaborative testing brings out a competative spirit in some people, and this leads to each person being a bit more productive than they might be working alone.
Intent
A bug harvest is an organized and focused blackbox group testing exercise with a twist. It is called a "harvest" because just as those who pick strawberries may not be the fastest at picking apples, you want to select testers who already work on similar areas or technologies for effective and quick group testing in this context. The word harvest also implies that the fruit must be ripe. It is called a harvest because of the notion that bugs are introduced into the software with code changes, making certain times very "ripe" for the picking of bugs. Most experienced testers know exactly when that time is, and are used to using code changes to prioritize their testing. In some companies there is a handoff process when certain features are ready for test. There may be multiple times in the project where fast coverage by multiple people is very beneficial, especially if the development is staged. It is not meant to replace any other test methods, such as an open bug bash or bug hunt, but instead is an additional method that for use when bugs must be found under tight time constraints. It is intended to cover new code changes as quickly and efficiently as possible to compress the time it takes to get feedback and bugs to developers. It can be used in basically any software development model so long as your testing department has more than one tester, but I have only personally used it in a waterfall development cycle and one staged development model.
The other time I use this method is when fire drill requests come in. Wait, a new operating system and we have 3 days to test on it before they lock down the code? Oops, we forgot to tell your team about our new dot release, and it could impact compatability with your software. In this case, why not throw every tester in the world at the problem? Well, because it leads to duplicate bugs, takes everyone off of what they are working on and stalls your project, and because you can do just as well, especially in early stages with a small focused swarm of testers attacking the area as with a large misdirected frantic swarm. I love Bug Harvest as a method for dealing with firedrills and have gone to it more than 5 times in my last product cycle with great results.
Who, When, Where
One of the keys to holding a successful bug harvest is to invite the right testers. This is not a bug hunt or an open call for all people to beat on a new feature. This will most likely overwhelm the developer with bugs, some "not bugs" and create a large pile of defects to be sifted through which may or may not be about the feature you wanted information on. Only testers with relevant experience should be invited. This is your chance to try to assemble a dream team of testers who know the related technology and have some experience testing in a similar area. The best harvests include between 2 and 7 testers for a time period of 1-4 days depending on the size of the area under test. This can be done in one location, such as a lab, or in many locations, but the instant communication is also helpful, so on my team we use instant messaging if we are in multiple locations. I have tried this over multiple time zones, and while you can do it, to make a long story short, it kind of sucks and try to avoid this if you can. The instant contact helps prevent us from writing duplicate bugs. I also try to get at least one developer to agree to be online to answer our questions. I've been very fortunate and have never been turned down by a developer. In fact, many times they will come watch us test as a group and will be interested in the feedback.
I usually try to include one tester who isn't the best choice for bug finding, but has shown interest in learning the technology. This isn't required, but if you care about mentoring new testers, and the future testing at your company, I believe it is important. I had some wonderful mentors as a new tester and I feel I owe it to them to pass on the things that they taught me. If you select random people who have no experience with the software or with the technology, most of the time spent will not be productive because the ramp up time is too long. Remember, this method is intended to be the most efficient first, and that is a higher priority than spreading product knowledge across groups. It will spread out the knowledge across your test team, but that is not the main intent.
How
Many times a bug harvest will be most effective before test documentation has been written. I have spent months writing out perfect step by step test cases only to have them become completely useless within one version of the product. I was beyond smug when I handed that document off for review and got feedback that it was the most complete test script that the person had seen before. It hurt me deeply to see my 22 pages of meticulously hand crafted test cases including many matrices disappear into nothing useful. I'm very glad to have had that experience 3 years into my testing career because it changed the way I think about testing. When I first introduced this idea at my company, I devised a new way to create quick test documentation which can be followed by any tester who knows the technology to some extent. I quickly realized that not only was my idea not a good fit in every situation, but that it really doesn't matter how you get people to start in different areas and track the test coverage, only that you use a method that works for you and that the setup overhead is low, the coverage is spread out for minimal overlap, and that you can report back easily what was covered. If you can't assign out the testing in one hour based on your method, I gently suggest you use one of these other methods.
Note: When I say "assign" I really mean distribute in whatever way works for your test group. Some people like a lottery, pull out of a hat thing, others like to select like picking teams, and on my team we just assign them out because it is quicker and really, it's only a place to start, not a definative assignment. As long as everyone isn't starting in the same area, it doesn't matter how you split up the testing.
Some of the methods that would work:
-Create and assign charters using Session Based Test Management
-Create a model and assign out areas of the model using Model Based Testing
-A basic text outline with assignments
-Actions
-Feature sub-areas from a database, bug or test case
-Split up sections of a design spec for starting points
-In case of a firedrill, such as the OS example, split out the new OS changes and areas of high risk into areas and assign them out. Not to name any names, but I cannot confirm or deny that I've used the information on a website such as this to do a bug harvest in light of the lack of information provided.
-Roadmap
Note: This is my idea that I referenced above as having mixed results and I'll make a whole blog about it with examples. It is sort of an unholy mixture of model based testing for dummies with action words.
Do not do the unthinkable evil, please:
This method is not intended to be used with step-by-step test cases. This ruins the bug harvest. If you do this, imagine me crying somewhere in a closet. It is meant to be a creative and exploratory process while still getting efficient test coverage. If any monkey who can read can follow it, this micromanaging guidance is way too detailed to be used for a Bug Harvest. Based on previous experience, I don't have a very high opinion of the effectiveness of test cases, but even if you do, please take a break from them if you want to try this method. Once again, this method doesn't replace anything you are already doing, it is just something to try in addition.
When is it Over and What Then?
The harvest ends either when you are blocked and can test no more, when time is up, or when the feature changes are covered pretty well and new bugs have slowed to a trace, much like the non-popping watched microwave popcorn bag.
If it ended because you were out of time, you create a small summary and send it out. I like to include what areas were covered, what remains to be covered, what bugs were found, and general comments from the testers at the bottom regarding the intangible data, such as, what was your gut feeling about the stability, which areas are of higher risk, how did it feel to use the feature/area, and that sort of feedback. I'm sure that touchy feely kind of information wouldn't fly at all companies, but so far only a few non-testers have really been annoyed at me for it.
If it ended because you were blocked, explain where you were blocked and plan for another harvest if appropriate. Send out the same summary as you would normally.
If it ended because of lack of action, send out the summary, explain what was found and that you believe the harvest time is over for the feature unless more extreme changes are introduced. I'll explain how you can go back using some of the tracking methods to quickly do regression on a covered area in another blog. I call this the "tree top" method. In short, you run the most complicated case for each branch in the model as most code is built on other functionality which has to pass for the complicated case to work. This is the opposite of most automated systems which try to do the simplest task in the most isolated fashion so that it's easy to maintain, reuse, and isolate defects.
Why You Should Try It
-It's cheap. Other people give up a few days for you, you give up a few days for them. You lose nothing.
-It builds relationships and spreads out knowledge.
-It is actually fun!
-You will improve your speed in the areas of test planning, risk management, and test coverage reporting by doing it more often in smaller doses. You will feel more comfortable when emergencies come up having a plan that you have tried and that you always have in your back pocket.
-Developers get a load of bugs, and then we "go away" to focus on something else so that they can fix them for the next round.
-New eyes on the feature.
-We don't have to wait until the documentation is fully done and the feature is fully ready to start testing. Less wasted time with an out of date spec.
-Collaborative testing brings out a competative spirit in some people, and this leads to each person being a bit more productive than they might be working alone.
I myself have used very much that method in a previous life when I ran a PC HW/SW compatibility test company (mid-late 1980's). We did not have any technical specifications and our requirements were that the products just worked. The method you described is ideal for that kind of situation, and there are many of those circumstances around.
I would caution those who embrace this method to use this prior to validation and user acceptance testing where Sarbanes-Oxley compliance is mandatory, such as in banking, insurance, securities - anywhere a public company is using software and especially where money is being manipulated including retirement and even bill-pay. Those circumstances require adherence to process and managed requirements.
Another place to exercise caution using this method is where an organization is CMM or ISO based in it's technology. As much as it's not necessarily fun to work within the confines of structure the fact is that audit is a fact of life and clear adherence to structured, reviewed, sign-off and validated process is mandatory.
In the traditional context of quality assurance the role of test is to validate a work product to its' specifications. The role that the method you described would be best deployed would be in organizations/projects where developers and testers are in the same organization and a separate group validates/accepts the product.
I am a senior SQA engineer that works with infrastructure projects, business applications, customer facing applications, marketing and sales tools, as well as enterprise-wide business systems. We don't sell or license out any of our software or processes, and our apps/systems are all based on specific business needs. In that environment our developers and systems analysts work in much the same way as you have described in the "harvest" idea, so I think you have a good, rational and usable idea. But not for qa in my case where I must validate the work products to specifications.
Thanks for articulating your concept in a well written and pragmatic manner. I would enjoy knowing what industry you work in and how you go about product validation.
~Will
Reply to this
I was quite confused, but then I don't know anything about testing. I'm sure if I was a tester I would have had more of an opinion. But I have no idea what you actually DO when you're testing. It seems to me that whenever I ask you about it I'll say something like, "This is what you do" and describe what I think testing is, then you tell me no, that's not right, and tell me what you do, but it sounds like I it the same thing that I originally said mostly. At least it doesn't seem to me to be significantly different. Oh, and there's a typo in the 2nd paragraph under "When is it over..." It should be intangible, not untangible. Unless it's some weird funky thing that I don't know about.
Reply to this
Good point! It's hard to describe my job partly because it's changed 3 times now. I was a tester on one product. Then I was a tester on two products. Then I was a test lead on one product. Now I'm a test lead working with every product that we have.
If I confused you with job 1, I'm sure to do even worse now.
Anyhow, I love having a sister who loves me enough to read my testing blog and I will try to make it less untangible. You have mad editoral skills even if you don't know testing.
Reply to this
Scratch that! According to the dictionary, Untangible is the primary spelling and intangible is the alternate. Strange.
Reply to this
I myself have used very much that method in a previous life when I ran a PC HW/SW compatibility test company (mid-late 1980's). We did not have any technical specifications and our requirements were that the products just worked. The method you described is ideal for that kind of situation, and there are many of those circumstances around.
I would caution those who embrace this method to use this prior to validation and user acceptance testing where Sarbanes-Oxley compliance is mandatory, such as in banking, insurance, securities - anywhere a public company is using software and especially where money is being manipulated including retirement and even bill-pay. Those circumstances require adherence to process and managed requirements.
Another place to exercise caution using this method is where an organization is CMM or ISO based in it's technology. As much as it's not necessarily fun to work within the confines of structure the fact is that audit is a fact of life and clear adherence to structured, reviewed, sign-off and validated process is mandatory.
In the traditional context of quality assurance the role of test is to validate a work product to its' specifications. The role that the method you described would be best deployed would be in organizations/projects where developers and testers are in the same organization and a separate group validates/accepts the product.
I am a senior SQA engineer that works with infrastructure projects, business applications, customer facing applications, marketing and sales tools, as well as enterprise-wide business systems. We don't sell or license out any of our software or processes, and our apps/systems are all based on specific business needs. In that environment our developers and systems analysts work in much the same way as you have described in the "harvest" idea, so I think you have a good, rational and usable idea. But not for qa in my case where I must validate the work products to specifications.
Thanks for articulating your concept in a well written and pragmatic manner. I would enjoy knowing what industry you work in and how you go about product validation.
~Will
Reply to this
Hi Will,
Thank you for your thoughtful comment. This method is not a good replacement for testing against strict requirements. For example, I wouldn't use this method even for something as simple as testing against a UI spec even though those requirements aren't as specific as what you are talking about. We don't use this method for testing OS logo requirements or other clearly outlined requirements either. That's a good cautionary note. I think it would be a total mess in that context. The main plus of this method is that it inspires creativity and is a fast way to get feedback early.
I don't want to reveal my company because this is not a company blog, but about testing in general, but I'm about to post another example of my roadmap idea, so one area that I was a QE lead for was XML on one product and we were on the same team as our developers. In my current job I lead testing across many different products. We test for integration, consistency, usability, performance, and many other things across entire product families. We work on commercially sold software that ships in boxes or is licensed mainly, although some of the products and features are web focused. I would say in a way, my testing is the polar opposite of someone working in one area of one product to strict specifications or government requirements. Often, by the time the documentation is ready, our testing effort would be too late.
I've counted, and I believe I've worked on 10 product releases at this point and have "owned" over 30 product areas. This is just one of the ways that we test, but this is the one that I came up with and started using, so I get really excited about it. While no ideas are really "new", this was a different type of testing at my company.
I've met a few other people working in the banking industry and similar areas of testing. While security testing is important for almost all software, I can envision is it a huge challenge and responsibility in those types of products.
Reply to this
Sorry, it's the editor in me:
s/Even if think/Even if you think/
s/an additional method that for use/an additional method that is for use/
Otherwise, looks good. It appears to be a "low-hanging fruit" testing technique that's pretty close to what I like to do on an initial run of a bunch of new code I've written up. Unfortunately, a lot of what I do does have regulatory requirements (such as SOX), so I have to go back and describe validation for each piece. However, this technique saves me time in the sense that I often have to change how the code works if I discover I've missed some logic flaw for a specific situation, so I'm saved all the time of re-writing the specific validation cases.
Reply to this
bug harvest => organized, focused blackbox group testing. "Harvest" for effective, quick group testing. What is an open bug bash? Bug hunt?
Tight time constraints, the tighter the better.Stretch goals enable humans to produce more. Stress is good. Assemble a dream team of testers who know the related technology in between 2 and 7 testers for a period of 1 - 4 days(intensive session pushed to exhaustion state, better than running a marathon, sweet sweat is good). I'm interested in learning the technology and would enjoy you being my mentor.
The summary in the form of a template. Automated test template web based and windows based GUI that performs calculations and statistics based on the input values. Collaboration among competitive people
increases overall productivity, make use of group dynamics. For example 6 male testers and you as the test lead. Just based on male instinct, each one will want to be the top performer/tester to impress you or win your heart.
Reply to this