Beyond the Requirements, Beyond the Bugs
Requirements-Not to be Taken as Scripture
I've been thinking of the important reasons to go beyond testing "to requirements". I've fought for and gotten 3 bugs fixed recently that are clearly stated as "not supported" in requirements. You have to make sure you have a good case for how many users it will impact, how leaving this as "unsupported" makes the product/company look bad, or has a negative impact on adopting the technology. Something like "_____________ happens when you use _____________ with _____________" and explain that ________ people use ___________ and the interaction is likely with __% of customers potentially having (customer impact). I find it helps to say that it is a usability problem and not a code bug so that it isn't instantly closed "as designed". "As designed" is no excuse if it is harmful to the user experience. I don't really check my deferred or closed "as designed" bug count or feel it is a problem to be corrected. I've had bugs closed "as designed" which were later huge customer complaints that got fixed in dot releases. I don't feel bad about having written those bugs. They were not a mistake. The key to testing "unsupported" areas and is making sure you know which ones are important, and never fighting for a trivial bug needlessly.
I attended some training where the teacher told us that it isn't our job as testers to be involved to this level and that those sorts of questions should be raised during requirements review. I agree that test needs to be involved in requirements review, of course, but some situations come up long after the requirements are set. How about new environments or technologies that become important? The web moves faster than our product cycles, so you have to be flexible and adapt as things change. I just realized, I seem to be making an argument for agile development as well as going beyond the requirements.
Keeping your mind active and the user first is always better than feeling "good" about the scope of your testing. If you are sure you did everything in "your area of ownership", but nothing beyond it, that isn't good enough to promote quality improvements.
On Bugs
As a test lead, I've had testers get really caught up in what their "bug count" is, or what percentage of them gets fixed. I hear little about what disaster was averted, or how important the bug was. Bug impact should be just as important as number of bugs. One high impact bug can be much more important than 50 bugs found. Don't measure your progress just in bugs. All it takes to have a high bug count is a very careless Dev counterpart. Testing talent isn't measured just in bugs. It should be measured on the improved overall quality going out the door. In fact, moving your testing upstream can be one of the most positive ways to impact quality. I'm pleased to see that happening more and more at my company.
Me, Company, Products, Presentation
In the past I haven't included anything about my specific company to avoid unforeseen problems. I'm speaking soon at http://www.pnsqc.org/ and put my blog URL in my profile, so who I am is now public domain on the internet. Please come see my presentation if you can. It is in Portland, OR in the Convention Center on Tuesday, October 14th from 10:15am-11am and is in the Testing track on the schedule and is called Testing for the User Experience. If you would like to read my bio, here it is.
My products were announced today! Well, I suppose since they are most of the work for an entire company, they aren't just MY products, but I'm proud as a new parent, so I consider them "mine".
Anyhow, do not ask me for software. I use my employee discount yearly to raise money for the MS Walk, and I adhere strictly to the rules. All money goes to my charity, the MS Society. My sister has MS and has lost vision in one eye due to it, robbing her of depth perception. It has a devastating impact on my family, so I'm very interested in finding a cure. Those who join my MS Walk team get first choice.
This is NOT a corporate blog or it would be hosted on the company website. These are my opinions as a tester in general. I do not speak for my company, but only for myself on my testing ideas, my career growth, and software in general. I'm proud to work on all of these products for the past 8 years as I used and loved them long before I ever worked on them. The things I love about blogging do not fit into a corporate blog easily. I think the company I work for is the best place to work in the industry.
I've been thinking of the important reasons to go beyond testing "to requirements". I've fought for and gotten 3 bugs fixed recently that are clearly stated as "not supported" in requirements. You have to make sure you have a good case for how many users it will impact, how leaving this as "unsupported" makes the product/company look bad, or has a negative impact on adopting the technology. Something like "_____________ happens when you use _____________ with _____________" and explain that ________ people use ___________ and the interaction is likely with __% of customers potentially having (customer impact). I find it helps to say that it is a usability problem and not a code bug so that it isn't instantly closed "as designed". "As designed" is no excuse if it is harmful to the user experience. I don't really check my deferred or closed "as designed" bug count or feel it is a problem to be corrected. I've had bugs closed "as designed" which were later huge customer complaints that got fixed in dot releases. I don't feel bad about having written those bugs. They were not a mistake. The key to testing "unsupported" areas and is making sure you know which ones are important, and never fighting for a trivial bug needlessly.
I attended some training where the teacher told us that it isn't our job as testers to be involved to this level and that those sorts of questions should be raised during requirements review. I agree that test needs to be involved in requirements review, of course, but some situations come up long after the requirements are set. How about new environments or technologies that become important? The web moves faster than our product cycles, so you have to be flexible and adapt as things change. I just realized, I seem to be making an argument for agile development as well as going beyond the requirements.
Keeping your mind active and the user first is always better than feeling "good" about the scope of your testing. If you are sure you did everything in "your area of ownership", but nothing beyond it, that isn't good enough to promote quality improvements.
On Bugs
As a test lead, I've had testers get really caught up in what their "bug count" is, or what percentage of them gets fixed. I hear little about what disaster was averted, or how important the bug was. Bug impact should be just as important as number of bugs. One high impact bug can be much more important than 50 bugs found. Don't measure your progress just in bugs. All it takes to have a high bug count is a very careless Dev counterpart. Testing talent isn't measured just in bugs. It should be measured on the improved overall quality going out the door. In fact, moving your testing upstream can be one of the most positive ways to impact quality. I'm pleased to see that happening more and more at my company.
Me, Company, Products, Presentation
In the past I haven't included anything about my specific company to avoid unforeseen problems. I'm speaking soon at http://www.pnsqc.org/ and put my blog URL in my profile, so who I am is now public domain on the internet. Please come see my presentation if you can. It is in Portland, OR in the Convention Center on Tuesday, October 14th from 10:15am-11am and is in the Testing track on the schedule and is called Testing for the User Experience. If you would like to read my bio, here it is.
My products were announced today! Well, I suppose since they are most of the work for an entire company, they aren't just MY products, but I'm proud as a new parent, so I consider them "mine".
Anyhow, do not ask me for software. I use my employee discount yearly to raise money for the MS Walk, and I adhere strictly to the rules. All money goes to my charity, the MS Society. My sister has MS and has lost vision in one eye due to it, robbing her of depth perception. It has a devastating impact on my family, so I'm very interested in finding a cure. Those who join my MS Walk team get first choice.
This is NOT a corporate blog or it would be hosted on the company website. These are my opinions as a tester in general. I do not speak for my company, but only for myself on my testing ideas, my career growth, and software in general. I'm proud to work on all of these products for the past 8 years as I used and loved them long before I ever worked on them. The things I love about blogging do not fit into a corporate blog easily. I think the company I work for is the best place to work in the industry.
I strongly agree with both your outlook and the teacher's, actually.
I believe that testers should be involved in requirements review, and that it should be possible to open bugs against the requirements, not just against the code.
So I think creating a bug against code which is known to be "as designed" is a poor use of existing channels. New channels need to be implemented so that reviewers (including testers) can cite problems with the requirements as what they really are - problems with the requirements, not as problems with the implementation. The current approach wastes too many resources as two groups (testers and developers) are forced to discuss an issue that neither one typically has the power to change.
A new forum where both testers and developers can cite defects with the requirements, and involve those responsible for the requirements directly, would be more efficient and less frustrating, I am confident.
Reply to this
You are so right on this one! Right now it is very difficult to write bugs against the requirements, but a process does need to be in place.
I'm going to bring this up and try to get a process started. I'll let you know what I find when I meet with Project Managers and Devs.
Reply to this