Unseen Metrics
How do businesses evaluate and measure their test effort? How do we know if we are doing the right thing, taking the right risks, or investing in the right area?
I've heard many presentations lately where automation teams will say that finding bugs is not their goal.
These developers in test have the goal of either driving a product direction (Test driven methodology), making sure bugs aren't reintroduced (Pure regression automation), validating that functionality is working and stays working (pure functional testing), or my favorite cause of all, inventing tools for other people to create tests (automation as a service).
So, what happens when the overall number of people expected to "find bugs" goes down and all of the investment is on the number of tests run? How are they prioritized, run, maintained, reported on, and how is quality evaluated? When you see a number, what numbers don't you see?
What did 90% code coverage cost you? What code did you cut? How is the user experience impacted? What did you trade for to get to that number? What didn't get tested? Is the product actually better for it?
I ask not because I have the answer, but because I am seeing some terrifying responses industry wide.
If testers don't find bugs and test the product who does?
Customers BUY and use the software. Why are businesses deciding not to invest in the product of value? That's the product of value? Why do I feel like I'm taking crazy pills to say something like this. I love good software. It makes my life better. I have passion for it. I don't want to use crappy software that some companies are making and I certainly don't want to participate in making really bad software to use.
Good software makes money. New methodology and internal tools do not, especially if they don't save money. Why is it politically incorrect to say this?
Metrics that matter=Customers buy product. Customers like product, can do what they want better than ever before, recommend product! Perceived quality from the customer is what matters, and long term at that.
Are you in quality? What are you doing to avoid buying the test snake oil? How do you convince others that having people to find bugs is important?
I've heard many presentations lately where automation teams will say that finding bugs is not their goal.
These developers in test have the goal of either driving a product direction (Test driven methodology), making sure bugs aren't reintroduced (Pure regression automation), validating that functionality is working and stays working (pure functional testing), or my favorite cause of all, inventing tools for other people to create tests (automation as a service).
So, what happens when the overall number of people expected to "find bugs" goes down and all of the investment is on the number of tests run? How are they prioritized, run, maintained, reported on, and how is quality evaluated? When you see a number, what numbers don't you see?
What did 90% code coverage cost you? What code did you cut? How is the user experience impacted? What did you trade for to get to that number? What didn't get tested? Is the product actually better for it?
I ask not because I have the answer, but because I am seeing some terrifying responses industry wide.
If testers don't find bugs and test the product who does?
Customers BUY and use the software. Why are businesses deciding not to invest in the product of value? That's the product of value? Why do I feel like I'm taking crazy pills to say something like this. I love good software. It makes my life better. I have passion for it. I don't want to use crappy software that some companies are making and I certainly don't want to participate in making really bad software to use.
Good software makes money. New methodology and internal tools do not, especially if they don't save money. Why is it politically incorrect to say this?
Metrics that matter=Customers buy product. Customers like product, can do what they want better than ever before, recommend product! Perceived quality from the customer is what matters, and long term at that.
Are you in quality? What are you doing to avoid buying the test snake oil? How do you convince others that having people to find bugs is important?
The conflict you are describing sounds like a discussion we had in my software metrics class a few weeks ago. In defining quality, you have quality which is "fitness for use" and quality which is "conformance to requirements." Both of these definitions are part of TQM (total quality management) which was a big deal in the 80's. What our prof. pointed out is that neither of these addresses whether or not the customer really likes the product. Perhaps an emphasis on too much automation has the potential to bring TQM back from the 80's like leg warmers.
Reply to this
Hi Marlena,
It sounds very much like TQM could be useful to companies. It's hard to believe that we are repeating some of the mistakes from the early days of software, but human nature remains a huge technical challenge and I think with all of the industry layoffs some of our development ecosystem is more like it was in the early wild west of software than it was 5 years ago in terms of the resources for testing and true quality analysis. We have so many new methodologies which should be aligning us more closely to the customer, but still many of the same challenges of creating quality software are coming up.
As much as I hate to admit it, I recently bought some stirrup pant tights and cracked up that they were available. Sometimes one DOES need to keep their legs warm bringing back the 80's.
Reply to this