What in the world did I say a test case was?
From my technical paper Reducing Test Case Bloat:
Clinically defined a test case is an input and an expected result.* For my purposes it does not matter if a test case is automated or manual so long as it is a planned test. For the purpose of reducing test case bloat, I’d go further and say that it is a test you plan to execute a minimum of once in the product life cycle.
*IEEE (1998). IEEE standard for software test documentation. New York: IEEE. ISBN 0-7381-1443-X.
Why would I say that? Did I think about it? What am I talking about with the "planned" stuff? Do I really think that to qualify as a test case that a test must be "planned" and what does "planned" even mean?
I want to summarize why I didn't go in to what a test case is in this paper. I didn't think it would help. My paper is a survival guide. If I set even ONE tester partly free from a useless legacy test case so that they can do testing of value again, it is worth this shoddy definition of a test case to me. My intention is to offer a paper that helps a real human testing in a company go back with a practical list they can apply to the legacy test suite in their test case database. I wrote this paper to help testers, and I hope at the same time I was careful enough to not harm testing as a profession by limiting the scope. If I was harmful with this definition I apologize. I didn't mean to be. I do think that most of the test case bloat is included in this narrow definition of test case. This is not how I define a test case most of the time, but it is how I look for test case bloat.
My paper has a purpose. The purpose is not aspirational nor is it a paper about theory. What I mean by that is this paper is intended to help a person testing at a company which has test case bloat. In the situation where you have scripted test cases piled up to deal with and it has become critical to address, my paper is intended to help a tester evaluate existing planned test cases in order build their own escape plan. Wouldn't it be better to prevent it from getting to that point? Yes, but it is too late for most testers in large companies already.
Think about that person responsible for legacy test cases which are scripted, in most cases with very low craftsmanship. How are they going to contribute to better software quality? All of their testing time is being eaten up by test cases of low value which are unlikely to find bugs. When I was in this situation I'd come to work in the morning, and have hundreds of failures reported from my automation to sift through. Most of the time they weren't even bugs. It was a total waste of my brain. Keep in mind that the testers who get these huge legacy regression suites assigned to them are treated as if it is no big deal. After all, "It's automated" so that means it should be free now, or at least low cost in terms of tester time, and no one wants to admit they have failed. A huge investment went into creating these fragile and low value automated tests. If you admit the amount of time it actually takes you are branded as a low performer with a bad attitude who isn't on board with "quality progress". I want to help those testers find away out so that they can go to lunch again and so that they can build a case to spend less time on these regression test suites and use their creativity to invent and run new tests which are more likely to find important bugs and improve the user experience.
So, do I believe that there SHOULD be huge legacy regression test suites that waste tester time, return little value, and are hard to maintain? I do not. I think that there are scripted and planned test cases of value, both automated and manual, which can be used together with other testing methods to create a balanced test plan in most situations. I think we have a long way to go towards making our test automation and scripted human executed test cases better, more maintainable, and most of all with verification logic that doesn't suck. The current way we are validating pass and fail results in test automation is usually limited and insufficient for returning major value for the time invested. If I like it or not, we individual contributor testers are facing a problem of test case bloat right now, so I wanted to bring up the topic and provide some helpful suggestions that I've tried in practice. Is my definition good enough? Certainly not, but it is accurate enough to find the majority of test case bloat in my opinion.
Craig told me that the internet doesn't get to give me homework. I said that James Bach absolutely does get to give me homework any time he wants. He is the smartest tester I've ever talked to and he has done more for testing as a profession than the testing organizations combined have in my opinion. I care very much about what a test case is, and I hope to participate in a conversation about it with other testers I respect soon. Now, I have to get out of trouble though as Craig is threatening to shut off our internet connection if I don't get off the computer and get ready for our guests!
Clinically defined a test case is an input and an expected result.* For my purposes it does not matter if a test case is automated or manual so long as it is a planned test. For the purpose of reducing test case bloat, I’d go further and say that it is a test you plan to execute a minimum of once in the product life cycle.
*IEEE (1998). IEEE standard for software test documentation. New York: IEEE. ISBN 0-7381-1443-X.
Why would I say that? Did I think about it? What am I talking about with the "planned" stuff? Do I really think that to qualify as a test case that a test must be "planned" and what does "planned" even mean?
I want to summarize why I didn't go in to what a test case is in this paper. I didn't think it would help. My paper is a survival guide. If I set even ONE tester partly free from a useless legacy test case so that they can do testing of value again, it is worth this shoddy definition of a test case to me. My intention is to offer a paper that helps a real human testing in a company go back with a practical list they can apply to the legacy test suite in their test case database. I wrote this paper to help testers, and I hope at the same time I was careful enough to not harm testing as a profession by limiting the scope. If I was harmful with this definition I apologize. I didn't mean to be. I do think that most of the test case bloat is included in this narrow definition of test case. This is not how I define a test case most of the time, but it is how I look for test case bloat.
My paper has a purpose. The purpose is not aspirational nor is it a paper about theory. What I mean by that is this paper is intended to help a person testing at a company which has test case bloat. In the situation where you have scripted test cases piled up to deal with and it has become critical to address, my paper is intended to help a tester evaluate existing planned test cases in order build their own escape plan. Wouldn't it be better to prevent it from getting to that point? Yes, but it is too late for most testers in large companies already.
Think about that person responsible for legacy test cases which are scripted, in most cases with very low craftsmanship. How are they going to contribute to better software quality? All of their testing time is being eaten up by test cases of low value which are unlikely to find bugs. When I was in this situation I'd come to work in the morning, and have hundreds of failures reported from my automation to sift through. Most of the time they weren't even bugs. It was a total waste of my brain. Keep in mind that the testers who get these huge legacy regression suites assigned to them are treated as if it is no big deal. After all, "It's automated" so that means it should be free now, or at least low cost in terms of tester time, and no one wants to admit they have failed. A huge investment went into creating these fragile and low value automated tests. If you admit the amount of time it actually takes you are branded as a low performer with a bad attitude who isn't on board with "quality progress". I want to help those testers find away out so that they can go to lunch again and so that they can build a case to spend less time on these regression test suites and use their creativity to invent and run new tests which are more likely to find important bugs and improve the user experience.
So, do I believe that there SHOULD be huge legacy regression test suites that waste tester time, return little value, and are hard to maintain? I do not. I think that there are scripted and planned test cases of value, both automated and manual, which can be used together with other testing methods to create a balanced test plan in most situations. I think we have a long way to go towards making our test automation and scripted human executed test cases better, more maintainable, and most of all with verification logic that doesn't suck. The current way we are validating pass and fail results in test automation is usually limited and insufficient for returning major value for the time invested. If I like it or not, we individual contributor testers are facing a problem of test case bloat right now, so I wanted to bring up the topic and provide some helpful suggestions that I've tried in practice. Is my definition good enough? Certainly not, but it is accurate enough to find the majority of test case bloat in my opinion.
Craig told me that the internet doesn't get to give me homework. I said that James Bach absolutely does get to give me homework any time he wants. He is the smartest tester I've ever talked to and he has done more for testing as a profession than the testing organizations combined have in my opinion. I care very much about what a test case is, and I hope to participate in a conversation about it with other testers I respect soon. Now, I have to get out of trouble though as Craig is threatening to shut off our internet connection if I don't get off the computer and get ready for our guests!
Comments