What is a test?
Version 2.5: A test is an action which produces discoveries that can be used to evaluate product quality.
I'm discovering my limitations with writing. Unfortunately I am not talented with words in quite the way Matt Heusser is. I am trying really hard to learn and sometimes I just close my eyes and think "Learn faster, Lanette!" Anyhow, I hope I'll have another update and I'm still thinking about this.
A test is an action which produces data that can be used to evaluate product quality.
Why is a test an action? What about those tests that sit in your database? Do they not count? Of COURSE they don't count. They do not become a test until someone or something takes action. They are just a plan. Remember, TEST is a verb. Also, just because you want it to be so does not mean that software is going to test itself.
There was a big battle in my mind over produces. Does it create data? Or does it produce data? Or does it discover data that is already there? Well, the action of the test produces some sort of data I believe. It may not create it, but produces to me also means presents, so it presents it somewhere else. When I am testing I am hunting data to tell me something about the software under test. What about a case where you look at the launched product, notice a button missing and did not have any input or output, but still reported a value bug. To me, this is still a test because there was an action, there was learned information and there was evaluation for the purpose of evaluating product quality. I thought the word data made sense, but then I talked more with James Bach and Ben Simo and the word data was confusing a few other people, and it is very unclear what I mean to say by that term. That word didn't fit in the definition and it didn't convey what I meant either. A test produces more than data. It isn't only inputs and outputs and results that a test produces, but it is learned information. By "discoveries" I mean any information that we have now which we did not have prior to the action of performing the test. The information may have existed, but until I learned of it, it could not be used by me to evaluate product quality.
"can be used" is key. It can still be a test even if it isn't used properly. If I do a series of tests and the learned information is never evaluated, the test still happened because the data exists, but until the data is learned the test is not complete. This happens all of the time with automated tests. The tests happen and the data is never used to evaluate product quality because the resulting data isn't human readable easily and the time to do the translation from the language of "suckish" to human ran out. It still was a test because the data exists, but I would argue that you haven't completed the test yet because the evaluation has not happened. This is a test in progress.
"evaluate" is a tough one. This is where the craft and skill of testing comes in. Taking action and producing learned information is not very difficult. Taking the right action and producing useful data is far more complicated. However, even if you do the first two well, evaluating that data takes skill. I'm not talking about human vs machine here because that doesn't matter at all. Evaluate is all HUMAN all the time, and this is why test automation is deceptively difficult. The evaluation part of a test is when the available powers of observation are used to determine "Is there a bug here?" Answering that question takes into account all of the past experience of the evaluator including their higher level thinking skills, their observation ability, and even their mood at the time of evaluation. There is far too little scientific research done in this space, and I'd like to know what makes the best testers more productive in this space and what factors contribute to their best evaluation versus the times they are just distracted or "off". I can already see the wheels spinning in some executive minds, NO, we can not just automate tests to get around this problem because this data applies to each and every person writing the test automation as well. Until machines can write their own tests, observe, evaluate, and create as well as humans we will have this issue regardless of the method used to take the action of "test".
"produces learned information"-I do not mean learned information in terms of just data, but I mean anything and everything learned during the process of the action of the test. You might have learned that the software is untestable just by trying to launch it, or even sooner, you might have gone to find a build and it didn't exist! So in the process of a step that didn't even state the expectation that a build would exist you've learned something of great value which is relevant to the software quality.
I digress into a daydream here: I wish I knew that there was a team of brilliant developers working on innovative ways to evaluate data resulting from software tests. This to me represents hope for the future of test automation. Finding better ways to validate, collect, and parse the relevant data so that it is more useful for the humans. I think machines could one day test rather than just "checking", but until we program them with more power in the ability to "evaluate" we will continue to just create more automated tests that suck rather than better automated tests. I want to go to a conference next year and see something amazing in terms of how the automated test evaluates the data after the test action.
"product quality"-I include this only because I think this is the difference between a test and random vandalism. When I was about twelve I got curious and tore apart a floppy disk to see what was inside. If I did that with other people's property, I would go to jail. The difference between a test and an act of vandalism lies in the intent. If the purpose is the improve product quality, and I use product and not software because I think this applies to manufacturing tests as well, then the end intent justifies the loss of product. What is the difference between testing and hacking? It is in the intent, not in the action, especially with security testing.
Why do I think this is what a test is and why do I disagree with the IEEE and also with myself? Well, because I was wrong. And because I think they are wrong. The definition they give does not fit the scope of testing. It is far too narrow. I'd love to discuss why the IEEE is wrong and why was wrong before, and if I am wrong now, please, let's talk about it!
I'm discovering my limitations with writing. Unfortunately I am not talented with words in quite the way Matt Heusser is. I am trying really hard to learn and sometimes I just close my eyes and think "Learn faster, Lanette!" Anyhow, I hope I'll have another update and I'm still thinking about this.
A test is an action which produces data that can be used to evaluate product quality.
Why is a test an action? What about those tests that sit in your database? Do they not count? Of COURSE they don't count. They do not become a test until someone or something takes action. They are just a plan. Remember, TEST is a verb. Also, just because you want it to be so does not mean that software is going to test itself.
There was a big battle in my mind over produces. Does it create data? Or does it produce data? Or does it discover data that is already there? Well, the action of the test produces some sort of data I believe. It may not create it, but produces to me also means presents, so it presents it somewhere else. When I am testing I am hunting data to tell me something about the software under test. What about a case where you look at the launched product, notice a button missing and did not have any input or output, but still reported a value bug. To me, this is still a test because there was an action, there was learned information and there was evaluation for the purpose of evaluating product quality. I thought the word data made sense, but then I talked more with James Bach and Ben Simo and the word data was confusing a few other people, and it is very unclear what I mean to say by that term. That word didn't fit in the definition and it didn't convey what I meant either. A test produces more than data. It isn't only inputs and outputs and results that a test produces, but it is learned information. By "discoveries" I mean any information that we have now which we did not have prior to the action of performing the test. The information may have existed, but until I learned of it, it could not be used by me to evaluate product quality.
"can be used" is key. It can still be a test even if it isn't used properly. If I do a series of tests and the learned information is never evaluated, the test still happened because the data exists, but until the data is learned the test is not complete. This happens all of the time with automated tests. The tests happen and the data is never used to evaluate product quality because the resulting data isn't human readable easily and the time to do the translation from the language of "suckish" to human ran out. It still was a test because the data exists, but I would argue that you haven't completed the test yet because the evaluation has not happened. This is a test in progress.
"evaluate" is a tough one. This is where the craft and skill of testing comes in. Taking action and producing learned information is not very difficult. Taking the right action and producing useful data is far more complicated. However, even if you do the first two well, evaluating that data takes skill. I'm not talking about human vs machine here because that doesn't matter at all. Evaluate is all HUMAN all the time, and this is why test automation is deceptively difficult. The evaluation part of a test is when the available powers of observation are used to determine "Is there a bug here?" Answering that question takes into account all of the past experience of the evaluator including their higher level thinking skills, their observation ability, and even their mood at the time of evaluation. There is far too little scientific research done in this space, and I'd like to know what makes the best testers more productive in this space and what factors contribute to their best evaluation versus the times they are just distracted or "off". I can already see the wheels spinning in some executive minds, NO, we can not just automate tests to get around this problem because this data applies to each and every person writing the test automation as well. Until machines can write their own tests, observe, evaluate, and create as well as humans we will have this issue regardless of the method used to take the action of "test".
"produces learned information"-I do not mean learned information in terms of just data, but I mean anything and everything learned during the process of the action of the test. You might have learned that the software is untestable just by trying to launch it, or even sooner, you might have gone to find a build and it didn't exist! So in the process of a step that didn't even state the expectation that a build would exist you've learned something of great value which is relevant to the software quality.
I digress into a daydream here: I wish I knew that there was a team of brilliant developers working on innovative ways to evaluate data resulting from software tests. This to me represents hope for the future of test automation. Finding better ways to validate, collect, and parse the relevant data so that it is more useful for the humans. I think machines could one day test rather than just "checking", but until we program them with more power in the ability to "evaluate" we will continue to just create more automated tests that suck rather than better automated tests. I want to go to a conference next year and see something amazing in terms of how the automated test evaluates the data after the test action.
"product quality"-I include this only because I think this is the difference between a test and random vandalism. When I was about twelve I got curious and tore apart a floppy disk to see what was inside. If I did that with other people's property, I would go to jail. The difference between a test and an act of vandalism lies in the intent. If the purpose is the improve product quality, and I use product and not software because I think this applies to manufacturing tests as well, then the end intent justifies the loss of product. What is the difference between testing and hacking? It is in the intent, not in the action, especially with security testing.
Why do I think this is what a test is and why do I disagree with the IEEE and also with myself? Well, because I was wrong. And because I think they are wrong. The definition they give does not fit the scope of testing. It is far too narrow. I'd love to discuss why the IEEE is wrong and why was wrong before, and if I am wrong now, please, let's talk about it!
Posted by Testy Redhead at 15 Nov 2009 
Categories: uncategorized
Tags: learning test thinking testing
Categories: uncategorized
Tags: learning test thinking testing
Hi Lanette,
I feel vindicated in my assessment that you are an uncommonly talented testing thinker. This is exactly the sort of examination of the subject I was hoping to see when I challenged you (and BTW, an essay like this is technically called an "exploratory essay", much like exploratory testing-- google it).
It can be difficult to get under the skin of a topic like this, as you have discovered. But you're not alone, and that's one of the keys to getting through it. You prepare yourself through introspection (and any other research you like), then you can enter a dialog with colleagues to develop things further and generally get unstuck.
I use the word "transpection" (http://www.satisfice.com/blog/archives/62) to describe a dialog where I question someone else not to test them but to help me test myself. I did a transpection last night with Michael Bolton on this subject ("Is a test an input and an observation?"). I'd be happy to send you the transcript.
As to the content of your post, a few ideas:
- It can help straighten things out by keep utility separate from ontology. By that I mean that the fact that "data" exists is a completely different issue from whether that data is used. For the purposes of testing, I don't necessarily care about data that exists somewhere unless I know about it and benefit from it. A test that produces only data I don't know about is not (yet) a test in the practical sense.
- I agree that test is an active process. It's a verb. But that doesn't mean that have to do something to the product in order to test it. Imagine a clock in a case. Imagine that you can't approach or influence the clock in any way. Can you test it? Sure! You can watch it (that's the active part of gathering data) and compare it to another clock (that's the basis of our evaluation). Thus, you can test even without providing input, in the classic sense of "entering data."
- It seems to me that a test requires four things: configuration (it must be in a state ready to be tested, such as installed and ready to run), operation (it must do what it does, somehow, whether the tester controls it or not), observation (the tester must be able to see it, somehow), and evaluation (without evaluation it is a tour, not a test).
- As for "expectation", that seems a narrow concept to me. I prefer the term "oracle." An expectation known in advance of the test is one sort of oracle. An oracle I define as "a principle or mechanism by which we recognize a problem." You can make bugs visible, but if you have no oracle, you won't recognize them AS bugs. Even if I have no conscious expectations of a product I may have unconscious expectations that wake me up when a problem happens, or I may have a tool that lights up (like a magic elf sword) whenever the right bug comes around (orcs). Not exactly an expectation, but certainly an oracle.
Anyway, good job opening up your thought process for us. Thank you.
-- James
Reply to this
Lanette,
Yes, Test is a verb! Test is a verb! Test is a verb! Without evaluating, were not testing.
We testers can create test plan documents. We can partition & document our test ideas into test cases or charters, or whatever else we want to call them. We can even automate some testing tasks and let machines generate data. However, none of these things we call tests are testing. They are data.
As you rightly point out, a test requires a human evaluating something.
Maybe I'm just nitpicking vocabulary, but I don't think of testing as producing data. Sometimes it does. Sometimes it only produces more questions. Sometimes testing is performed only on existing data.
What testing -hopefully- produces is understanding -- either of new or pre-existing data.
I've witnessed many testers and their tools produce huge volumes of data but no understanding. In my opinion, those testing efforts are generally failures. That said, I find that the testing value of data is often not realized until long after it was produced.
Again, I am thrilled to see others thinking of test as a verb.
Cheers,
Ben
Reply to this
What you've written about regarding "evaluation" really speaks to me. Maybe this is why I haven't noticed many attempts to measure complexity of a test. "Evaluate" happens in so many different ways and is at the heart of software quality's "Turing test."
I'm very impressed with your writing here and the work you are doing with this particular entry. It is one example of why you are the best role model for perseverance I know.
Reply to this