Knowing for Certain
"That is the only way we can know for certain," insisted a wise co-worker after he requested detailed analysis on how the application handled a missing reference.
Do we have to know for sure? What will we know for sure? If we ask the developer, we may know for sure their opinion and intent in creating the code. Not what will actually happen, but what they anticipate will happen.
I explained that what I like to do is perform a test. To force the missing reference, and test what will happen. This way what we know for certain is the behavior of the application under test in this environment, which means we know something more important than what the developer intended. We know the impact and the result of what actually happens.
I'm not discounting the importance of understanding the code, but I am saying that asking other people is of far less value than performing a test. If you aren't sure HOW to test or force that condition, by all means, ask. But an answer is not the same as a test. A test result is something that may be influenced by fewer assumptions than the opinion of a developer on how their code works.
Do we have to know for sure? What will we know for sure? If we ask the developer, we may know for sure their opinion and intent in creating the code. Not what will actually happen, but what they anticipate will happen.
I explained that what I like to do is perform a test. To force the missing reference, and test what will happen. This way what we know for certain is the behavior of the application under test in this environment, which means we know something more important than what the developer intended. We know the impact and the result of what actually happens.
I'm not discounting the importance of understanding the code, but I am saying that asking other people is of far less value than performing a test. If you aren't sure HOW to test or force that condition, by all means, ask. But an answer is not the same as a test. A test result is something that may be influenced by fewer assumptions than the opinion of a developer on how their code works.
Great post. More developers should know how to write unit tests to fix bugs using this same logic.
Reply to this
I guess this is the weird thing about being a tester first. People who code first often want to dive in, get in depth about all of the details on what it could mean if a reference is missing in the data.
As a tester first, I don't see the point. We run the test first to see if it is even important. If it doesn't matter to the end user, and it doesn't matter to the application, I'd rather force the error to prove it isn't important to move on to something that is.
Reply to this
I acknowledge that I don't know the context in that this discussion has taken place. I'd like to ask though if the approach is actually important? Isn't the information that there's something wrong (whatever wrong means in this context) in the system that needs to be fixed either way?
Just thinking about that question, if the missing reference is outside your control, for example in loading bad data that can't be prevented for one reason or other, how would that affect my thinking about it?
When testing, are you considering all states that the system could be in when a missing reference pops up? Can you consider all possible scenarios with your system or is it too complex?
If it's the latter I'd argue that best approach might actually be to do both, static analysis of the code AND testing.
I also realised that I'm only assuming that you mean static analysis as "...detailed analysis on how the application handled a missing reference." could be the outcome of an ET session as well.
Time to finish my tea before I get my thoughts in a knot here.
Reply to this
See. That's my point. Is the approach important? We don't know until we test.
I don't know. The test isn't done.
My point is, if we ask any number of people, the whole discussion, all of the time that went into the research, all of the human hours may add up to waste. We'll never know until we do the test. In this case the analysis is the informed opinion of developers on an anomaly in the data that is loaded right now in the application.
In this case we can look at the data and although we don't control it, we know there IS a missing reference in the data that was loaded. All we have to do is use it.
Reply to this