Boundary Testing
I believe that boundary testing is in the calculator exercise in Testing Computer Software, which is also known as "The Bible" among software testers. Despite the fact that most professional testers know that we should be testing for boundaries, I'm finding that often, these tests aren't well performed, and even when we do perform the tests, the bugs aren't fixed! Here is a true story about several different bugs found in one day, just by testing ONE boundary.
A bug was found that when a certain number or more characters were entered, truncation did not happen. This error was handled, and an error dialog was brought up, alerting the user. Unfortunately, this application is expected to be run without a person watching (without a UI, or also known as headless in some circles). This meant that for all practical purposes, the application was frozen. A fix was put in place, but I was curious to test all of the boundaries. The team created an exploratory testing charter.
When the charter was run, we found MANY diverse behaviors depending on where we exceeded the expected characters.
1. In some cases, the application would crash. We wrote a bug on this straightforward issue with ease.
2. In other places, truncation happened, leaving unusable and incorrect email addresses and ages. These are easy to report, but difficult to know what is "expected" or what would be desirable in that case.
3. The most interesting issue we found was a silent failure. The entire entry would be skipped, and the log would report for example--Success! 10 entries imported. 9 Correct 0 Skipped 0 Errors. Subtle silent failure is the riskiest issue we found in my opinion, because if you are dealing with many enterprise solutions, you might have millions of imported pieces of data per day, and nothing to check that any are ignored, truncated, or missed.
A few things to think about in terms of boundaries. It isn't just how many characters, or what characters, or is "replace with nothing" or "null" allowed, or can you delete nothing that concerns me. The better question is, on what scale are these issues fixed? Will the headless system ever be fixed so that NO error dialogs will put it into a useless state again? It is easy to test for the error conditions that we handle. It is the error conditions that we don't handle and don't expect that we need to test for. This is why unless we have tested the boundaries and error handling ourselves, we should not assume that because it is basic that it is covered, or that because in one part of the application it is handled, that it is handled everywhere.
A bug was found that when a certain number or more characters were entered, truncation did not happen. This error was handled, and an error dialog was brought up, alerting the user. Unfortunately, this application is expected to be run without a person watching (without a UI, or also known as headless in some circles). This meant that for all practical purposes, the application was frozen. A fix was put in place, but I was curious to test all of the boundaries. The team created an exploratory testing charter.
When the charter was run, we found MANY diverse behaviors depending on where we exceeded the expected characters.
1. In some cases, the application would crash. We wrote a bug on this straightforward issue with ease.
2. In other places, truncation happened, leaving unusable and incorrect email addresses and ages. These are easy to report, but difficult to know what is "expected" or what would be desirable in that case.
3. The most interesting issue we found was a silent failure. The entire entry would be skipped, and the log would report for example--Success! 10 entries imported. 9 Correct 0 Skipped 0 Errors. Subtle silent failure is the riskiest issue we found in my opinion, because if you are dealing with many enterprise solutions, you might have millions of imported pieces of data per day, and nothing to check that any are ignored, truncated, or missed.
A few things to think about in terms of boundaries. It isn't just how many characters, or what characters, or is "replace with nothing" or "null" allowed, or can you delete nothing that concerns me. The better question is, on what scale are these issues fixed? Will the headless system ever be fixed so that NO error dialogs will put it into a useless state again? It is easy to test for the error conditions that we handle. It is the error conditions that we don't handle and don't expect that we need to test for. This is why unless we have tested the boundaries and error handling ourselves, we should not assume that because it is basic that it is covered, or that because in one part of the application it is handled, that it is handled everywhere.
Posted by Testy Redhead at 30 Apr 2011 
Categories: Troubleshooting, strategy, test strategy, Testing, test methodology, test cases, Test, Test methodology, testing, Software Testing
Categories: Troubleshooting, strategy, test strategy, Testing, test methodology, test cases, Test, Test methodology, testing, Software Testing
My favorite find was on a mobile app that I was testing. I discovered by fiddling with the phone settings that I could change the date of things executed by changing the date on the phone. The design was to use the system date from the server and wasn't even on the radar for testing. That's what we call a big Oops
Reply to this
Well noted, Lanette.
There's a point in here that I'd like to emphasize, though: boundary analysis is often discussed in terms of a very trivial and confirmatory approach: find out about a boundary by reading a specification or requirements document, and then try values of Boundary+1, Boundary, and Boundary-1. That occasionally finds some bugs. Yet real boundaries--differences in behaviour at some threshold of processing or analysis--often exist in places that we haven't been told about, and tests at the boundaries don't always reveal information in the places where one might expect to see it. Therefore, we must explore. That's the difference between boundary checking and boundary testing.
---Michael B.
Reply to this
Would one example of this be that some UK government websites will only accept Latin-1 but offer services to overseas citizens with foreign addresses?
Or that some airlines can't book people with the same name on the same booking?
Reply to this
I'm not quite sure what you're referring to when you say "that", Ric.
The attempt to use non-Latin1 characters and an attempt to book two people of the same name on the same flight for me fall under the category of domain testing. They're nice examples of domain techniques, but I wouldn't think of them as boundary tests as such.
---Michael B.
Reply to this
Thank you for sharing this interesting post. I am keen to know about Performance Testing Services and how exactly a QA Testing experts should go about performance testing assignment. Any Suggestion?
Reply to this
Thanks for sharing the great stuff.
Finding worthy blogs is not easy now and it's a pleasure to run across your site.
Reply to this