Automation Often doesn't Accurately Represent the User
This weekend we used my Garmin Nuvi GPS to direct us to a friend's house we've never been to before and got some interesting results.
1. According to the GPS we were driving on water! As I don't own an amphibious vehicle, this data was false.
2. The humans told us why we shouldn't go the way the GPS had us go even though both paths technically would work.
Good things about the GPS
-It didn't seem to mind that we were driving on the water, so at least it didn't stop working or error out. Our car was on a ferry, so in a way, we were driving on water.
-The GPS efficiently got us from point A to point B with the fewest turns possible.
Factors the GPS Disregards
-Safety. The humans don't take that route because it is hard to see around the corner with the overgrown vegetation.
-Comfort. The GPS doesn't know which roads are bumpy or smooth or which have potholes.
-Police. One of the roads contains a speed trap often. Locals avoid this road.
I bring this up to explain why unit testing and a high percentage of code coverage is NOT enough testing alone. It's a great start. We most certainly should perform this testing and move quality upstream as much as possible, but when we neglect user research, user input, and devalue product expertise in the profession of testing, we are driving like a GPS and not like a human. We are shipping software with an unknown user experience which lowers overall quality.
Using the product is not enough. Testing the code is not enough. A brilliant combination of both can come together to make a good software testing strategy.
1. According to the GPS we were driving on water! As I don't own an amphibious vehicle, this data was false.
2. The humans told us why we shouldn't go the way the GPS had us go even though both paths technically would work.
Good things about the GPS
-It didn't seem to mind that we were driving on the water, so at least it didn't stop working or error out. Our car was on a ferry, so in a way, we were driving on water.
-The GPS efficiently got us from point A to point B with the fewest turns possible.
Factors the GPS Disregards
-Safety. The humans don't take that route because it is hard to see around the corner with the overgrown vegetation.
-Comfort. The GPS doesn't know which roads are bumpy or smooth or which have potholes.
-Police. One of the roads contains a speed trap often. Locals avoid this road.
I bring this up to explain why unit testing and a high percentage of code coverage is NOT enough testing alone. It's a great start. We most certainly should perform this testing and move quality upstream as much as possible, but when we neglect user research, user input, and devalue product expertise in the profession of testing, we are driving like a GPS and not like a human. We are shipping software with an unknown user experience which lowers overall quality.
Using the product is not enough. Testing the code is not enough. A brilliant combination of both can come together to make a good software testing strategy.
Automated testing has its place, and that place is making sure that nothing breaks as you add and change things (aka regression testing). Using it for anything else, such as usability testing, is just silly. (:
Reply to this
I totally agree that automation is useful, and I am working on putting more of it in place in some areas. There is no doubt in my mind that test automation is critical, but it isn't enough testing by itself. My concern is that in the move companies are making to automate everything, good user experiences are being lost and software quality is slipping, not raising overall.
Usability testing is interesting because I think it needs to be done by actual users trying to do something with the software, a good sample of the actual people using the software. It also needs to be done to some extent by test professionals who understand both how to use the software and how to isolate bugs to drive fixing them. I worry that testers now ARE developers, and that building better test software is of higher perceived value than testing talent/skill. The issue is, having a cheaper, better, more maintainable automation suite may save the company money, but it doesn't always directly have any customer impact. What happens to the "time saved"? Usually it goes towards releasing software sooner rather than to testing the scenarios that aren't easily automated so that the software is better for humans to use.
I would never argue that Automation isn't importnat in software testing. It is. My only point is that it isn't enough YET. It's a critical part of "this balanced breakfast", but not to be used as a substitute. Just passing the automation suite shouldn't equal "release the software".
Reply to this
I agree Automation testing has its place ,but now a day Test Driven Development TDD leads to quality software,but it is a necessity in our less than ideal world.
Reply to this
Does TDD lead to quality software without good human evaluation and usability data though? I haven't seen any data that proves this, although I've seen testable builds far earlier in the product cycle as a result of some test driven development when combined with scrum. I just wonder, long term, is the quality at the end better for the customer? I think it is not consistently better in every case because the ideas that go into the requirements vary, the talent of those making the software varies, and so it takes more than more test automation upfront to raise total software quality.
What happens after the features are complete? I think even when using scrum and TDD that some testing is needed beyond just certifying that the automation still runs in order to fill gaps and test those scenarios which are really hard to automate, such as the user experience and end to end customer workflows, and also to have someone thinking beyond the requirements in terms of stability, security, and performance before shipping a product.
Reply to this
Hi,
Good post......In testing many software test cycle has to be follow....automation testing can be do on different tools....
E-safety
Reply to this
Which tools do you find most effective? One difficulty I've found is that when the product is ready to be tested finally sometimes the automation breaks, and time that could be spent testing the product is instead spent on troubleshooting and fixing the automation, and often on finding out an error is an error with the testing tool and not the product being tested which is a waste of time. So far my favorite tools are for virtualization and automation of test setup.
Reply to this
Very well phrased and well explained post. I find that product knowledge is indispensable in software testing. With so many things that could be missed, it is too easy to increase the number of omissions by not understanding how people (humans) are actually using the product.
Reply to this