If you read just one thing on test automation before trying it,..(or trying it again after poor results)
Test Automation Snake Oil by James Bach in 1999. Of particular interest to me are the "Reckless Assumptions" and the Sensible Approach to Test Automation section. In fact, if you could only read 1/2 a page, the Sensible Approach section alone would save time and money.
I've read this article a few times. Sadly I then got feedback on my Automation Plan telling me to "put in (this) tool". I tried to explain why it doesn't help us or suit our needs, as well as why it wasn't realistic for our entire test team to become full Java developers in one month, no matter how intense the training is. Positive thinking is good, but in testing, realistic thinking is more important. Well, I try to balance it out with this link. This is the link I wish was in my test automation plan instead of the link to the automation tool which made me think seriously about becoming a waitress.
For those of you not familiar with test automation I'd like to also provide a few fundamental things that most people don't seem to understand who have never tried it before.
"Scripting" means that you are using a scripting language such as Javascript, VBscript, AppleScript, Perl, ect. Generally it does not need to be compiled and runs within a scripting environment of some sort. While it does work outside of one application, the scripting DOM is vastly different between applications. You can learn the core concepts the basic syntax and those transfer, but just because you know how to script in one context doesn't mean you are a scripting expert in another context.
"Automation Tool" is a very generic and broad term. Does the tool schedule and run automation? Does it allow you to test through the GUI? Does it allow you to write unit tests? Does it report back results? If so, how? Can you set it to a tolerance to avoid unneeded false negatives on cases? Does it just track your automation? How easy is it to use? My point is, don't say "Automation" and think that everything that falls into the category is the same.
"Scripting" is not equal to "Test Automation". If you are using scripting to run a step by step test case, even that isn't an automated test case unless you've created proper validation and reporting. Sure, you're scripting it to DO something, but if you didn't validate properly you didn't test anything. I've seen test teams with "80% Automation Coverage" but NO Test Automation. Don't just DO something, TEST something. They are different entirely. If you didn't validate, all you know is that the machine didn't explode when you executed that script. Any number of other things could go wrong, including it not working at all, and you still wouldn't know.
"Automated" does not mean that human tests no longer need to be run.
I took a risk management class last week and we followed the "cause chain" up to the highest level. The two causes for the biggest customer issues in the last release (which correlate directly to the highest support costs) both came down to one thing. A misguided but well intended attempt to save money. Now, I love a good bargain, but if it costs me more down the road, it really isn't a bargain. How popular do you think I would be if I announced, "Saving money isn't that important." I don't believe that overall, but in this context we need to have value, not just savings. So, I'd say that being stingy with test automation is one sure way to make it fail. The converse isn't true. If you spend lots of money on test automation, that in no way ensures that you will end up with something useful.
In my personal life I got a great example of this. After a horrifying jump in grocery and gas prices, I set myself a goal to reduce the monthly grocery budget by 20% or more in November. I'll let you know how that went. I think I exceeded my goal, but at a very HIGH cost. Because it was on sale, I purchased a $5 roast. Once I got home I started reading more about how to prepare the cut of meat I purchased. All I saw were warnings online saying, "Do not put this roast in the oven unless you like to eat shoe leather." It required a "Dutch Oven" which I'd never heard of outside of the very unpleasant pop culture reference involving blankets and gas, which seemed a terrible way to prepare roast, but considering I'd essentially purchased the rear end of a cow, I was getting discouraged. I ended up not cooking it. Then, along came someone to save the day. My dear friend who is an expert cook boiled it in a bizarre mixture of dark beer and coca cola for hours, and low and behold, it turned out great. How is this a lesson for me for test automation? Maybe there is someone with tons of experience and talent who can make my leather like meal work with hours of time and patience, but for the average person, it is cheaper to just buy a better cut of meat to get a good result. To top matters off, I won't be accused of pilfering my boyfriend's beer if I pay a few more dollars. If I consider the cost of boiling that roast for 6 hours in energy costs, that also may mean my savings were short term only.
Notes from article: Test Automation Snake Oil.
Case #1: No formalized hand off process. Documentation out of date. It's legacy the moment it's written. Build process broken??? Stay on the job until the product
is feature complete: executable documentation, 100% code coverage, comments, and a written documentation too.
Case #2: Got to take risks and whatever the client desires we will develop. He signs the checks and may his will be done to his utmost satisfaction, quality.
Case #3: Plan scalability into product design from the start.
Case #4: If the specs change all the time then the project will fall behind schedule. Output of system unreliable? Red Flag! Because of development efforts? Or changing requirements?
Case #5: Rudimentary testing? Says it all that failure is preprogrammed from the outset.
Most software projects fail? According to conference speaker at MS TechDays only 32% of software projects are on schedule, meet requirements and planed resources.
The devil is in the details is key. Approach each new project with a wary and skeptical mind. Automated tests execute a sequence of actions without human intervention.
Automation only capable of narrow, shallow test set. Manual testing is an interactive cognitive process and every evaluation must be explicitly planned. 80% of bugs
found manually despite years of investment in automation at Borland. Good to quantify the precise amount. Small companies do a poor job of testing their software, due to resource
constraints. So they will never be able to grow. Automation must be tested and reviewed on a regular basis. A controll for the controller. The bigger the test suite gets
the less likely anyone will know what the test suite actually tests. In agile development, features being developed iteratively manual testing is more effective. Quality
Software and Quality Test Automation. Status report mission critical to analyze how effective automation is. Advance The craft of testing. Think testing first( formalized
and explicitly documented, sequence per sequence), automation second. What am I doing here? What shall I do now? You tell me. I'm rereading this article again tomorrow and recommenting it. It's already 1:17 AM and I have a lot of tasks on my list. A waitress if automate? Did you watch the movie: "Million Dollar Baby" from Clint Eastwood? Keep in mind the quote: "Winners do the things, loosers don't want to do." Don't just DO something, TEST something.
Reply to this
Thorny: From the outside, software seems so simple, but the devil is in the details. The idea of Test automation boils down to this: computers are faster, cheaper and
more RELIABLE than humans; therefore automate everything. Highly repeatable testing; automated testing is stepping into somebody else's footprint thus minimizing the
chance of being blow up by a land mine and finding some really deep, interesting bugs. When selecting a test tool some factors to consider:
Capability => all critical features? Test result validation, test suite management?
RELIABILITY => work for long periods without failure, or full of bugs? Are the test tools under test and have 100% code coverage?
Capacity => Proof of working in a production environment? Test run for days involving thousands of scripts?
Learnability => Steep learning curve? Documentation, Books, Training available?
Operability => Ease of use? or cumbersome?
Performance => Execution speed, save time in development?
Compatibility => Interop? Tool work with our technology we need to test?
Non-Intrusiveness => Does the tool simulate an actual user? Software behave same with automation?
Red flag: if humans consider their testing activity to be a long list of mundane mental and tactile activities. Recheck and reformat the attitude!?!
Or else the tests will be hard coded to report success regardless of the problems in the product. Hot: in all cases automation and manual testing:
1 Test cases must be documented carefully.
2 Automation must be tested and documented.
3 Analyze the results after each test execution.
4 Changes in code base will result in new test code that must be written.
5 Communication, meetings and coordination op of test suite.
6 Pain of porting the test must be endured. No pain, no gain.
Just as there can be quality software, there can also be quality test automation. Testing first, automation second. I read that you are not content with your test automation
tool. Did you calculate the time you spent working on the cut of meat? Opportunity cost? What other activities you could have been pursuing instead? I always calculate
when doing something if it's worth the effort and it usually results that when I'm not in the editor increasing my knowledge base, everything else is a waste of time.
Reply to this
This
Reply to this