Test Automation Planning and Dogma
I'm working on my automation plan today and found this article. I can't believe it is written in 1999 and the same mistakes are being made at most companies still. My blogging time is really short right now, but I'm hoping in a few weeks I'll be able to blog more.
I was told by someone that my dogmatic talking about human testing is dangerous and that most people don't agree with me. It feels sort of like the mafia to hear that sort of thing, although, I know the intention was friendly, just to let me know that I'm most likely making career limiting moves. My desire for balance in testing is unpopular? With whom? Certainly not with most people who test or use software that results from it.
I must however give thanks for the reminder that I'm fighting a losing battle, but I've never been a political or popularity motivated person. I'm the dangerous kind. The kind who does what they believe is right despite criticism. While I view this as being ethical, I think others view it as stubborn, naive, and delusional. I also realize that taking on all of the things you perceive as injustice is one really good way to ruin your life if you don't put limits on it. Activism is tiring, so you have to be quite selective.
I may not have enough power or influence to do anything, but I have something still more important to me than having a big impact. I respect my own actions, even if other people don't. That is worth more to me than my job. I wish that more people had the courage to speak dogmatically about testing if they felt that way. Holding back ideas because they are unpopular just makes me want to scream that the Emperor is naked. Hey you, yes you, Test Automation. Put some pants on!
I was told by someone that my dogmatic talking about human testing is dangerous and that most people don't agree with me. It feels sort of like the mafia to hear that sort of thing, although, I know the intention was friendly, just to let me know that I'm most likely making career limiting moves. My desire for balance in testing is unpopular? With whom? Certainly not with most people who test or use software that results from it.
I must however give thanks for the reminder that I'm fighting a losing battle, but I've never been a political or popularity motivated person. I'm the dangerous kind. The kind who does what they believe is right despite criticism. While I view this as being ethical, I think others view it as stubborn, naive, and delusional. I also realize that taking on all of the things you perceive as injustice is one really good way to ruin your life if you don't put limits on it. Activism is tiring, so you have to be quite selective.
I may not have enough power or influence to do anything, but I have something still more important to me than having a big impact. I respect my own actions, even if other people don't. That is worth more to me than my job. I wish that more people had the courage to speak dogmatically about testing if they felt that way. Holding back ideas because they are unpopular just makes me want to scream that the Emperor is naked. Hey you, yes you, Test Automation. Put some pants on!
I think the solution is to just keep doing what you're doing ... to address the "dogmatic" portion of their statement.
It's only dogmatic if you don't analyze it. You're collecting numbers for how much effort automation really costs, and you're collecting numbers on how much effort human-testing is costing. And you're collecting numbers on the number and types of bugs each type of testing discovers, and you probably already have (or can find) numbers of on the cost of fixing different types of bugs. As soon as you can point to your numbers with any meaningful data... you're NOT talking dogmatically about anything anymore. You'll be talking sense.
Reply to this
The dangerous kind testy? I like the dangerous kind.
Recap of the article. 7 is the magic number, guess why?
7 Steps to Test Automation Success
1) Improve the Testing Process => go overboard here, spell out every step that needs to be taken to execute the test. Keep the CPU's busy.
2) Define Requirements => Test design detailed test requirements. Detailed notes of every step in the process. Micro notes.
3) Prove the Concept => quick test suite that demonstrates on the right track.
4) Champion Product Testability => CLIs & APIs are easier to test than GUIs.Demand early testability as a product requirement.
5) Design for Sustainability =>
a) Performance: get more hardware and reduce redundant tests.
b) Ease of analysis: don't hide the complexity of the tests and thoroughly analyze failures.
Create a sustainable test suite:
a) Reviewability: Code Review and Test Review. Know your test base as you know yourself.
b) Maintainability: Build an abstraction barrier to minimize changes to the tests due to product interface changes. Implement an interface.
c) Integrity: Verify the correctness of the tests. Double check. Tests for the tests.
d) Independence: Tests can be run in isolation and unattended.
e) Repeatability: Make sure tests run exactly the same way every time. 100% Repro.
Test Automation Architectures that support these design goals(like this)
1) Libraries => create libraries of testing function that can be used in many projects. Well documented and well written tests.
2) Data Driven Tests => write a parser to interpret and execute test statements. DSL with a test parser.
3) Heuristics verification => verify the results. Look at specified portions of output and make reasonable comparisons. Develop useful heuristics(must be investigated? What are useful heuristics?)
6) Plan For Deployment => Tests need to be packaged so other people can use them. Document setup and make test suite easier to install and run. Test the test suite (sweet).Get the users to use your test suite. Training and documentation in order process.
7) Face the Challenges of Success => Train staff to learn to diagnose failures found by the automated test. Testers need to learn to Repro failures manually or with minimum scripted tests. Add new tests to improve the coverage and test new functionality. Maintain the tests like a baby and schedule a formal Test Review after each major release. Continuously raise the bar of your tests to detect new bugs.
Reply to this