Snake Oil or Substance? Test Automation
It would be too simplistic to say that the software testing profession is changing. It HAS changed and is changing even more at a blazing pace. Some of what was foretold and we were warned of has started to happen. Companies like Toyota have seen quality slip due to software errors. Software companies like Adobe face huge threats to the PDF format because of security issues. Even President Obama had his twitter account hacked using password retrieval guessing and research to break in. What is the cause? Are things getting worse out there?
In a few ways they are. As we rely more and more on automation, some risks come up time and time again. The weaknesses in automation are more likely to become the weaknesses in shipped software. However, because automation has weaknesses does not mean that it is all snake oil. Some of it is snake oil and some of it is needed and valued medication, a cure for a specific ailment PROVEN to work. Some of it is instead just a cooking oil. When applied by a skilled cook it makes tests faster, better, more powerful, and adds to the cooking that was already happening. It doesn't select ingredients for you or teach you to cook, but it can enhance what you are already doing.
I've been practicing my scripting both at work and in my spare time because many of the jobs will require me to script on paper in an interview. I've NEVER scripted that way. It is an entirely new skill for me to learn.
This week on twitter, Shrini said something that shocked me. Part of what he said I disagreed with. He said, "Demonstration of coding - even for testers? As primary mode of evaluation? I wonder why your focus is "code" or "coding" ? how about testing without coding? Let me put it this way. Unless scripting is 100% of my job as tester, I will learn what is required minimum in 2-3 weeks. To me a tester is required to "test" that involves a varying degree of scripting effort. It can vary between 10% - 60%. Once I know the basics of a scripting language, I will develop skills as I go on, that is a secondary skill for tester, anyway. Unless you are an SDET (in Microsoft lingo) a tool developer for testers - you need to focus on doing testing not scripting." I understand and agree with some of this. Testing methodology is vital! I feel like I am very solid in my testing ability. I feel confident. I count myself among the many skilled, informed, and passionate software testers in the community who continue to learn and practice the craft of our profession. I am not the best tester in the entire world, but there is no one out there I'd be afraid to test with or talk to about my testing. I have talent for testing and believe I belong in software quality.
That is not true with scripting. My scripting is a new skill. I'm a novice. I've only just started sharing my code with other people. I stumbled upon scripting on accident because it was what I needed to do to add value. It hasn't been my main job. It wasn't much a part of my job until the last year. This wasn't a career strategy, or something I planned on. I simply found that on this project, in this context, scripts were very helpful and needed. As a tester who cares about providing value, I started doing more scripting because it was needed, not because I think it is the only way to test. I would NEVER say that scripting could be learned in 2-3 weeks. That is like saying you can learn all you need to know about testing by reading one book and taking a course! There are all levels of skill and scripting in most languages as a whole discipline and craft. Right now I can put together some very useful scripts to test with, but the more I learn the more they will be reusable, elegant, maintainable, and efficient. I have a deep respect for those who craft good code. I believe you could study your whole life and still learn new things to make your code better. Developers would be wise to respect how challenging the task of software testing is. It can be done in many ways, but when you see world class testing, you know the difference. In that same spirit, testers would be wise to respect how vastly complicated it is to write good code of any sort.
It doesn't matter if you feel that testers should not need to know how to code. In most cases, they DO now. If you want that to be the focus or not. I want to be the best tester I can be, and for some assignments, that means I do some scripting. I'm not out to become a developer. I am a tester first and I believe that is where I belong.
There is one other thing going on in our industry which is many of the skills that are being "required" aren't all needed for the job. There are so many applicants in this economy that "nice to have" has become "required" even if the boss isn't planning to use it right away. I'll be asked to do things in interviews that do not apply specifically to the job I'm interviewing for. I see no problem with that. Just as I would study for any test, I am trying to study for my interview. I don't expect someone who is just looking for all scripting to want to hire me. However, I'm hoping to continue learning scripting to support my primary function. Primarily I am a tester. As part of testing, I enjoy writing scripts to support my testing. I'm new at it, but learning fast. I can write some scripts that are useful to do some tests that wouldn't happen otherwise! That doesn't make me an expert script writer, but it does make me a more flexible and useful tester in my opinion.
Then I heard from Marlena Compton who added, "I used a *lot* of coding skills at my last job and expect to do the same for my new job. Do you want to work for someone who would expect you to spend more time thinking about scripting and less time testing?"
I replied, "Ideally, a team will want to collaborate more. So early on, more scripting. Later on more testing. I want variety. I don't want to work on crappy lame automation, but I'm glad to make scripts that scale and make sense for the project. I'm very annoyingly big picture in my thinking. I can't happily be a cog in the machine without seeing what is accomplished."
So, is test automation snake oil? It can be. It can also be really useful. It depends how you make it and use it. I now have one experience where I know for a fact the automation I made had a good impact on quality. I'd like to add to that experience.
In a few ways they are. As we rely more and more on automation, some risks come up time and time again. The weaknesses in automation are more likely to become the weaknesses in shipped software. However, because automation has weaknesses does not mean that it is all snake oil. Some of it is snake oil and some of it is needed and valued medication, a cure for a specific ailment PROVEN to work. Some of it is instead just a cooking oil. When applied by a skilled cook it makes tests faster, better, more powerful, and adds to the cooking that was already happening. It doesn't select ingredients for you or teach you to cook, but it can enhance what you are already doing.
I've been practicing my scripting both at work and in my spare time because many of the jobs will require me to script on paper in an interview. I've NEVER scripted that way. It is an entirely new skill for me to learn.
This week on twitter, Shrini said something that shocked me. Part of what he said I disagreed with. He said, "Demonstration of coding - even for testers? As primary mode of evaluation? I wonder why your focus is "code" or "coding" ? how about testing without coding? Let me put it this way. Unless scripting is 100% of my job as tester, I will learn what is required minimum in 2-3 weeks. To me a tester is required to "test" that involves a varying degree of scripting effort. It can vary between 10% - 60%. Once I know the basics of a scripting language, I will develop skills as I go on, that is a secondary skill for tester, anyway. Unless you are an SDET (in Microsoft lingo) a tool developer for testers - you need to focus on doing testing not scripting." I understand and agree with some of this. Testing methodology is vital! I feel like I am very solid in my testing ability. I feel confident. I count myself among the many skilled, informed, and passionate software testers in the community who continue to learn and practice the craft of our profession. I am not the best tester in the entire world, but there is no one out there I'd be afraid to test with or talk to about my testing. I have talent for testing and believe I belong in software quality.
That is not true with scripting. My scripting is a new skill. I'm a novice. I've only just started sharing my code with other people. I stumbled upon scripting on accident because it was what I needed to do to add value. It hasn't been my main job. It wasn't much a part of my job until the last year. This wasn't a career strategy, or something I planned on. I simply found that on this project, in this context, scripts were very helpful and needed. As a tester who cares about providing value, I started doing more scripting because it was needed, not because I think it is the only way to test. I would NEVER say that scripting could be learned in 2-3 weeks. That is like saying you can learn all you need to know about testing by reading one book and taking a course! There are all levels of skill and scripting in most languages as a whole discipline and craft. Right now I can put together some very useful scripts to test with, but the more I learn the more they will be reusable, elegant, maintainable, and efficient. I have a deep respect for those who craft good code. I believe you could study your whole life and still learn new things to make your code better. Developers would be wise to respect how challenging the task of software testing is. It can be done in many ways, but when you see world class testing, you know the difference. In that same spirit, testers would be wise to respect how vastly complicated it is to write good code of any sort.
It doesn't matter if you feel that testers should not need to know how to code. In most cases, they DO now. If you want that to be the focus or not. I want to be the best tester I can be, and for some assignments, that means I do some scripting. I'm not out to become a developer. I am a tester first and I believe that is where I belong.
There is one other thing going on in our industry which is many of the skills that are being "required" aren't all needed for the job. There are so many applicants in this economy that "nice to have" has become "required" even if the boss isn't planning to use it right away. I'll be asked to do things in interviews that do not apply specifically to the job I'm interviewing for. I see no problem with that. Just as I would study for any test, I am trying to study for my interview. I don't expect someone who is just looking for all scripting to want to hire me. However, I'm hoping to continue learning scripting to support my primary function. Primarily I am a tester. As part of testing, I enjoy writing scripts to support my testing. I'm new at it, but learning fast. I can write some scripts that are useful to do some tests that wouldn't happen otherwise! That doesn't make me an expert script writer, but it does make me a more flexible and useful tester in my opinion.
Then I heard from Marlena Compton who added, "I used a *lot* of coding skills at my last job and expect to do the same for my new job. Do you want to work for someone who would expect you to spend more time thinking about scripting and less time testing?"
I replied, "Ideally, a team will want to collaborate more. So early on, more scripting. Later on more testing. I want variety. I don't want to work on crappy lame automation, but I'm glad to make scripts that scale and make sense for the project. I'm very annoyingly big picture in my thinking. I can't happily be a cog in the machine without seeing what is accomplished."
So, is test automation snake oil? It can be. It can also be really useful. It depends how you make it and use it. I now have one experience where I know for a fact the automation I made had a good impact on quality. I'd like to add to that experience.
Posted by Testy Redhead at 26 Mar 2010 
Categories: uncategorized
Tags: respect checking scripting automation testing methodology
Categories: uncategorized
Tags: respect checking scripting automation testing methodology
If I were interviewing you, the fact that you have spent time teaching yourself a new skill such as scripting means lots more to me than the fact that you can do scripting. It tells me you have the right attitude to fit in well with our team. It tells me you have the mindset we are looking for - ready to jump in on any task to help the team.
Reply to this
Thank you Lisa! It may crack you up, but I'm still running all of my scripts from a folder named "post". Initially I was going to post some test builds to our staging database and that was it. However, I had to read through a few scripts to write the directions. As I was doing that, I noticed some other python scripts were available, and I took some bits from those and made my first script!
Now we can prep all of our builds and I ended up through pairing making code to simulate shipping the entire CS5 Suites in advance to improve test coverage and performance. All of that from 6 months ago poking around to post a test build. I am SO thankful to my team for supporting me in learning and to the dev who paired with me, and to the people who encouraged me when I got stuck. Those 2 weeks when my biggest script became such a mess that I had to start over was the point when I almost quit.
In 6 weeks, our tests builds went from a pretty low pass rate to 100% passing. That's how committed the team was to making the product better.
It seems silly that in a few months I'm out of a job, but I'm just thankful to the devs who supported my learning, reviewed my code, and started even running it themselves.
I'm not really just looking for a job. I'm looking for my team. I love having a team. It makes the job.
Reply to this
I remember a few months ago you were tweeting/writing about "finishing well." In this case, it looks like taking your job to its conclusion is helping you pick up some newer skills as well. Bravo!
Reply to this
Thanks Marlena! I still have until June, but I'm encouraged.
Reply to this
As someone who (now) makes a living from automation, here is my current take on things.
Automation can be used in one of two ways*
1) Automate Checks
2) Facilitate Testing
From watching you learn Python, I believe you are doing some of the first but primarily the second (but reserve the right to be wrong).
The snake-oil aspect sneaks in when you do not distinguish the two and sell your automation as doing both and use claims like:
- speeds up your testing by x %
- with automation you don't need manual testing
- or testers
- etc.
I strongly think that automation is a key part of a modern testing process. But it is only a part, is not the silver bullet of 'quality', and should be used only when appropriate.
Similarly, I think that all testers need to learn to program [a scripting language] in order to do their own test facilitation.
* Hat tip to Michael Bolton for pointing out the distinction
Reply to this
I consider myself someone who is still learning Python. I don't feel like I can put a past tense on it. I'm so excited, proud, and encouraged that my first attempt has been a good experience that helped the quality of the project and my team. I find myself now in a strange situation though. Will you be at StarEast? I'd love to talk in person of you have time.
Basically, I'm very new. My experience with item 1 on your list hasn't been very positive. I've automated checks, but the results weren't very maintainable.
One side advantage of learning programming (even just starting the journey, which is where I'm at) has been gaining more understanding of what my developers are facing. I've uncovered some subtle bugs by reading the code that I wouldn't have been aware of before.
I have so much respect for you and I just wanted to thank you again for writing the scripting for testers tutorial and for being encouraging.
The silver bullet of quality in my opinion is action. There is this idea that you can be so darn clever that no work has to happen. I think that we need a clever logical plan, and then to DO the work (take action). Then we need to keep doing the work (If that is making scripts, exploring, testing performance, and YES even retesting). Designing a good test is the same if it is a script, using a tool, or human executed.
Reply to this
Regarding the need for “demonstration of coding” for an automation tester; the issue with not having an understanding of the design and implementation of software is that automation scripts will end up being fragile. You mention this in your response to Adam Goucher; you’re results “weren’t very maintainable”.
It takes thought to design an approach for automation that allows your scripts to be resistant to change. Not every automation tester needs to have this understanding but the person who is defining the approach an organisation takes with its automation tests has to.
If everyone following that person adheres to the standard then the scripts will be as maintainable as designed.
Failing to design your automation approach properly will result in tests that break under change or can only executed or maintained by the person who wrote it in the first place.
If you’re interested in what is needed for a robust automation approach have a read of this article where I detail what is required. http://distributedlife.com/blog/2010/03/an-approach-to-software-testing-automation.html
Any automation effort you take is beneficial for you; learning SQL so you can find data more easily is automation; it’s the ‘facilitates testing’ part of automation.
Reply to this
How many versions of the software under test have your implemented automation solutions been used in?
If everyone following that person adheres to the standard then the scripts will be as maintainable as designed.
That is not my experience. I believe this is counter to human nature and not likely to work well in practice in most cases. This is an if condition that is unlikely to be met over time.
I've seen this plan touted over and over again. I've seen time, effort, and countless dollars poured in to making this happen, even punishing anyone who deviates from the standards and rewarding anyone who mindlessly complies. In each case the large automated test suites diminish in value and become more costly to run and maintain as time goes on. In fact, I'll step up and say that I believe that an automated test that has value for more than three years is a rare exception at most companies. This is the snake oil part. Pretending that we automate once and BOOM-now we don't have to invest anymore because those cases are going to work forever and have value long term. This is not only bad for ROI, it is bad for the entire culture of the test team. Instead of keeping creative thinkers, and people with testing talent, we keep compliant code monkeys. It is possible to use tools, use code, and still be a thinking tester. I see many respected colleagues do this well.
Regarding the need for “demonstration of coding” for an automation tester; the issue with not having an understanding of the design and implementation of software is that automation scripts will end up being fragile. You mention this in your response to Adam Goucher; you’re results “weren’t very maintainable”.
In the many pushes for automation over the years. I've seen the entire range of schemes. The least successful in terms of being maintainable has been UI pixel based automation. The most successful has been when rather than a large enforced command and control process, people on a team pair together to make something custom with low overhead for the team.
Any automation effort you take is beneficial for you
I totally disagree with this. Thoughtful automation that improves quality long term is the kind I'm interested in. The test coding that in my experience is useful is what I'd like to expand on rather than just going with what is most popular.
I'm most excited about working on smaller scale automation which is customized to solve a specific testing problems.
Reply to this
The current setup that I am working on involves approx 10,000 tests across 40 services that are in stable in production now. During development they changed, and I mean at least half of them changed, every week. The change may be been an internal change (didn’t break existing tests) or an interface change which would have broken any test that consumed the interface; with the setup I described we are able to keep up, to some extent with the changes as they occurred.
I will point at out that these are not UI components and I find non-UI components are easier to automate; I will also point out that you never manually test a service.
"If everyone following that person adheres to the standard then the scripts will be as maintainable as designed.
That is not my experience. I believe this is counter to human nature and not likely to work well in practice in most cases. This is an if condition that is unlikely to be met over time."
This ‘if’ condition is exactly my point. If you have someone who designs the automation approach so that tests, fixtures and pipeline code have been isolated you can be more accepting of change. If you don’t have someone who does that then you will have bad experiences; I have had plenty of them too. I needed to take a software development approach to our automation strategy before it finally worked. Initially I set it up where way we had a few thousand tests that didn’t like being changed every week. Or more accurately we couldn’t handle changing them every week.
It’s not the outright enforcement of the standard that achieves success; it’s the quality of the standard that makes enforcement easy; your testers can see that their scripts are more resilient to change and therefore they don’t write scripts that are directly dependent on interfaces and implementations.
The next point I’ll make is that automation is test execution; not test design. You have to have the tests to execute and if you just have ‘compliant code monkeys’ then you are not going to have people to design good tests in the first place. Automation isn’t a solution that means you will never have to test again. If you read my article on approaching automation you will see I detail the types of changes that will require new tests and those that require the changing of existing tests. We know there is change; we just need to ensure we limit the impact of change.
If an automated test isn’t valuable in three years time it means the application has changed. That is good. I don’t design my tests to last for x years. I design to last for as long as they test the criteria and then we delete or change the test to follow the change. This isn’t a cost if it only takes 5 minutes to write a new test and can be run every night against the code until it changes in three years time.
...damn I hit the comment limit.
Reply to this
I’m also not saying that you must apply automation to your work otherwise you’re will be left behind in the world of testing. Automation testing is only one small part of a big world of testing and I think some types of automation are prematurely done within organisations. In my experience UI automation should only be done after the UI has been bedded down and provides its biggest benefit with regression suites.
“Any automation effort you take is beneficial for you”
This is just a misquote. I went on to say that learning SQL to help you find data is automation. If you do something in an automated way such as writing small scale automation which is customised to solve specific testing problems then you are doing automation and that is beneficial for you. Automation is a tool in the shed but it is not the only tool we have.
Personally, if you want to write an automated test that is resilient to change look at your current automation effort and work out what has to change for each type of change you can foresee (interface, implementation, behaviour, etc). Then abstract that code out so that your tests don’t use it directly, they use the conceptual abstraction. With this setup when the concept changes you only have to make one change to keep your tests inline.
Reply to this
Hi Ryan,
That is useful and detailed. You have a solid understanding of the automation that works for your software. I did read your article (all of it) before responding to your comment.
Here's the thing. For someone as smart as you, why are you using such an instructional tone? It makes it hard to listen and interact. I don't want to argue with you. In fact, I agree with everything you've said here. Yet still I felt like your tone was dismissive, condescending, and unfriendly. That is a total waste because you obviously care about testing or you wouldn't write an article to share your findings.
I'm also defensive because I have regression automation whiplash. I've seen so many bad attempts at huge regression automation test suites which harmed product quality, cost many jobs, and failed to deliver what was promised. Basically, I love software. Even if I leave the industry, I'll still be a fan and enthusiast, using software on a daily basis. My distaste for bad automation isn't personal. It is about the software. On that level, I begrudge the failed automation which has harmed quality. I believe in automation, when it is done well and used in balance with testing for the user experience.
Automation that is a disservice to software users is upsetting to me. Most upsetting to me is what companies have traded for ill planned, failed automation attempts. It doesn't sound like you are making any of that or advocating for it.
Feel free to ignore this feedback if it isn't useful for you, but for me it would be great if you shared your experience just for what it is. What is working really well for you and also explaining the context around it. When I went to see Harry Robinson, he showed me how his automation is working. He is insanely innovative. He's proud of his work. Even he doesn't say things along the lines of, "You mention this in your response to Adam Goucher; you’re results “weren’t very maintainable”."
I had no power to make those tests maintainable. They were doomed from the start. It wasn't my coding that was the root cause of failure, although I'm sure to get myself into some situations where it IS in the future as I learn. I wish the only thing wrong with some of those attempts was my coding, because then they would have worked. This topic is taboo, and failures are swept under the rug.
Thanks for taking the time to explain what you meant in more detail. For now, I'm interested in doing more scripting rather than automation on a grand scale, but that is because it aligns better with my skill and experience. I am glad there are involved people skilled at larger scale automation working on innovating in that space as well, even though it isn't my area of interest. I'm also not a ballerina, but I admire good dancers.
Lanette
Reply to this
Hi Lanette,
I apologise for my tone. I was hoping to provide some guidance, rather than instruction and I didn’t intend it to be a lecture nor did I intend to disregard your experiences as not being valid. I felt that you or your readers could have used some guidance but I realise now that you were not necessarily asking for guidance and nor do you necessarily need it. It was presumptuous of me to think that.
I’ll happily take on board your feedback and the next time we converse I’ll be clearer, friendlier and hopefully it’ll be more of a discussion.
Ryan
Reply to this
Hi Ryan,
This is meant to be a discussion, not a speech on either side. I'm hoping we can learn from each other. I do have a great deal to learn. I have experience with automation, but so far the only kind I've enjoyed has been scripting. Once I'm better with scripts I might give some larger automation a try. I'm most interested in tools for web testing, like what Adam is working on, and I feel like stronger scripting skills will be a good match with those tools.
Please feel free to share your experience and I'll try to assume good intent and not be overly sensitive. It is great to hear of changes made that are successful at other companies. The test community is shrinking, but at the same time, more of us are sharing ideas online. Thanks for being so cool about discussing the non-technical stuff too.
Thanks,
Lanette
Reply to this
It is very correct that the profession of software testing is not the same as it was used to be now. There is a big difference if you look closely. Earlier, nothing much bothered but now things are changing at a rapid pace such that we didn’t even get the time to have a closer look into it. Even though it may sound flawless that these things are supposed to happen, it was our responsibility to have an eye on it! Well, Snake oil really got the better of us!
Mathews
CAAS
Reply to this
Thanks for your post, and I appreciate your honesty.
I find it odd that the direction of many software development organizations is to "shift" away from “good” test design and development skill set to a more "technical" or "development" centric approach to testing, which is, a developer or those who know how to code somehow make for better testers or more "qualified" testers.
I can't disagree more with this shift, seeing that I've spent 18 years automating software, from the early days using Virtual User under MacApp to more evolved frameworks like keyword test driven solutions or BDD solutions like cucumber and jBehave - both of which I remind those around me are designed for testers who may lack seasoned programming skills.
My experience has shown me that just learning scripting for a tester moving over to automation is just inadequate, that is, I believe it takes a person who has not or never programmed 12-18 months to learn Python or Java – this is to just become "proficient" - which to me is a law of diminishing returns. I work to help testers utilize automation frameworks that shift away the focus from technology to “good” test design and allowing "all" members of a qa team to participate. Frankly, a framework is not a framework unless is easily understood and adoptable by all, otherwise it's not a framework but a Frankenstein. I hear the word all the time and it pains me to see how mis-used it is.
So, I find it sad that this shift is happening, since i believe if a tester who doesn't program is given the right method to design tests that just so happened to be automated (aka keywords) then they can apply all the creative mechanisms they want to the approach. Recently, I heard Cem Kaner talk about automating exploratory tests, which I found quite interesting - since my focus these last 10 years has been helping companies achieve high volume automation with good proper test case design. I like learning how one applies different creative testing techniques and approaches to anything around automation.
BTW, Keywords are nothing more than an abstraction that allows for efficient and maintainable automation execution, but it will never replace the skill of the tester who uses those keywords to construct their tests.
I have to say, when I work with companies, I spend far less time working with them on automation technology implementation (coding) and MUCH more time teaching them the lost art of software testing, sad by true.
Reply to this