Trackbacks

• No trackbacks exist for this post.

Comments

• 5 Mar 2011 Jim Hazen wrote:
  Yes, automation is a tool only. How the tool is used is important. The human behind the tool is what is the driving force. Intelligent design and use of automation (be it for Regression, Performance, etc.) is what matters, automation has its boundaries and constraints.
  
  The problem has always been the hype of the tools and what they really can do and be used for. I lay blame on the tool vendor's sales people. They sell to the people with the money (management) and dazzle them with smoke & mirrors. When it gets to us, the people in the trenches, we are handed the tool and the misconception to go forth and "automate everything". When we don't deliver management gets pissed off. Someone else wrote a check with their mouths (salesperson) that our bodies cannot cash (us).
  
  After all, It's Automation Not Automagic! Both, automated and manual testing, have their place and role in the project. A realistic view and balanced approach is the best way to go as you said. That is the hard sell to the upper layers (Management and C-level) to get the understanding and buy-in on that concept.
  
  Automated 'Functional' Regression testing (which is that the vast majority of people do with GUI test tools) is only good for finding things that have been changed/broken in existing and perceived stable code/functionality. It will not find new defects in new code. You would need Watson and HAL to do that (maybe).
  
  Structured & Exploratory manual testing will never be replaced, it does take a human brain to do it. But automation as a tool to supplement the work, and free a tester up from the mundane repetitive nature of regression testing does have some value and benefit. So let's not be totally happy that Harry and James gave a Mea Culpa. They have probably always known this fact, but are now admitting it to the rest of us (and there are some of us veterans of the trenches who have known this for years).
  
  Our mission now is to take that message and get the 'other' people to understand it. We don't want to totally discount automation (at any level), but make sure it is seen as what it truly is... a tool and not the end all solution (silver bullet).
  
  Respectfully submitted,
  
  Jim Hazen
  Reply to this
  1. 6 Mar 2011 Lanette wrote:
     Hi Jim,
     
     I'm not trying to discount the efforts of automation. I just feel that in the last 5 years the balance has been off. When we have big companies literally throw away ALL testing except for automation on purpose, and engineering groups proud to have no testing at all, except for what they write into their own code, something has to change to swing us back into balance.
     
     The last 5 years those who specialize in automation have enjoyed the freedom of their value being assumed to be there without having to explain, prove, or otherwise justify the expense, while the human testing has been hunted down and nearly become extinct. To see any human testing flourish or any signs that the balance may return to a healthy level is my dearest dream. That in no way means that I'm flippant about the value of automation. I want to see high value testing overall, which to me means the best of human and machine working together for a common quality goal.
     
     The damage done by Microsoft and Google to the human tester can't be easily measured, and it saddens me to know that they think they have blazed new trails. They have created a market for snake oil salesmen to generate charts and false coverage metrics that are misleading at the cost of testing that has good impact to the user experience. I see the decision made years ago to get rid of all STE's and hire and keep only SDETs as the most misguided and harmful decision made in software testing history. I equate that choice as a human deciding between looking with their eyes and through radar instead deciding to blind themselves. They had the option to use both, but they selected temporary blindness instead. At least at Google they still hired contractors to do actual testing, even if coding on a whiteboard is required to ever be a direct employee. Anyhow, my point here is that we practitioners have a duty to share what testing actually works in the trenches. These large companies with their big dollar initiatives shouldn't get to divorce reality and results from what they are buying or building for the entire industry.
     
     Overall, I've never had more opportunity in my career. Those who have seen the results of an automation only strategy are now looking for human testers. They have woefully thinned the herd down to just a few stubborn gnarled up survivors who refused to quit. I mean by that, I was actually offered chances to leave software testing, and at the cost of a job opted out. I belong in testing. I'll prove it by taking the risk. I want to be here working on software quality problems. I want to work with great engineers and also tool designers to make something the returns great value. I don't want to harm the progress that automated testing is making. I only want to pull back the curtain and make the results viewable to all of us.
     
     Thanks,
     Lanette
     Reply to this
     1. 8 Mar 2011 Jim Hazen wrote:
        Ah... now I see said the blind man. I see the POV you are taking, and... Yep, I can totally agree with you.
        
        Too much of a good thing is not a good thing to do. And some companies have gone off the rails with the whole automation thing. MS & Google aren't the only ones who have done this, they are just the most visible.
        
        For as long as I've been working with automation & its tools I haven't ceased to be amazed at the things that can happen in relation to it. The pendulum did swing heavily to one side over the last few years, nice observation.
        
        Now the pendulum is swinging back. As you said there needs to be balance. There does need to be a human brain involved in the testing, both from a manual standpoint and an automated one. Because in the end the automation is code written by a human and it is only as smart as the person who wrote it. And that is another topic for debate.
        
        I respect the work that Harry Robinson and James Whittaker have done in this area. It has definitely given me new ideas and techniques to do my work, and even some of it (MBT) I'm still figuring out.
        
        Admittedly management has also been their own worst enemy on this one too. They keep looking for a silver bullet (and we are not hunting werewolves here) or the next best Chinese toy (if you know what I mean) that they can get. Unfortunately for those of us in the trenches we are the cannon fodder that get killed.
        
        So... yeah, I see where you are going now. Agree, we're cool.
        Reply to this
• 5 Mar 2011 Alan wrote:
  In a recent talk on test design, I talked about the notion of "useful tests", and defined useful tests as tests that provide *new* information. Regression tests generally provide new information only once (when they run and pass and you know the feature works) - after that, they only every supply new information if they fail. The solution is _not_ to stop writing automation entirely, but to write automation that results in tests being *useful* more often. MBT, data driven testing, randomization techniques, and random abstractions in the framework are all methods to combat the staleness of tests that we run forever despite the fact that they never give us anything new to work with. Note that writing automation like this is much more challenging that writing a test that does the exact same thing every time, but it's the direction that I think automated tests need to head in. I think throwing automation out entirely (which I know isn't exactly what you said) is definitely throwing the baby out with the bathwater.
  
  Oh - and although Harry is a wonderful proponent of MBT, I'm pretty sure he's not the "inventor" - but I'm sure he'd be flattered. If I remember correctly, I know that Beizer and Jorgensen both mentioned finite state machine testing as early as 1995, but not as a new idea, so mbt has probably been around at least that long. Now I'm intrigued to go discover the history - thanks for that :}
  Reply to this
  1. 6 Mar 2011 Lanette wrote:
     Hi Alan,
     
     I thought Harry won a patent in 2001 for Model Based Testing? At least his website shows that. That is why I believe he invented it, or at least significantly furthered our understanding of it. Let me know what you find out!
     
     I do not want to throw out all automation, in fact, I want to talk about how to use it and attach it more closely to human thinking to create a better cyborg tester. I want cloud repeatability of bugs so I can send a link and my exact steps happen for another person. I want to easily quantify performance and security issues. We are still at the infancy of understanding how automated and exploratory testing can work together. I don't even want to throw out regression automation. I just want the value of thoughtful human testing to be more widely understood by people outside of testing. I'm frustrated by the damage caused to talented human testers by misconceptions that people have.
     
     Overall, I have to say Thank You to the people who doubted my talent, dedication, or that I belonged in software testing. I was content sitting at my desk being a top bug finder for years. It is only by threatening me and hunting the people I care about to near extinction that I was willing to put myself out there, risk it all, and join the public conversation. So, it all started with the fine decision to fire all STEs. RK Johnston? Yep. He's partly responsible for me being at conferences. Thanks, Dude. It was pretty funny at StarEAST when I mentioned my impression that the SDET only decision was the most harmful in the history of software testing that the guy sitting next to me was part of the decision. Well, hey, at least it had an impact?
     
     Let me know if you find out why Harry has the patent for MBT if it isn't for inventing it. I think perhaps it is for his ideas on how to automate using it?
     
     Thanks,
     Lanette
     Reply to this
     1. 7 Mar 2011 harry robinson wrote:
        Hi Lanette,
        
        Nope, I didn't invent MBT; and Alan is right - I'm flattered.
        
        MBT techniques go back to at least 1976 ("ATLAS – An Automated Software Testing System", Jessop et al) and the FSM side got particular attention in Chow's 1978 paper, "Testing software design modeled by finite state machines".
        
        MBT concepts have been kicking around for a while:
        
        They are implicit in Beizer's 1990 book "Software Testing Techniques", and explicit in his 1995 book, "Black Box Testing".
        Jorgensen's "Craftsman" book is a great intro to MBT-type abstractions, though I don't believe he made specific reference to their use in test generation until his 3rd edition.
        And, of course, Bob Binder's tome, "Testing Object-Oriented Systems" should be required reading for everyone interested in MBT.
        
        My contributions have been in bringing MBT techniques into mainstream industry testing, especially as a low cost test generation approach. My patents deal with test sequence generation (7464372) and the user interface for the first MS MBT tool (6976246). (I'm actually not a big fan of patents on test technology, since it is so difficult to detect infringement.)
        
        Cheers,
        Harry
        Reply to this
• 5 Mar 2011 Jon Waite wrote:
  Jim,
  
  Your assessment is right-on. It's funny how many times I run into this with clients, but test automation is tool (and nothing BUT a tool). It doesn't replace testing, in fact it creates more of a need for it because automating mundane, repetitive tasks frees-up good testers to do more exploratory and boundary testing where meaningful bugs are caught. That's where the quality comes from.
  
  So I never take an approach that implenting automation will replace testers. The work involved with simply writing and maintaining test suites is time-intensive, and should be a full-time effort all on its own. Arguably it's necessary in some respects. But automation is by no means a complete testing solution (probaby less than 10% of the required overall testing effort). Managers who think it should comprise any more than that are selling themselves short and really compromising the quality of their product. They usually don't last long in their roles anyway.
  
  So I look at testing methodologies and process approaches to 'assure' quality in a product. This includes things like test modeling, requirements and gap analysis, etc. because I feel a lot of work that needs to be done should start WAY before the test pass.
  
  Anyway, test automation isn't something that will go-away and it shouldn't. It just needs to recognized for what it is, just a tool for testing what you already know, and that really only has a small role in the quality concept.
  
  I really appreciate your post and would love to meet you sometime!
  Reply to this
  1. 6 Mar 2011 Lanette wrote:
     Hi Jon,
     
     I totally agree. Very professional and well written summary by Jim. Also, I'm hoping to work towards a great overall balance of automated and human exploratory testing as part of a large quality strategy which includes prevention, pairing, and coverage at the code level as well.
     
     My expertise and interest is in the user experience, so you can see my bias for that in my blogs. I still believe in the value of other types of testing.
     Reply to this
• 6 Mar 2011 Alan wrote:
  Lanette - you have an interesting viewpoint on the SDET role (to say the least). I could give you a ton of more information on the reality behind your statements, but probably can't comment (or blog) about them withough pissing someone off (which bugs me because I hate non-transparency).
  
  Anyway - There's enough of an opionion on the subject among the community that I'm sort of thinking of an invite only "Myths of Microsoft" conf call just so I can give you and others the full story on the where, why, and what of SDETs at MS. I'll probably have to disguse my voice and route my IP address through the Ukraine, but it may be useful.
  
  Or you and I can just talk when we have a chance.
  
  The VLogs are interesting - but I have to say that I rarely catch up on my blog reading in a place where it's ok to listen to audio, so I have to remember to come back and listen when I have a chance. Other than that, I think it's a cool experiment.
  Reply to this
  1. 6 Mar 2011 Drew wrote:
     If there's enough info for a blog post of errata/additional material to HWTSaM, I'd love to be able to share that with people. It's one of my favorite books to hand out to new testers, especially SDETs. And then I explain to them that we're a startup and not Microsoft and we talk about what still applies for us. I was an SDET at Microsoft, myself, and I don't mean to knock anything/anyone when I say "still applies" - just a matter of scale.
     
     (Also, @Lanette - I lurk on your blog sometimes but I'm not much of a commenter. I really enjoy the perspective and I point people here.)
     Reply to this
• 8 Mar 2011 Lanette wrote:
  Hi Alan,
  
  The vlogs are just something I'm trying. I'd like to find a way to offer text transcripts of all of my vlogs and audio. I may also try making podcasts of all of my written blogs in the future so that people who prefer to talk can interact with me that way, even leave me audio comments. I'm thinking of ways to make my blog more interactive. I have a list of ideas. I"m hoping we can talk at WAT2, which I just need to negotiate with a few people for a couple of days off in order to secure.
  
  You might be surprised how much fact has gone into my opinion of SDET life at Microsoft. First, I was married for 5 years to a tester at Microsoft who became an SDET on the Windows team. Then, post Vista, we hired many ex-microsoft employees at Adobe. Many of the practices they put in place changed how I felt about testing at Adobe. It went from utopia, the place I loved above all else, with the best quality practices in the industry, to being much more shareholder focused. I worked with many SDETS.
  
  My criticisms are not towards SDETS themselves, but towards the leadership or lack thereof, or the decisions made that SDETS are of more value than testers. I disagree with that notion and I believe there is evidence that the SDET only approach is flawed. I do value the SDET role and think it is an important part of testing. I just have no idea why test leadership at Microsoft has been so dedicated to eradicating human testing and devaluing the role of product expertise in testing. It's almost like someone has a vendetta. I can't wrap my mind around it. It looks like an intentional plan to discredit all human testing to those of us outside. I can't imagine it really is some sort of conspiracy, but I don't get the actions. What I mean is, we hear good things that make sense, but the hiring actions and firing actions are clearly speaking louder to me and some others. When every STE is let go, it doesn't matter what you say. Actions speak so loud that words don't matter. I know that you have a point when you say the company is large and different teams do different things. However, every team company wide did the SDET only hiring, so that is why I see it is a Microsoft wide decision which had an impact on testing overall.
  
  Thanks,
  Lanette
  
  p.s. One of the things I adore about being a consultant is that I speak for myself. My own views. Not that of a company.
  Reply to this
• 8 Mar 2011 Ken wrote:
  Hey Lanette,
  
  Thanks for some partial credit for encouraging your public speaking. I hope to get back to a STAR conference in 2012 and hopefully see you present again. Bing's keeping me busy so I'm taking a year off from industry talks.
  
  As for credit as slayer of all STEs at Microsoft I can't really take credit for that one. Yes I was in the room but there were 20 or so other directors of test in the room that were debating the new title for the combined test discipline at Microsoft. One option was Software Quality Engineer but everyone thought SQE would turn into some joke about squeegees so that was dropped and we went with SDET. Okay I did then craft the emails and the roll out plan for the company wide announcement so in that sense I did have a major role.
  
  Yes, it was very funny when we first met (I thought it was Better Software and Agile Development) and you were just hammering away at the mindless obsession with test automation and not having great testers at Microsoft anymore. Funny thing is I'm working on a white paper to document our use of live site manual exploratory testing for Bing. Hope to have it out late summer.
  
  Odd how things eventually come around full circle.
  Reply to this

Leave a comment