Selenium Meetup West Coast Style
Last week I was in Mountain View and very much looking forward to SFSE! I'm happy to report that it was packed, and if you missed it, before reading my take on it, you may want to check it out yourself without my biased point of view interfering at http://saucelabs.com/blog/index.php/2012/01/sfse-meetup-video-keeping-selenium-tests-100-blue/
I'm interested in learning more about selenium, well, to be specific, I'm interested in using Web Driver with Python and Ruby to automate tests where it makes sense and basically expand my capabilities in web testing. I believe automated testing is very important, and automated checks should be created, maintained, and relied upon as one aspect of quality by the development team. The whole team. I mean that when you ask who is responsible for quality the whole team says, "We are." When asked, "Who fixes the automated tests when they break?" I also want a clear, "We do and we don't develop new features until they are fixed." Anything less means your automation really isn't working to the level that your team can depend on it, so it doesn't have a bright future at this point, or should be rated "needs improvement" by the team. I've seen great test automation only twice. Both times the team owned it, relied on it, and would literally fix it right then. No shutting off tests. If it isn't sacrilege to turn off broken tests that used to work, the automation just isn't reliable enough to be counted on by your team.
So, with all of this in mind, let me share first the happy. So many things to love about San Fran and Mozilla. The patio over there is WOW. Mozilla's got some bucks. That's all I'm saying! I almost felt bad about drinking their soda for a minute, but I think maybe they can afford even my Diet Coke for a night.
Also, a kind fellow took this shot for me! Volunteered even. That makes for a happy tourista. No, my hair hasn't turned black, it's was just a bit dark outside at this point.
Lest you think I was cranky, I had a blast that day. I've been waiting for a huge merge, and my devs did testing together! Yes! They fixed bugs and swarmed for the first time. This team had NO testers, and now they are testing their own code and I feel so proud of them I could burst because they are showing just what a team can accomplish together, and they care about testing their own work. Then I spent the day finding issues, and I even handed off a feature I coded, so I'm learning and teaching on the job which basically means that my self-esteem is at an all time high. I'm seeing a great friend who I enjoy, the sun is shining. I had a new type of food, even red wine the night before. I've basically been drinking free cool Diet Cokes between lovely sun breaks, and despite being pale I'm not sunburned at all since the sun is so mild in the Bay Area this time of year. I'm expecting to learn more automation and even see some developer testing, which I believe is essential for delivering quality software.
During setup, the meetup hosts had some problems, and I was able to suggest ways to use the seating already there so everyone could see. I felt helpful, competent, happy, and above all, like a part of this group excited to see what new things companies are doing with Selenium. It doesn't matter to me where the testing is happening. I am a fan of all testing. I believe that different companies and contexts can tolerate different amounts of risk, so I don't assume that every single company has a use for a test team.
Now the Sad Part
When the talk started, I had an open mind. I am learning more about presenting for developers and mixed tester/developer crowds. Personally, I felt that the slides weren't attractive and that they looked a bit "retro" like a 1995 slide-deck, but I said that before I was upset at all. Honestly, it's a great point that the content is all that really matters. The speaking was good and the story did flow. People were extremely engaged. At least you can tell from this blog, that despite the lack of beautiful fonts or lovely photography in the slides, that the presentation did invoke some emotion in me.
Large crowd had about 100 people. All polite and friendly.
By the time the speaker was done, I felt totally deflated. The presenter said things to me that felt derogatory to testers and dismissive to testing. I felt this way specifically because of the following statements, which lI'll try to be careful to quote and please verify via the recording to get the tone used.
1. We run 90 minutes of tests so we KNOW that everything is safe.
2. We do not have a QA team and we never want one.
3. It is ALWAYS best to automate every test. (this was said despite admitting vast difficulty with automating some tests)
It isn't only that the speaker said these things, but that many others in the room were nodding with such sycophantic vigor that their heads nearly fell off. As I looked around I envisioned the sinister large framed hipster glasses hiding evil eyes identifying me as a tester, and I felt like I wasn't welcome. The inexperience reeked to me, and I felt as if I was the only person who noticed that the demos AREN'T on a real website.
Then--want to know the secret? How they keep the tests all BLUE? It's this secret we've called resetting the baseline in testing for the past 15 years. They aren't even testing the actual THING. They are just notifying themselves of what has changed. It isn't new. If they had testers they might have found that solution sooner. It's a great way to resolve the false positives that are always a part of the normal complexity when doing automated checks. It is sad to me that developers who are inexperienced at testing are so certain that they have all of the answers that they don't even attempt to get any training in testing from those of us who have been doing testing for many years.
I find these slides retro, and note that code review is for people who don't like each other, pairing is for folks who do, but as a tester, especially as a tester who shines in the areas of integration, collaboration, usability, and finding requirements gaps, my opinion was irrelevant in this case. Perhaps a title change from Test Lead to Overlord of Automated Checks (90 minutes to total perfection code confidence) is in order. Maybe some provisos can be added such as "So long as they happen in Chrome, we thought to validate them, they weren't an issue that needed human judgement, and they were in the subset of important tests that we picked, but other than that we are 100% confident."
I feel pretty awful as a public speaker saying anything negative about any person, and especially a fellow member of the tiny group of females in technology who speak, and I want to be clear that this isn't personal. Not only were the speaking skills very good, but her enthusiasm for automation was infectious in a good way. If I've misunderstood, or misquoted, I apologize in advance and will make edits. I do believe that this is a common view in the Bay area, and just a team approach that was being candidly shared for the benefit of other people. I do appreciate hearing the plain truth of the work that is done by the doers and I know that having a good set of automated tests is hard work that takes technical skill and persistence. It isn't just what was said, but the feeling of such enthusiastic agreement and unquestioning that really emotionally resonated with me. The speaker was an experienced Automated Test Creator also speaking on team philosophy. While she's not likely up for Testing Advocate of the Year award(if such a thing existed), but she may be nominated with a few others for "Most Divisive Talk 2012". I've got many more talks and conferences to attend, so this may not even make the top 5 before the year is out, but I sure hope the "Let's trash all testing done by humans for any reason" trend can make the rounds in the Bay area and the rest of the country can keep doing good teamwork, like the agile teams have been lately. I'm seeing progress in developers and testers working together. I'm seeing good classes in mobile testing. From what I understand, this dismissal of the value of all human testing isn't what is happening in Europe. I think this is common in the start-up community and also in the web space specifically where there is venture capitol. It's less prevalent elsewhere from what I can tell so far.
It isn't that they don't have or want testers that bothers me. Why do you hate testing? Without understanding what you are dismissing why don't developers even ASK for our help to learn testing? I promise, we want you to test. Our numbers are shrinking. I never want to visually compare data again. Nearly all of the testing I do is tool assisted. There is lots of testing going on outside of the bay area. If you look, even in the space of the web, the secret shame of even the large companies is the human testing is STILL being done because it is finding critical issues that the automation still isn't finding. Even the boatloads of cash Google and Microsoft have thrown at automating all testing hasn't solved the issue.
When you aren't writing all new code, what then? When you ship your first catastrophic error and find that the dashboards you have aren't really as good as you thought, will that 90 minutes seem like it was enough? Or, when you simply want a human to test for usability, error handling, or the code you forgot to write or put in which even the best code coverage can't check for, the few of us who still put up with the emotional battering it takes to actually give two craps about testing in this hostile environment will be here to help you. How would you feel if a manager took a look at a code generation tool and because it gave an impressive demo felt it wrote better code than you do because it was more consistent? It couldn't replace your talent and experience. What if despite evidence to the contrary, they insisted it was true, and the room full of people clapped? Why are you still sending it to uTest? Because you've only checked it. If no humans have used it, you really have no idea what you are shipping to the public. That may be an appropriate risk in your business. Unlike you, I'm not one to tell people in other professions what is and is not true in their business.
I'm just thankful that there are places where developers and testers still work together. I'll be found in those places. It was difficult to write this. I'm glad I did, because I'm not sure I could give an overview without getting teary at our Seattle meetup tomorrow.
Oh San Francisco Selenium Meetup, you've got yourselves some major fans. On the way there in the traffic Marlena and I joked that the California state motto is, "Keep up or Get Out!" Well, I'm out. I also feel thankful that I work for a client who's been in business with NO testing team for 20 years. I'm the first tester. We now have a team! I'm working full time and another tester is working part time, and even our developers are learning to test. I'm also coding some features, shockingly enough.
It makes me very happy that now the person who hired me says, "I get it! You don't have to explain testing any more to me. I'm seeing it!" Yes. He can survive without testing, but we can all deliver features that have value to the customer faster and more reliably when we all care about and do testing. We don't have all of the automation we would like, and yes, I'd like more of a balance. I care too much for my profession to wish for anything other than forward progress for the industry. I love using software, not just earning a living. That is a big part of why I love testing. Each person I prevent from having a bad experience with software is a point of pride for me. That is true if I do the testing, or even if I teach one developer of a test he might run himself. If I teach a tester one idea they may not have tested before, I'm as happy as if I'd prevented the bug with my own code or my own test (I use both manual tool assisted tests and automated checks).
It isn't the unintentional innocence that upsets me. It's the willful ignorance combined with marginalizing an entire profession. I have more than one skill. I'm not terrified that testing will die, because if it does, there are many other possibilities. I fear for the users of this software, because I am a heavy user of software. I also worry about who you are going to hire to do this totally boring work that no one wants to do. This repetitive droning work that so many companies keep trying to staff. You don't get it. You don't get why you suck at hiring testers. You don't understand why your SDETS are all moving to be full coders now? No one wants to be treated like an idiot and to be your code monkey boy. Stop treating people like they are less than human and then wondering why it's so hard to hire and retain the test coders with your laundry list. Then again, that's for another blog. I'm just thankful that the trip didn't end on this note, or I may have started to drink shots on the plane ride home to soothe my sore feelings rather than working on my testing ideas. There is a reason I'm working so hard to talk beyond just testing. A story of how easy it is to feel "100% safe" after running 90 minutes of automated tests is seductive. There is nothing that can be said to that. You can't warn. You just let it run it's course. Either there will be pain or there won't. I'm certainly at my limit when it comes to fighting a gorilla with a toothpick. I'm here for those who want to work with me, not to convince those who've got the answers already.
The bridge was still beautiful later. The structure wasn't as visible, but it was still there holding up the cars. The lights were far more shiny when the sun was down. It didn't make the supports less useful. I hope when they were last tested, it wasn't just in the emulator.
So, for me, the Selenium SF meetup was the most depressing night of January. I'm happy I met some nice people, and I don't let the opinion of a few folks lead me to assume that everyone shares the same opinion. I have hope for the future, and I am really optimistic to have a happier report for you about the Seattle meetup, which I'm going to be participating in. I'm still planning to take a class to improve my understanding of testing using Selenium, hopefully from Adam Goucher. By the way, if you think testing isn't important, please ask Sauce Labs what they think of the importance of testing and if they have any QA team or testers. I believe they have an agile team, which includes testing in many forms, including developer testing and other agile testing, including some thoughtful human exploratory tests.
Now, on to February and BEYOND! I hope the happy post from earlier was enough to sustain you through this long and pretty sad post.
I'm interested in learning more about selenium, well, to be specific, I'm interested in using Web Driver with Python and Ruby to automate tests where it makes sense and basically expand my capabilities in web testing. I believe automated testing is very important, and automated checks should be created, maintained, and relied upon as one aspect of quality by the development team. The whole team. I mean that when you ask who is responsible for quality the whole team says, "We are." When asked, "Who fixes the automated tests when they break?" I also want a clear, "We do and we don't develop new features until they are fixed." Anything less means your automation really isn't working to the level that your team can depend on it, so it doesn't have a bright future at this point, or should be rated "needs improvement" by the team. I've seen great test automation only twice. Both times the team owned it, relied on it, and would literally fix it right then. No shutting off tests. If it isn't sacrilege to turn off broken tests that used to work, the automation just isn't reliable enough to be counted on by your team.
So, with all of this in mind, let me share first the happy. So many things to love about San Fran and Mozilla. The patio over there is WOW. Mozilla's got some bucks. That's all I'm saying! I almost felt bad about drinking their soda for a minute, but I think maybe they can afford even my Diet Coke for a night.
Also, a kind fellow took this shot for me! Volunteered even. That makes for a happy tourista. No, my hair hasn't turned black, it's was just a bit dark outside at this point.
Lest you think I was cranky, I had a blast that day. I've been waiting for a huge merge, and my devs did testing together! Yes! They fixed bugs and swarmed for the first time. This team had NO testers, and now they are testing their own code and I feel so proud of them I could burst because they are showing just what a team can accomplish together, and they care about testing their own work. Then I spent the day finding issues, and I even handed off a feature I coded, so I'm learning and teaching on the job which basically means that my self-esteem is at an all time high. I'm seeing a great friend who I enjoy, the sun is shining. I had a new type of food, even red wine the night before. I've basically been drinking free cool Diet Cokes between lovely sun breaks, and despite being pale I'm not sunburned at all since the sun is so mild in the Bay Area this time of year. I'm expecting to learn more automation and even see some developer testing, which I believe is essential for delivering quality software.
During setup, the meetup hosts had some problems, and I was able to suggest ways to use the seating already there so everyone could see. I felt helpful, competent, happy, and above all, like a part of this group excited to see what new things companies are doing with Selenium. It doesn't matter to me where the testing is happening. I am a fan of all testing. I believe that different companies and contexts can tolerate different amounts of risk, so I don't assume that every single company has a use for a test team.
Now the Sad Part
When the talk started, I had an open mind. I am learning more about presenting for developers and mixed tester/developer crowds. Personally, I felt that the slides weren't attractive and that they looked a bit "retro" like a 1995 slide-deck, but I said that before I was upset at all. Honestly, it's a great point that the content is all that really matters. The speaking was good and the story did flow. People were extremely engaged. At least you can tell from this blog, that despite the lack of beautiful fonts or lovely photography in the slides, that the presentation did invoke some emotion in me.
Large crowd had about 100 people. All polite and friendly.
By the time the speaker was done, I felt totally deflated. The presenter said things to me that felt derogatory to testers and dismissive to testing. I felt this way specifically because of the following statements, which lI'll try to be careful to quote and please verify via the recording to get the tone used.
1. We run 90 minutes of tests so we KNOW that everything is safe.
2. We do not have a QA team and we never want one.
3. It is ALWAYS best to automate every test. (this was said despite admitting vast difficulty with automating some tests)
It isn't only that the speaker said these things, but that many others in the room were nodding with such sycophantic vigor that their heads nearly fell off. As I looked around I envisioned the sinister large framed hipster glasses hiding evil eyes identifying me as a tester, and I felt like I wasn't welcome. The inexperience reeked to me, and I felt as if I was the only person who noticed that the demos AREN'T on a real website.
Then--want to know the secret? How they keep the tests all BLUE? It's this secret we've called resetting the baseline in testing for the past 15 years. They aren't even testing the actual THING. They are just notifying themselves of what has changed. It isn't new. If they had testers they might have found that solution sooner. It's a great way to resolve the false positives that are always a part of the normal complexity when doing automated checks. It is sad to me that developers who are inexperienced at testing are so certain that they have all of the answers that they don't even attempt to get any training in testing from those of us who have been doing testing for many years.
I find these slides retro, and note that code review is for people who don't like each other, pairing is for folks who do, but as a tester, especially as a tester who shines in the areas of integration, collaboration, usability, and finding requirements gaps, my opinion was irrelevant in this case. Perhaps a title change from Test Lead to Overlord of Automated Checks (90 minutes to total perfection code confidence) is in order. Maybe some provisos can be added such as "So long as they happen in Chrome, we thought to validate them, they weren't an issue that needed human judgement, and they were in the subset of important tests that we picked, but other than that we are 100% confident."
I feel pretty awful as a public speaker saying anything negative about any person, and especially a fellow member of the tiny group of females in technology who speak, and I want to be clear that this isn't personal. Not only were the speaking skills very good, but her enthusiasm for automation was infectious in a good way. If I've misunderstood, or misquoted, I apologize in advance and will make edits. I do believe that this is a common view in the Bay area, and just a team approach that was being candidly shared for the benefit of other people. I do appreciate hearing the plain truth of the work that is done by the doers and I know that having a good set of automated tests is hard work that takes technical skill and persistence. It isn't just what was said, but the feeling of such enthusiastic agreement and unquestioning that really emotionally resonated with me. The speaker was an experienced Automated Test Creator also speaking on team philosophy. While she's not likely up for Testing Advocate of the Year award(if such a thing existed), but she may be nominated with a few others for "Most Divisive Talk 2012". I've got many more talks and conferences to attend, so this may not even make the top 5 before the year is out, but I sure hope the "Let's trash all testing done by humans for any reason" trend can make the rounds in the Bay area and the rest of the country can keep doing good teamwork, like the agile teams have been lately. I'm seeing progress in developers and testers working together. I'm seeing good classes in mobile testing. From what I understand, this dismissal of the value of all human testing isn't what is happening in Europe. I think this is common in the start-up community and also in the web space specifically where there is venture capitol. It's less prevalent elsewhere from what I can tell so far.
It isn't that they don't have or want testers that bothers me. Why do you hate testing? Without understanding what you are dismissing why don't developers even ASK for our help to learn testing? I promise, we want you to test. Our numbers are shrinking. I never want to visually compare data again. Nearly all of the testing I do is tool assisted. There is lots of testing going on outside of the bay area. If you look, even in the space of the web, the secret shame of even the large companies is the human testing is STILL being done because it is finding critical issues that the automation still isn't finding. Even the boatloads of cash Google and Microsoft have thrown at automating all testing hasn't solved the issue.
When you aren't writing all new code, what then? When you ship your first catastrophic error and find that the dashboards you have aren't really as good as you thought, will that 90 minutes seem like it was enough? Or, when you simply want a human to test for usability, error handling, or the code you forgot to write or put in which even the best code coverage can't check for, the few of us who still put up with the emotional battering it takes to actually give two craps about testing in this hostile environment will be here to help you. How would you feel if a manager took a look at a code generation tool and because it gave an impressive demo felt it wrote better code than you do because it was more consistent? It couldn't replace your talent and experience. What if despite evidence to the contrary, they insisted it was true, and the room full of people clapped? Why are you still sending it to uTest? Because you've only checked it. If no humans have used it, you really have no idea what you are shipping to the public. That may be an appropriate risk in your business. Unlike you, I'm not one to tell people in other professions what is and is not true in their business.
I'm just thankful that there are places where developers and testers still work together. I'll be found in those places. It was difficult to write this. I'm glad I did, because I'm not sure I could give an overview without getting teary at our Seattle meetup tomorrow.
Oh San Francisco Selenium Meetup, you've got yourselves some major fans. On the way there in the traffic Marlena and I joked that the California state motto is, "Keep up or Get Out!" Well, I'm out. I also feel thankful that I work for a client who's been in business with NO testing team for 20 years. I'm the first tester. We now have a team! I'm working full time and another tester is working part time, and even our developers are learning to test. I'm also coding some features, shockingly enough.
It makes me very happy that now the person who hired me says, "I get it! You don't have to explain testing any more to me. I'm seeing it!" Yes. He can survive without testing, but we can all deliver features that have value to the customer faster and more reliably when we all care about and do testing. We don't have all of the automation we would like, and yes, I'd like more of a balance. I care too much for my profession to wish for anything other than forward progress for the industry. I love using software, not just earning a living. That is a big part of why I love testing. Each person I prevent from having a bad experience with software is a point of pride for me. That is true if I do the testing, or even if I teach one developer of a test he might run himself. If I teach a tester one idea they may not have tested before, I'm as happy as if I'd prevented the bug with my own code or my own test (I use both manual tool assisted tests and automated checks).
It isn't the unintentional innocence that upsets me. It's the willful ignorance combined with marginalizing an entire profession. I have more than one skill. I'm not terrified that testing will die, because if it does, there are many other possibilities. I fear for the users of this software, because I am a heavy user of software. I also worry about who you are going to hire to do this totally boring work that no one wants to do. This repetitive droning work that so many companies keep trying to staff. You don't get it. You don't get why you suck at hiring testers. You don't understand why your SDETS are all moving to be full coders now? No one wants to be treated like an idiot and to be your code monkey boy. Stop treating people like they are less than human and then wondering why it's so hard to hire and retain the test coders with your laundry list. Then again, that's for another blog. I'm just thankful that the trip didn't end on this note, or I may have started to drink shots on the plane ride home to soothe my sore feelings rather than working on my testing ideas. There is a reason I'm working so hard to talk beyond just testing. A story of how easy it is to feel "100% safe" after running 90 minutes of automated tests is seductive. There is nothing that can be said to that. You can't warn. You just let it run it's course. Either there will be pain or there won't. I'm certainly at my limit when it comes to fighting a gorilla with a toothpick. I'm here for those who want to work with me, not to convince those who've got the answers already.
The bridge was still beautiful later. The structure wasn't as visible, but it was still there holding up the cars. The lights were far more shiny when the sun was down. It didn't make the supports less useful. I hope when they were last tested, it wasn't just in the emulator.
So, for me, the Selenium SF meetup was the most depressing night of January. I'm happy I met some nice people, and I don't let the opinion of a few folks lead me to assume that everyone shares the same opinion. I have hope for the future, and I am really optimistic to have a happier report for you about the Seattle meetup, which I'm going to be participating in. I'm still planning to take a class to improve my understanding of testing using Selenium, hopefully from Adam Goucher. By the way, if you think testing isn't important, please ask Sauce Labs what they think of the importance of testing and if they have any QA team or testers. I believe they have an agile team, which includes testing in many forms, including developer testing and other agile testing, including some thoughtful human exploratory tests.
Now, on to February and BEYOND! I hope the happy post from earlier was enough to sustain you through this long and pretty sad post.
I think it was either Bolton or Bach (at CAST) who said something along these lines (paraphrased, barely remembered in fact) ... product owners crave definitive results, whether real or imagined, & are often willing to sacrifice overall quality to get results that answer some questions, any questions, without posing new ones.
Something like that, anyway. I've observed the same thing in my career. I could call it the elipsis effect. Humans tend to be uncomfortable with questions they can't answer, and unless you're a tester or some other type of subject matter expert pertinent to the project, you (as the product owner) crave finality & fear anything that seems even remotely like ambiguity. You crave the period & fear the elipsis, & god save us if there's a question mark.
Reply to this
I expect to have this kind of reaction from product owners and CEOs, and others who don't really understand the complexity of the software. Why can't coders just get it right the first time, and then we don't need to test. It isn't new that testing isn't well understood and is undervalued. I guess I just expect better from developers doing testing. I expect more insight from any test lead. I also realize that maybe I misjudged the audience. I just had such a different experience when I went to SFAgile. The crowd there included the guy who WROTE The Lean Startup and he understood more about the value of testing than the impression I got from the presentation and questions asked and answers given at this meetup.
Feel free to quote me, but when you do please fix my typos.
Reply to this
Hi Lanette,
Thanks for sharing your experience. I do find it sad when teams fail to see the value in devoted testing activities and consider 100% developer automation to be a noble achievment.
When you say "this dismissal of the value of all human testing isn't what is happening in Europe" - from my experience as a UK based tester these philosophies do seem to be more prevalent from the US but maybe that is just because we haven't caught up yet. I am fortunate to be working on a team where the developers place a high value on the testers, to the extent that the Dev Director when offered two new developers actually lobbied for 1 dev and 1 tester to better balance the team.
I'm actually speaking on the subject of some of the risks facing testers in the current market, one of which is exclusive developer testing, based on some posts that I wrote recently at a TestBas. Would it be OK to reference/quote your experience here as part of that talk?
Many thanks
Adam
----------------------------------------
http://a-sisyphean-task.com/
twitter: adampknight
----------------------------------------
Reply to this
Hi Adam,
I went to SFAgile last year and I felt that testing was well accepted and understood in that group more than at this meetup, but I also had all day to talk to people and more speakers, not just a short time and a few people.
Feel free to quote me, but when you do please fix my typos. I'd been up too long with a sick cat, so it isn't my cleanest writing. I accidentally put that comment to Blake but I had intended it for you. Good luck in your talk! Very cool that you are talking about risks testers face. I am talking about Agile Perversion as a big risk testers face at the online STPCon. I'll try to post a link when it's online. I like the fact that it is possible to attend from anywhere.
Reply to this
Hi! I was at that meetup and I work at Sauce Labs.
I don't think things are really as bad as you think here in hipsterland. When Denali said Okta doesn't have a testing team, she means they don't want a team that *only* tests. She means there's no distinction between people who write code and people who test it. They're the same people. At Sauce, we don't have a testing "team" either. That's because here in SF we have finally figured out what you knew all along: testing is the responsibility of the whole team.
When you give programmers the responsibility to test, they automate it as much as they can. Which is great! When we have the rote, routine tests automated, we only have to manually test the things that humans are better at than machines. Like, is the page unreadable, or is the submit button covered by a stray div or something.
I can see how you'd get upset by the idea that we don't care about testing, but in fact the opposite is true. We're making testing cheaper, so we can do more of it.
Reply to this
Hi Joe,
I think that you are correct. I went to SFAgile and was very inspired. However, I had all day for 2 days, many speakers, and we talked in an interactive way.
I guess the things that were shocking to me are what I consider perpetuating myths of automation. I particularly dislike hearing it from anyone who has done enough testing that they should know better. I had very high expectations because the last time I was in town there was a general understanding of what tasks need human evaluation, and how we can all use tools so that specific changes can be looked at sooner instead of testers manually repeating mundane tasks.
I am happy to hear any speaker who is enthused about how cool it is to gain speed and confidence through the use of tools. I've experienced that joy as well, which is why I'm spending my time to learn more. I just want to do so without feeling trashed, defensive, or hated. It is quite possible to be happy about progress with automation and share it without the belief that 90 minutes worth of tests on one platform should be all that it takes to believe your code is perfectly bug free. That comment particularly made me want to hurl. It is the sort of soundbyte that people cling to, and it does subtle damage that is impossible to prove until the ship is sunk and the team who put it in place has moved on to another startup.
I went to see Jason Huggins present at STPCon, and he did a fantastic job of explaining what you can do that is exciting without giving the impression that there are no limitations or risks associated with an unbalanced test strategy.
I believe that many people do care about testing, but that there is also a trend, a fad, or a misconception. The myth that large companies are shipping great software with NO testing happening beyond the automated checks is dangerous and untrue. I thought I heard that popular and seductive idea being spread about like confetti, which I consider reckless, unproven, and misguided. Not just because I'm a tester, but because even if I do another career, I don't want to be forced to use the software created like that. It's the opposite of craftsmanship. It's crapmanship. I'm glad you are making testing cheaper, but can you care about making it better too? Because better is a good option in many contexts. In some ways, it matters more than just cheaper. Also, is it really cheaper? Based on what? Are we even comparing 2 things with the same outcome? I'm not convinced that anyone is even asking. They just watch the amazing screens go by without a concern about how it's validated. I want to know. I don't want to be tricked.
This is so long it's almost another blog. Thanks for sharing your take on the talk.
Lanette
Reply to this
Hi Lanette,
I found me nodding my head to what you think about Testing and testers.
It is what i have seen and observed. Testers are treated as no big deal, in most of the organizations. I have noticed that devs have some special bonding with a tester who codes and can get into the code than the tester who would give more usability related inputs to improve the product.
And this question constantly bangs in my head, should I learn to dig the code and find flaws? Or should I be focusing on Usability issues ??
Thanks for writing this up...
Reply to this
Hi Prerna,
I believe that all testers can benefit from understanding object oriented programming, at least to the pseudocode level, but further if possible. It is something that you can train yourself on over time, with projects. Lots of online help exists for getting started with programming, including learning how to program. there are many tutorials. The main thing to remember is that coding is just another way of communicating. The more you can understand, the more you can participate in the discussion. This is why it is helpful to know programming basics. Once you understand what is being talked about, it makes it easier to learn and apply. I hope this helps some.
Thanks,
Lanette
p.s. I started with scripting for testers and free tutorials on the web.
Reply to this
Hmmm, I'm a some sort of product owner (hoping to get better), but funnily my background is in testing. So I deeply care about testing too, as I know what happens if we don't worry about quality early on.
Testing, test cases are also an asset for the shippable product. So in describing the thing-to-be-done, one part for both dev+test are some "hook-questions" to *prevent* some obvious errors and initiate the dev+test discussion more. By discussion we get better feature, testing, quality and understanding all around. If delivering better quality, takes a tad bit of time, so be it.
Reply to this
Hi T.,
Glad that you are making testable requirements. Many of the "hook questions" are really a checklist that the developer can prevent when they know the acceptance tests that will be run by the tester.
I think anything that improves the team's delivery of quality software sooner is a key measure of success on the team, as is the retention of your best testers.
Reply to this
Lanette,
Apologies if I didn't pick up on this from your narrative, but did you talk with the speakers after the presentations?
Time-limited presentations, tend to focus on strongly making a point, sometimes to the exclusion of balance. Articles and blog posts are often couched the same way: Taking an extreme position in order to generate controversy and debate, even if it's not representative of the author's actual position.
It would help me understand your position better if I knew that you took the time to check the speaker's actual opinion. If this account is based only on what you saw and heard in the talks, then that would be useful to know, too.
Thanks.
Reply to this
Hi James,
I waited all the way to the end of the meetup, but did not get a chance to ask. I did wait, but people were so excited about it that finally, over 30 minutes after both speakers had finished I had to leave without ever asking.
I do not know this speaker at all, and my opinion is based only upon what was presented (check the link to verify what was said including tone and so on). In fact, I thought the speaker did a good job even though I didn't like the slide style. I do not know the speaker personally and did not get a chance to speak with her after, although I did try when the presentation was over, but gave up eventually as we had to go.
Thanks,
Lanette
Reply to this
Okay I just watched it, and in general the talk seemed a lot like an overview to me. Not a lot of details, so you are left to imagine how exactly the testing worked. On some of the issues you mentioned I have the following reactions:
When I hear someone say anything remotely like: "You should never manually test, you should always test in an automated way." My tendency is to immediately send up a red flag, what are they talking about, and clearly you might need to automate the testing if the only thing that interacts with the people were other computers. Yet I think if you follow the lines close enough, you'll realize that there are very few computer systems, if any that run entirely without some sort of human intervention or interaction through input changes, controls, or consumption through displays or other such outputs.
My reaction to this, is that this person, probably is focused on her specific context, and is making broad generalizations at times without really thinking about what from her experience would apply elsewhere. At least that's how the initial part of the talk seemed to me. Later in QA she mentions experience with other systems, and the problems she had there, so that raised other issues in my mind relating to that statement.
Coming from the engineering side, having worked as a coder and a tester, I feel I have a particular impact on this. A lot of talk is given on things like unit tests, and some developers may try to treat Selenium as if its just another type of unit test, but really these people aren't testers, they are coders, who seem to have little real knowledge about testing, outside of testing this function or that function. The fact that she mentions her netflix UI changes as an issue, pretty much shows this idea that they are just 'checks' and that without some clever intelligence introduced they'd fail.
The other thing that caught my eye was the # of tests, "Full Tests", "Exhaustive Tests", etc. So they have a continuous integration setup 8K unit tests, 3K selenium tests, but the slide says 90 minutes Full Testing and 6 Hours exhaustive. That seems confusing to me. She clarifies later at around the 6 minute mark that the Exhaustive test is if they ran a bunch of things, they don't really want to run. I'd have been curious to ask, why don't you want to run them? Are they valueless? what quality does that group of tests that takes an additional four and a half hours have which makes you say that? The other thing she doesn't say till later, is whether she's running this on a distributed setup, or at least how many machines (later it looks like she shows an example of 50 threads, but is that 50 threads on 1 machine, multiple machines, that wasn't entirely clear to me). 90 minutes might be a lot of time if you have 100 machines cranking on tests, but if its just one machine, I'd question whether that's sufficient. That's just me.
Reply to this
Part 2:
When she asks if that's a lot of test, I think the question, rhetorical or not implies what our community has been saying for a long time. Saying you have X tests is like saying you have X test cases. Without knowing the scope, length, duration, and areas each test covers it's difficult to say whether that is a small number, a large number, a reasonable number, or an effective number. The fact that she asks, planted doubt in my mind at least, now maybe they are expansive tests, and maybe her product is covered well with that breadth of tests.
She also talks about buy in and investment early on automation, and I think that's a good idea, in most cases, however, buy in early still doesn't make me totally feel manual testing should go out the window, especially if as she says, they do 'weekly' releases. I don't understand Jenkins, its not a CI environment I've had a chance to work with, so when she talks about master/slave, while I get the analogy generally, I lack sufficient technical knowledge to know if they have a master that loads a slave onto one machine, running tests on just that machine, or if it spawns surrogates to divide that workload with Sauce Labs work. That at least was something that caught my interest, and maybe I need to read a bit more to get an idea, as to how this helps.
Overall, I found this to be more of an overview story and kind of dry on details aside from setups used. I understand Lanette's chaffing at hearing some of these things being said, and given I read her blog first, I was prepared for some of these things to be said. However, I can't help feel that she seems a bit 'fresh to testing'. I know a lot of developers who think this way, that testing is a minor thing, we'll just automate all of it and be done, as if that's enough. They get that behavior because of little time spent on the discipline in university coursework IMO. I know that was the case for me until I hit a brick wall, and said there has to be a better way, and began to look for it.
I understand the risk, but it almost seems, based on what she said about their product, that there is manual testing going on, likely by the consumers of her service. She just is oblivious or forgetting they exist. Am I wrong in thinking that?
Reply to this