Testy Testy: Recent Comments

Comment on Selenium Meetup West Coast Style

Tim Western — Thu, 02 Feb 2012 14:22:33 GMT

Part 2:

When she asks if that's a lot of test, I think the question, rhetorical or not implies what our community has been saying for a long time. Saying you have X tests is like saying you have X test cases. Without knowing the scope, length, duration, and areas each test covers it's difficult to say whether that is a small number, a large number, a reasonable number, or an effective number. The fact that she asks, planted doubt in my mind at least, now maybe they are expansive tests, and maybe her product is covered well with that breadth of tests.

She also talks about buy in and investment early on automation, and I think that's a good idea, in most cases, however, buy in early still doesn't make me totally feel manual testing should go out the window, especially if as she says, they do 'weekly' releases. I don't understand Jenkins, its not a CI environment I've had a chance to work with, so when she talks about master/slave, while I get the analogy generally, I lack sufficient technical knowledge to know if they have a master that loads a slave onto one machine, running tests on just that machine, or if it spawns surrogates to divide that workload with Sauce Labs work. That at least was something that caught my interest, and maybe I need to read a bit more to get an idea, as to how this helps.

Overall, I found this to be more of an overview story and kind of dry on details aside from setups used. I understand Lanette's chaffing at hearing some of these things being said, and given I read her blog first, I was prepared for some of these things to be said. However, I can't help feel that she seems a bit 'fresh to testing'. I know a lot of developers who think this way, that testing is a minor thing, we'll just automate all of it and be done, as if that's enough. They get that behavior because of little time spent on the discipline in university coursework IMO. I know that was the case for me until I hit a brick wall, and said there has to be a better way, and began to look for it.

I understand the risk, but it almost seems, based on what she said about their product, that there is manual testing going on, likely by the consumers of her service. She just is oblivious or forgetting they exist. Am I wrong in thinking that?

Comment on Selenium Meetup West Coast Style

Tim Western — Thu, 02 Feb 2012 14:16:45 GMT

Okay I just watched it, and in general the talk seemed a lot like an overview to me. Not a lot of details, so you are left to imagine how exactly the testing worked. On some of the issues you mentioned I have the following reactions:

When I hear someone say anything remotely like: "You should never manually test, you should always test in an automated way." My tendency is to immediately send up a red flag, what are they talking about, and clearly you might need to automate the testing if the only thing that interacts with the people were other computers. Yet I think if you follow the lines close enough, you'll realize that there are very few computer systems, if any that run entirely without some sort of human intervention or interaction through input changes, controls, or consumption through displays or other such outputs.

My reaction to this, is that this person, probably is focused on her specific context, and is making broad generalizations at times without really thinking about what from her experience would apply elsewhere. At least that's how the initial part of the talk seemed to me. Later in QA she mentions experience with other systems, and the problems she had there, so that raised other issues in my mind relating to that statement.

Coming from the engineering side, having worked as a coder and a tester, I feel I have a particular impact on this. A lot of talk is given on things like unit tests, and some developers may try to treat Selenium as if its just another type of unit test, but really these people aren't testers, they are coders, who seem to have little real knowledge about testing, outside of testing this function or that function. The fact that she mentions her netflix UI changes as an issue, pretty much shows this idea that they are just 'checks' and that without some clever intelligence introduced they'd fail.

The other thing that caught my eye was the # of tests, "Full Tests", "Exhaustive Tests", etc. So they have a continuous integration setup 8K unit tests, 3K selenium tests, but the slide says 90 minutes Full Testing and 6 Hours exhaustive. That seems confusing to me. She clarifies later at around the 6 minute mark that the Exhaustive test is if they ran a bunch of things, they don't really want to run. I'd have been curious to ask, why don't you want to run them? Are they valueless? what quality does that group of tests that takes an additional four and a half hours have which makes you say that? The other thing she doesn't say till later, is whether she's running this on a distributed setup, or at least how many machines (later it looks like she shows an example of 50 threads, but is that 50 threads on 1 machine, multiple machines, that wasn't entirely clear to me). 90 minutes might be a lot of time if you have 100 machines cranking on tests, but if its just one machine, I'd question whether that's sufficient. That's just me.

Comment on Selenium Meetup West Coast Style

Lanette Creamer — Thu, 02 Feb 2012 07:39:15 GMT

Hi James,

I waited all the way to the end of the meetup, but did not get a chance to ask. I did wait, but people were so excited about it that finally, over 30 minutes after both speakers had finished I had to leave without ever asking.

I do not know this speaker at all, and my opinion is based only upon what was presented (check the link to verify what was said including tone and so on). In fact, I thought the speaker did a good job even though I didn't like the slide style. I do not know the speaker personally and did not get a chance to speak with her after, although I did try when the presentation was over, but gave up eventually as we had to go.

Thanks,
Lanette

Comment on Selenium Meetup West Coast Style

James Martin — Thu, 02 Feb 2012 07:06:38 GMT

Lanette,

Apologies if I didn't pick up on this from your narrative, but did you talk with the speakers after the presentations?

Time-limited presentations, tend to focus on strongly making a point, sometimes to the exclusion of balance. Articles and blog posts are often couched the same way: Taking an extreme position in order to generate controversy and debate, even if it's not representative of the author's actual position.

It would help me understand your position better if I knew that you took the time to check the speaker's actual opinion. If this account is based only on what you saw and heard in the talks, then that would be useful to know, too.

Thanks.

Comment on Selenium Meetup West Coast Style

Lanette Creamer — Thu, 02 Feb 2012 06:46:14 GMT

Hi T.,

Glad that you are making testable requirements. Many of the "hook questions" are really a checklist that the developer can prevent when they know the acceptance tests that will be run by the tester.

I think anything that improves the team's delivery of quality software sooner is a key measure of success on the team, as is the retention of your best testers.

Comment on Selenium Meetup West Coast Style

Lanette Creamer — Thu, 02 Feb 2012 06:43:09 GMT

Hi Prerna,

I believe that all testers can benefit from understanding object oriented programming, at least to the pseudocode level, but further if possible. It is something that you can train yourself on over time, with projects. Lots of online help exists for getting started with programming, including learning how to program. there are many tutorials. The main thing to remember is that coding is just another way of communicating. The more you can understand, the more you can participate in the discussion. This is why it is helpful to know programming basics. Once you understand what is being talked about, it makes it easier to learn and apply. I hope this helps some.

Thanks,
Lanette

p.s. I started with scripting for testers and free tutorials on the web.

Comment on Selenium Meetup West Coast Style

T. — Thu, 02 Feb 2012 03:31:44 GMT

Hmmm, I'm a some sort of product owner (hoping to get better), but funnily my background is in testing. So I deeply care about testing too, as I know what happens if we don't worry about quality early on.

Testing, test cases are also an asset for the shippable product. So in describing the thing-to-be-done, one part for both dev+test are some "hook-questions" to *prevent* some obvious errors and initiate the dev+test discussion more. By discussion we get better feature, testing, quality and understanding all around. If delivering better quality, takes a tad bit of time, so be it. ;)

Comment on Selenium Meetup West Coast Style

Prerna — Thu, 02 Feb 2012 03:29:45 GMT

Hi Lanette,
I found me nodding my head to what you think about Testing and testers.

It is what i have seen and observed. Testers are treated as no big deal, in most of the organizations. I have noticed that devs have some special bonding with a tester who codes and can get into the code than the tester who would give more usability related inputs to improve the product.

And this question constantly bangs in my head, should I learn to dig the code and find flaws? Or should I be focusing on Usability issues ??

Thanks for writing this up...

Comment on Selenium Meetup West Coast Style

Lanette Creamer — Thu, 02 Feb 2012 01:46:05 GMT

Hi Joe,

I think that you are correct. I went to SFAgile and was very inspired. However, I had all day for 2 days, many speakers, and we talked in an interactive way.

I guess the things that were shocking to me are what I consider perpetuating myths of automation. I particularly dislike hearing it from anyone who has done enough testing that they should know better. I had very high expectations because the last time I was in town there was a general understanding of what tasks need human evaluation, and how we can all use tools so that specific changes can be looked at sooner instead of testers manually repeating mundane tasks.

I am happy to hear any speaker who is enthused about how cool it is to gain speed and confidence through the use of tools. I've experienced that joy as well, which is why I'm spending my time to learn more. I just want to do so without feeling trashed, defensive, or hated. It is quite possible to be happy about progress with automation and share it without the belief that 90 minutes worth of tests on one platform should be all that it takes to believe your code is perfectly bug free. That comment particularly made me want to hurl. It is the sort of soundbyte that people cling to, and it does subtle damage that is impossible to prove until the ship is sunk and the team who put it in place has moved on to another startup.

I went to see Jason Huggins present at STPCon, and he did a fantastic job of explaining what you can do that is exciting without giving the impression that there are no limitations or risks associated with an unbalanced test strategy.

I believe that many people do care about testing, but that there is also a trend, a fad, or a misconception. The myth that large companies are shipping great software with NO testing happening beyond the automated checks is dangerous and untrue. I thought I heard that popular and seductive idea being spread about like confetti, which I consider reckless, unproven, and misguided. Not just because I'm a tester, but because even if I do another career, I don't want to be forced to use the software created like that. It's the opposite of craftsmanship. It's crapmanship. I'm glad you are making testing cheaper, but can you care about making it better too? Because better is a good option in many contexts. In some ways, it matters more than just cheaper. Also, is it really cheaper? Based on what? Are we even comparing 2 things with the same outcome? I'm not convinced that anyone is even asking. They just watch the amazing screens go by without a concern about how it's validated. I want to know. I don't want to be tricked.

This is so long it's almost another blog. Thanks for sharing your take on the talk.

Lanette

Comment on Selenium Meetup West Coast Style

Lanette Creamer — Thu, 02 Feb 2012 01:12:41 GMT

Hi Adam,

I went to SFAgile last year and I felt that testing was well accepted and understood in that group more than at this meetup, but I also had all day to talk to people and more speakers, not just a short time and a few people.

Feel free to quote me, but when you do please fix my typos. I'd been up too long with a sick cat, so it isn't my cleanest writing. I accidentally put that comment to Blake but I had intended it for you. Good luck in your talk! Very cool that you are talking about risks testers face. I am talking about Agile Perversion as a big risk testers face at the online STPCon. I'll try to post a link when it's online. I like the fact that it is possible to attend from anywhere.