Test Driven Development Reality Check
When I first heard about Test Driven Development, I think my bias for testing blinded me. I was only thinking about the first part! Test Driven. That is when the fantasy began. What would Development be like if it was driven by testers? Testers of power? Like capes and spandex? I wonder if I'd get a tiara, or at least a lunch invite if my tests were really great? Would Developers really care about testing if we used this magic bullet? All I'd heard was the title. In the 1 day between hearing the title and my first research, I'd built up this idea of what this exciting new thing was going to do! Every possible misconception was there. Some of the false things I believed were:
1. It is for testers.
2. Testers do this!
3. The testers start FIRST, and THEN, well, the coding happens.
4. The testers DRIVE the development, so they have the ULTIMATE POWAAAAH!
5. No testers, no code.
6. No testers, no project.
7. Developers really are going to be thinking about testing and learning the practice because they are interested.
8. We are going to pair with these Developers and put in tests right as they code! This is going to be so cool.
9. Integration testing WILL be a part of the definition of done (of course).
10. Tests will have to pass, including error scenarios, complex end-to-end tests, and usability for this code to be considered done.
11. If the tests break, the code moves out of done and is now "undone".
Let's just say that Test Drive Development isn't that. I don't know why I thought it might be. Not that my idea is so great, but I'd be interested to know which parts of it would be good if it were tried for releasing software.
So, what IS Test Driven Development from what I've seen? Well, it is this practice where almost always, when coding, the developer thinks of a test and makes it fail. Then the dev will write code to make it pass! Quite simply, it is a discipline. I think many devs at first would find it torture. It is test hindered development when you first do it, but long term it can save lots of time. But what is it really? It is a way to build in your own unit testing. Now, it could be MORE than that. You could work with a tester and after your basic functionality is completed, you could add some more tests to it, but that isn't what the practice is right now. Now, at best, you get code with related tests that work completed in the code. In a way, that is some self-unit testing code. Well, that is something cool too. It isn't the same as delivering "FULLY TESTED" code like some of us may dream of, but even in my moment of maximum disapointment that I wasn't wearing a superhero costume afterall, in fact I wasn't even invited to the dance, could I ever say that it isn't a step in the right direction for quality.
It is one practice for coding. It isn't even a whole process for delivering software, but at least it is practical. Now, I'd personally shoot for less than 100% of all code having tests that come with it. Some things are just not worth the time, and if the risk is low, any practice taking to the extreme can become silly. I'd be happy with "We have tests with our code in all cases where it makes good business sense. There may be exceptions and when agreed on by the team, we'll make them." I don't see a month delay to a release for writing one test. That would be silly, unless that test was really really important. Again, I'm not saying that TDD isn't testing. I'm just saying that delivering tests with code, even writing tests first isn't what I'd consider "Test Driven". I'd consider this "Test Inclusive Development", and also I'd consider it "Unit Check Validated Development", I'd also consider it, "Semi Self-Validating Code Modules". That is revolutionary stuff. It isn't anything to sneeze at. I just wonder, could it be taken farther?
Test Driven to me is a natural state of relating everything back to testing. When I hear about new things, I start thinking about how I'd test that. So I literally thought testing was in CHARGE of development somewhere. Again, just like testing is gating the pace of development in TDD, I guess I miss there being SOME criteria where it really mattered when you hit a milestone.
I wonder what would happen if a project were "Test Driven" in the way I define it. I mean that the decision over if something was "Done" or not was entirely on test results. No exceptions. Also agreed upon integration, performance, and usability tests would have to be run, and if the coding was done, everyone would run test cases. Automated, manual, or tool assisted, all people would get the software up to where it should be for delivery. When the testing was done for that part, maybe the tester would work with the dev to come up with a list of tests for that functionality in the context of the whole application, including the integration tests. Now that, to me, would be a full version of "Test Driven Delivery". That is a different TDD. Test Driven Development addresses only how we craft software code with quality. It is done by software craftspeople (sorry, can't do the craftsmen thing), and also is talked about far more than it is done in practice. In practice it takes PATIENCE. I'm getting tense just thinking of the patience it takes. If all you have is developers, there is no other way to code. How else would you even get a minimum level of consistent quality? It makes tons of sense to me. However, what if you do have testers? Wouldn't you use them more than just when you are finished writing your code with unit tests written in to it?
To me, TDD is one small part of cooking with flavor. It says, "To make a juicy bird, insert delicious flavor under the skin at regular intervals. It does not say that you also should test the entire bird is done all the way through. It doesn't have to. It isn't Test Driven Delivery of software. It isn't a test strategy. It is just one practice.
1. It is for testers.
2. Testers do this!
3. The testers start FIRST, and THEN, well, the coding happens.
4. The testers DRIVE the development, so they have the ULTIMATE POWAAAAH!
5. No testers, no code.
6. No testers, no project.
7. Developers really are going to be thinking about testing and learning the practice because they are interested.
8. We are going to pair with these Developers and put in tests right as they code! This is going to be so cool.
9. Integration testing WILL be a part of the definition of done (of course).
10. Tests will have to pass, including error scenarios, complex end-to-end tests, and usability for this code to be considered done.
11. If the tests break, the code moves out of done and is now "undone".
Let's just say that Test Drive Development isn't that. I don't know why I thought it might be. Not that my idea is so great, but I'd be interested to know which parts of it would be good if it were tried for releasing software.
So, what IS Test Driven Development from what I've seen? Well, it is this practice where almost always, when coding, the developer thinks of a test and makes it fail. Then the dev will write code to make it pass! Quite simply, it is a discipline. I think many devs at first would find it torture. It is test hindered development when you first do it, but long term it can save lots of time. But what is it really? It is a way to build in your own unit testing. Now, it could be MORE than that. You could work with a tester and after your basic functionality is completed, you could add some more tests to it, but that isn't what the practice is right now. Now, at best, you get code with related tests that work completed in the code. In a way, that is some self-unit testing code. Well, that is something cool too. It isn't the same as delivering "FULLY TESTED" code like some of us may dream of, but even in my moment of maximum disapointment that I wasn't wearing a superhero costume afterall, in fact I wasn't even invited to the dance, could I ever say that it isn't a step in the right direction for quality.
It is one practice for coding. It isn't even a whole process for delivering software, but at least it is practical. Now, I'd personally shoot for less than 100% of all code having tests that come with it. Some things are just not worth the time, and if the risk is low, any practice taking to the extreme can become silly. I'd be happy with "We have tests with our code in all cases where it makes good business sense. There may be exceptions and when agreed on by the team, we'll make them." I don't see a month delay to a release for writing one test. That would be silly, unless that test was really really important. Again, I'm not saying that TDD isn't testing. I'm just saying that delivering tests with code, even writing tests first isn't what I'd consider "Test Driven". I'd consider this "Test Inclusive Development", and also I'd consider it "Unit Check Validated Development", I'd also consider it, "Semi Self-Validating Code Modules". That is revolutionary stuff. It isn't anything to sneeze at. I just wonder, could it be taken farther?
Test Driven to me is a natural state of relating everything back to testing. When I hear about new things, I start thinking about how I'd test that. So I literally thought testing was in CHARGE of development somewhere. Again, just like testing is gating the pace of development in TDD, I guess I miss there being SOME criteria where it really mattered when you hit a milestone.
I wonder what would happen if a project were "Test Driven" in the way I define it. I mean that the decision over if something was "Done" or not was entirely on test results. No exceptions. Also agreed upon integration, performance, and usability tests would have to be run, and if the coding was done, everyone would run test cases. Automated, manual, or tool assisted, all people would get the software up to where it should be for delivery. When the testing was done for that part, maybe the tester would work with the dev to come up with a list of tests for that functionality in the context of the whole application, including the integration tests. Now that, to me, would be a full version of "Test Driven Delivery". That is a different TDD. Test Driven Development addresses only how we craft software code with quality. It is done by software craftspeople (sorry, can't do the craftsmen thing), and also is talked about far more than it is done in practice. In practice it takes PATIENCE. I'm getting tense just thinking of the patience it takes. If all you have is developers, there is no other way to code. How else would you even get a minimum level of consistent quality? It makes tons of sense to me. However, what if you do have testers? Wouldn't you use them more than just when you are finished writing your code with unit tests written in to it?
To me, TDD is one small part of cooking with flavor. It says, "To make a juicy bird, insert delicious flavor under the skin at regular intervals. It does not say that you also should test the entire bird is done all the way through. It doesn't have to. It isn't Test Driven Delivery of software. It isn't a test strategy. It is just one practice.
Posted by Testy Redhead at 8 Oct 2010 
Categories: testing, unit test, test strategy, automation
Tags: test test-driven delivering test driven scrum Agile automation unit testing testers agile TDD testing development
Categories: testing, unit test, test strategy, automation
Tags: test test-driven delivering test driven scrum Agile automation unit testing testers agile TDD testing development
Hi Lanette.
With kindness, I'm not certain you quite grok the essence of TDD (at the unit/micro test level), because some of your phrasing is incongruent with the practice and results. This probably doesn't matter that much, since you identify yourself clearly as a tester and not a developer, but it may be colouring your view.
However, some of what you describe in your dream is now being done under the name 'Acceptance Test Driven Development', and also relates to 'Specification by Example' and to aspects of 'Behaviour Driven Development'.
A typical scenario in these approaches is that the *right* group of people get together when a story / requirement / feature is going to be built and talk about how it is going to be tested. It is common at this time to split the story if too many tests are identified, which allows a focus on higher value parts. Then, those tests that can be, will be automated either before or at the same time as the relevant production code is being produced (probably in a TDD way).
I emphasise 'right' above, because development should not be totally controlled by testers, or developers, or project managers, etc. It is a team game after all, and we do it to deliver value to a business and joy to our users.
I think you know Lisa Crispin, Elizabeth Hendrickson and Matthew Heusser, who are all exponents of this approach. Also Gojko Adzic is well worth looking up (and has a book coming out soon).
Finally, you wondered if this could be taken further. Well, probably a limit of this approach is demonstrated in the 'continuous deployment' approach, which seems to be in the very early adoption part of the curve at the moment.
Regards,
Lance
Reply to this
Hi Lance,
Thank you for the thoughtful comment. I come from a testing background and my limited coding experience has been writing some Python for testing purposes in the past year. The biggest caveat anyone reading should consider is that I've never DONE TDD. I've only seen in done.
When I had some trouble with my test code, some suggested I use TDD and test my own tests. That went over like a lead balloon with a newbie. Great! Now I have code to check the code that is checking other code. Should I check that code too? Not sure how deep it goes.
My only example in practice of TDD has been other people showing me how they do TDD in a few different contexts and reading some articles. I first heard about it after I'd already taken Scrum Master training from Ken Schwaber.
When I spoke of testers totally controlling a project, that is only the incorrect impression I got from the title. I don't think it is "right" or even a good idea. I agree that quality is a team responsibility. That is the only way it can happen.
I find that what I've read about TDD and what I see in practice is about as far apart as what I learned from Ken about agile, vs what happened when I was on a project that actually used Scrum. When it comes right down to it, talented and goal oriented people will subvert any process put in place to deliver good results. Some employees will use any process you give them to deliver crap. The process just isn't as important as the humans. That is what I find so odd about what has happened with Agile.
I read the manifesto and think it is great, sensible, and the practices SHOULD be fairly adaptable. I'd like to think that my misunderstanding of TDD is rare & specific to me, but when I talk to other testers, I worry that it is universal. I'd like to know what I'm missing in what I describe so that I can understand better what it can be outside of the few examples I've seen.
Thanks,
Lanette
Reply to this
Hi Lanette.
I'm aware of some of your background from your Twitter activity.
I'll take a few quotes from your original post and from your more recent reply to me to show why I think you're not grokking TDD. A better way would be to get Kent Beck's book on TDD.
You said: 'Well, [TDD] is this practice where almost always, when coding, the developer thinks of a test and makes it fail. Then the dev will write code to make it pass!' - The intent is really important here. The developer will actually start by deciding what is the next increment of functionality they want to implement. They will then express that formally using a unit test. This test must fail, otherwise the functionality already exists. It may not even compile because it depends on stuff in production code that doesn't exist yet. Once that is achieved, they will then write the minimal production code necessary to get the test to pass. They may then refactor some code, or they may go round the loop again. Another important thing to understand is the time to go round this loop. It is typically a few minutes.
'But what is it really? It is a way to build in your own unit testing.' - Oh no; it is so much more than that. It is an aid to design, a way of communicating and collaborating with colleagues, a problem solving technique, a way of keeping code simple and plastic, etc.
'Now, I'd personally shoot for less than 100% of all code having tests that come with it. Some things are just not worth the time, and if the risk is low, any practice taking to the extreme can become silly.' - Even for hard-core TDD people, you're not likely to get 100% coverage all the time. One reason for this is that you might need to make use of a badly written library that depends internally on some stuff that is just too onerous to deal with in a unit test. So, you wrap it with a very thin layer that is not unit tested, but which isolates the yuckiness. The rest of the time though, with TDD, code does not come into existence without a test existing first. It is never a case of needing to establish risk, or needing to find the time. And, it doesn't take more time to do this. As a result you, probably get meaningful coverage in the high 90%s.
'I don't see a month delay to a release for writing one test.' - TDD would not cause this.
'I'm just saying that delivering tests with code, even writing tests first isn't what I'd consider "Test Driven"' - Indeed not. There is a profound difference between the XP practice of Unit Test First and the approach taken with TDD.
'Great! Now I have code to check the code that is checking other code. Should I check that code too? Not sure how deep it goes.' - Each unit test checks one thing of interest. Production code often has mangled combinations of those things of interest (I don't just mean partitions and arcs). The difference in complexity is enormous. Also, unit tests are usually declarative, whereas production code is usually ... not.
Regards,
Lance
Reply to this
Hi Lance,
First, I'm excited to see Ron write about this. I wasn't trying to slam what TDD is. I've seen it done in two different companies, by two totally different projects, but the way I describe what I've seen is true to what they were doing. It may not be "right", but it is how TDD is practices in business right now in some places. Totally in a silo away from all testing. In fact, they aren't even showing the testers what the tests are in most cases. We still have testers who don't look at any code here.
Ron suggested I try TDD. I would suggest some lowest bar to try TDD. Until you are making some code with moderate skill, it might be too overwhelming. Maybe a staged learning approach. For example, just trying in comments to write out what would be a test for this code. Just like I wouldn't put my newest tester on the trickiest most high risk feature or interaction on a project, I wouldn't toss a brand new coder into doing full scale TDD as intended and expect them to do well. It isn't simple or easy in practice from what I've seen, and I've heard that what I've seen is a dumbed down practice from what was intended.
The hardcore TDD people told me that the gap between what you do and what you say you do is the percentage of bull**** you are. I take that to mean even the most zealous are over 0% bs in reality. I'm glad to hear it. I've seen silly things done just to prove a point or show superiority. I don't think that serves anyone who wants to see the software development industry progress overall.
The last paragraph is way over my head, but the reason why I used the unit test comparison was the intent (to cover the code written), even though the approach is different and "coverage" may be defined differently.
Reply to this
Hi Lanette.
I don't doubt that you are accurately describing what you saw. I see a lot of weird stuff happening too. I also didn't interpret what you wrote as being in opposition to TDD.
However, it is perfectly appropriate that TDD (at the unit test level) is done without involving testers, either before or after the test is written, and before or after the production code is written. The kinds of tests produced with TDD are not the kind that testers are generally going to be interested in.
What I meant by the last paragraph is that a test written for the purposes of TDD (and also frequently for other kinds of test) will declare that given certain preconditions, performing a particular action will result in certain postconditions. Almost be definition therefore, this test is only interested in *one* path through the production code.
Now, the introduction of another test will drive further development, and some of this may involve introducing branching logic in the production code.
The result of this is that the tests are always simple (in the sense that they declare a single fact). However, the production code that is driven into existence by those tests needs to simultaneously 'satisfy' all of those facts, and may therefore consist of conditional logic, looping logic, and all sorts of other branching logic.
Hence the tests are always simple, but the production code may be 'complex'.
One thing you might like to do is look at Kent Beck's videos on YouTube where he demonstrates how TDD should be done. He invented this stuff (including the name 'TDD')- I think we should take his approach as definitive
I also suggest you try TDD. I'm sure Ron would be only too happy to pair with you until you get the hang of it.
Regards,
Lance
Reply to this
You definitely should take a look into "Bridging the Communication Gap" from Gojko Adzic, or his upcoming book on "Specification by Example", though he told me it will not be in print until January.
To me, the thing we currently call ATDD (I think this name might stick just as the TDD thing stuck) looks more appealing to testers, because it is essentially and supportively the thing you described in your list. Testers are involved in building a mental model of the application that is going to be built. We are involved in deriving examples together with the customer, we are involved in getting these literally automated - and since we focused on those examples that bring value to the customer, we bring value to the project.
Reply to this
I've heard good things about his work. I tend not to read any dead tree books, but perhaps a digital edition.
Reply to this
I agree with Markus. The thing you describe sounds more like ATDD (Acceptance Test Driven Development) which is fairly new to the scene and an infinitely better analysis tool, in my humble opinion. If you don't know what it is, in ATDD the requirements themselves become an executable system/integration test case. Unfortunately this concept has gone over like a lead balloon with most programmers because it requires them to think strategically and possibly work with non-technical customers to analyze the requirements. Most programmers I know are like those annoying people who never let you finish a sentence. They rush to what they think is the end, pronounce "got it," and make whatever it is they think you were going to request. ATDD sort of forces them to slow down and make sure they get the analysis right.
Reply to this
What? Slower and more expensive isn't a big hit in business?
*shocked look*
I think some proof that it delivers better results & more sustainable products might help. Then again, maybe we'll just put out free stuff online and let the people test it? Quality is whatever is done fastest?
Reply to this
Actually its not about slower and more expensive, but "slow down and work with the non-technical folk." The whole l33t culture of the hacker uber-folk is so baked into the way most techies see themselves, they turned a series of mediocre movies into a massive hit when it fed into that conceit. And as proof despite mounds of evidence that pair programming succeeds in delivering usable features faster and with fewer bugs, most developers will claim it's not for them since it only "slows them down." They all agree its great for someone else who needs it, though. Just not them. They're code is not the problem and they're five times more productive than the others.
Reply to this
I have to admit, stuck up or not, the reason I love being in software the most is there is a pretty high bar to entry, and as stupid as your average twitter trending topic isn't going to make the cut. Working with people who are mostly smarter than me encourages me to limit my blonde moments & keep learning.
Reply to this
You might find the following useful:
Test-Driven Development Is Not About Testing http://java.sys-con.com/node/37795
Test-driven Development Is Not About Testing http://parlezuml.com/blog/?postid=517
How test-driven development works (and more!) http://blog.jbrains.ca/permalink/how-test-driven-development-works-and-more
TDD is not test-first. TDD is specify-first and test-last. http://blog.orfjackal.net/2009/10/tdd-is-not-test-first-tdd-is-specify.html
It’s Not TDD, It’s Design By Example. http://bradwilson.typepad.com/blog/2009/04/its-not-tdd-its-design-by-example.html
The Sensitivity Problem http://butunclebob.com/ArticleS.UncleBob.TheSensitivityProblem
TDD Derangement Syndrome http://blog.objectmentor.com/articles/2009/10/07/tdd-derangement-syndrome
Echoes from the Stone Age http://blog.objectmentor.com/articles/2009/10/06/echoes-from-the-stone-age
TDD Triage http://blog.objectmentor.com/articles/2009/10/08/tdd-triage
Test Driven Development is Design - The Last Word on TDD http://www.hanselminutes.com/default.aspx?showID=164
It's all been said before, but it's worth repeating: what is TDD, really? http://www.bigvisible.com/asroka/what-is-test-driven-development/
Reply to this
Holy Links, Batman!
I found most amusing the following:
"Each cycle takes no more than a minute or two, and each cycle builds incrementally on top of the prior ones until we have created some working feature that the customer has asked for."
*snicker* Each cycle takes a variable time depending on skill. For this writer, a minute or two. In my experience, up to half a day for some developers.
This is the reason on occasion to ask someone who isn't involved in the process what they see. It may be usually a minute or two. But it might also be half a day.
Reply to this
See these screencasts to get an idea of cycle length:
James Shore's Let's play TDD series: http://jamesshore.com/Blog/Lets-Play/
A taster of Kent Beck's TDD screencasts: http://vimeo.com/user3553347/videos
Software Craftsmanship – katas: http://www.katacasts.com/
Reply to this
I had the opportunity to do a video interview with JB Rainsberger about the stages of automated testing, from test-after to test-first to test-driven.
http://programmingtour.blogspot.com/2009/07/test-first-and-test-driven-conversation.html
Reply to this
Those crazy Canadians!
Ok, I'm on a mission to annoy only 1 Canadian at a time. Adam Goucher is my target this month.
Reply to this
You may find this link interesting http://www.jroller.com/perryn/entry/bdd_on_a_multi_disciplined
Reply to this
Very cool! I read that. I just joined a small side project with the hope that I can practice using the tools you mentioned.
So what is rake? Is that a stupid question? I may be showing my squirrely ignorance here.
Reply to this