Wasting Time With Test Driven Development?
Thursday, August 5th, 2010 - By Vineet Sinha
Many teams are moving to test driven development and very often this a good thing. In a drive towards increased code stability and maintainability, good test cases can be very helpful. But this is not always the case; while working with development teams, most teams seem to have one or more critical problems. It seems that it is easy to do Test Driven Development (TDD) badly – below are five situations that I have seen multiple times.
1. The Drive for Code Coverage
One of the teams that I worked with had been effectively outsourcing development. As a metric towards doing effective development they focused on making sure that they had high code coverage. While this was a good goal, a number of their developers wrote test cases which just called the new code to execute it and then did not do anything with the results. The test cases would all end with ‘Assert(true)’ and therefore would always have passing test cases.
2. The Absence of User Feedback
A software system with plenty of unit tests is generally considered to be well tested. But it is often all too easy to ignore what is being tested. In TDD the unit tests only point to specific potential problem areas in the projects. Getting user feedback ensures that the features being added (and the specs as defined by the tests) are actually easy to use.
3. Excuse for lack of Documentation
TDD is very helpful in understanding code and figuring out what should be going on. But it can never replace quality documentation. Unit tests are just one aspect of a well documented project. Well maintained UML Diagrams, inline class and method descriptions, as well as architecture guidelines are needed to easily figure out the most important use cases, the key core, and the high level architecture.
4. Letting Architecture Decay
Often codebases that rely heavily on TDD are slower to refactor when the need arises. It becomes more difficult to modify the architecture when you need to update your test cases. A young project with swiftly changing use cases will need the flexibility to rapidly switch architecture whenever necessary. It should not be put off based on the number of unit tests that will need to be updated.
5. A roadblock to ‘innovation’
Sometimes doing new things means that you don’t know what the result will look like, how well your algorithms will work, or whether the caching will be fast enough. In such cases it is hard to have all the specifications (tests) written upfront. Often tests have to be written well after the experimental phase of a new component is completed. Waiting to create tests until you have a clearer picture of an innovative component will save many headaches and test rewrites.
Others?
The above is just a list of some of the more common forms of problems caused by applying test driven development that I have encountered. Have you found any others? Have you figured out a way to make sure teams stay on track?
photo credit: flickr
The Drive for Code Coverage – Your example is not a failure of TDD because it is an issue with the developers not the method.
The Absence of User Feedback – Not true. Lots of unit tests indicate only one thing. Lots of unit tests. Using your example in point one I could write an application with a million unit tests, all which pass but add no value. Again not an issue with TDD but with the people writing the code.
Excuse for lack of Documentation – True. But you should never be using TDD without knowing what you are creating anyway. Your design should be high level IE these classes do this etc.. Let TDD sort out the methods in each class. So again not really a problem with TDD.
Letting Architecture Decay – This is only the case where you are writing brittle tests. If you re-factor a method and your tests break this is a good thing. If you re-factor your class and your tests on other classes break this is bad and means you are not following good design of your application or your tests.
A roadblock to ‘innovation’ – Again not true. Unit tests should, if properly written, give you the confidence to experiment with your codebase at will knowing you aren’t going to break any underlying existing tested functionality. You just branch your code, make changes, and if you get your desired outcome of increased speed etc… and your tests pass then good. If they fail well you need to look at what you are doing again.
Not to cause offence but it looks to me like you are doing TDD wrongly. If you are just coding without thinking of design then the issues you are encountering are an indication of doing something wrong. If you are changing code and tests not related to the method you are changing break then you have brittle tests or tests that are actually integration tests.
Thanks for your comment. I think you missed my point. I was actually talking about people who are doing TDD badly.
I have clarified it in my post now.
Good points, quite common problems. I’ve been doing learning/doing TDD for a few years in my projects. Some of them were small, others really big.
My favorite point is ‘A roadblock to ‘innovation’’. Generally almost all my code i produce is kind of experiment. Many times i just don’t know what i want on unit test level, i’m just learning what current system, library, tool allows – what can i get from it. So mostly i write something like “exploring test” which plays a role of runner for my experimental code. When i have shape of the idea then i convert it to typical unit test.
I skipped acceptance and itengration tests here which generally i always have before. Because those tests operate on higher level (customer) are just easier to establish.
alider, thanks for your comments. Perhaps I should do some research on a set of tips for doing TDD effectively. I like your idea of ‘exploring tests’.
In point 1, your developers were clearly taking the piss.
You can’t make people write good tests, just as you can’t make them write good code.
I would also argue that point 4 is one of the main points of doing TSS – a safety net against change. Refactoring is a (necessary) risk. Surely its better to have some failing tests after a refactor than to just assume that everyone went ok.
@Neil, Great comments.
Regarding point 1 – yeah, developers in some situations do strange things. The manager in this case was very capable and very hands on but the team consistently had this problem. The manager and I were discussing various tricks to drive home the point on the need for writing good tests. So, yes, you can’t make people write good tests, but you can try to get them to understand what a bed test is (i.e., the intent of this post) and you can give them a reason to write the good tests.
Regarding point 4 – yes a safety net is a key benefit of TDD, but it does have unintended consequences. We need to realize this and make sure we don’t fall in this bad situation (letting Architecture decay).
I am not proposing skipping TDD, I think it is a very good approach. I just want to raise awareness of ways that it can fail, so that we make sure to navigate around them.
For all these points, the real answer is "you’re doing it wrong." TDD is supposed to, over the course of the project, allow faster development, allow easier and safer refactoring, and overall allow you to feel confident in the system you’ve built.My personal experiences have shown the following:
* If you’re not allowing yourself to write prototyping / exploratory code without tests, you’re doing it wrong.
* If the first thought when writing tests for a method / class / etc is "What do I need to mock", you’re doing it wrong (mock over-usage is the worst TDD mistake I’ve ever seen).
* If the tests take too long to run, you’re definitely doing something wrong.
I’ve run the full gamut of extreme white-box testing to full black-box testing. There are merits to both, and there’s a time and place for both, but anyone who adheres to strict ideals isn’t developing good software, they’re doing Agile masturbation.
That said, anyone writing production code without a test suite is making an even bigger mistake than doing any of the above. Building a good test suite is a very hard problem, and you really need to tailor what you build and how you test with your team and your product.
This comment was originally posted on Hacker News
When I write tests, they serve exactly to make a safety net so I can change my architecture and keep my behaviour. So I have to disagre with #4, although I understand that it might happen if the team are writing very low level unit tests for each methods and so on. To do something like this, IMO is the wrong way to go, you should write tests and develop from the outside to the inside.
When you do this, you specify the behaviour you want and with the tests ready you can play with lots of architectures. think about this: you build a thin facade with a clear interface and write the tests for that facade. then you can experiment with very different architectures inside this layer without the need to change any tests.
So I reinforce my point here: tests from the outside to the inside actualy help to change architecture.
Bruno, I agree that in general good TDD does provide a great safety net for architectural changes. Your approach is a great solution for the problem we mentioned. Hopefully these potential pitfalls will become a distant memory as more developers utilize TDD.
James, I agree with you point, but I don’t feel the answer is in just saying that "you’re doing it wrong". I think we need to figure out the main cases of such failing TDD instances, document it, and raise awareness so that we can be vigilant – if only to make sure that an inexperienced developer does not make these mistakes.
This comment was originally posted on Hacker News
Extreme Programming anticipated item 5 with the concept of doing a “spike.”
You just code something up to see if it works, and if not, you try something else. When you figure out which approach works, you throw it away and re-implement it using TDD.
Great point! Spikes can indeed be very useful in TDD. For others looking for more information check out http://www.extremeprogramming.org/rules/spike.html and http://www.c2.com/cgi/wiki?SpikeSolution
[...] Here’s a post (albeit dated) where a developer lists a few problems with test driven development. There are plenty more where that came from. What I’ve found works better is a hybrid approach, where we write tests at the same time as code (or just after). The idea behind pure TDD is one of those that sounds good on paper but is difficult to implement practically. Developing to the test means that we abandon some of the best parts of Agile by again tying our hands to strict requirements (this time the requirements are automated tests that don’t work until the code required is implemented). While I am a big supporter of functional automated tests and their inclusion in CI, I don’t think pure TDD is practical. A much better approach is to write functional code and tests together. [...]
Please continue discussion on the forum: link