Home > Other > Why it doesn’t pay to write unit tests

Why it doesn’t pay to write unit tests

I’ve unfortunately had to do a lot of traveling in the last couple days and I’ve been reading No one makes you shop at Wal-Mart on the plane. Among other things, the book describes the economic model underlying  the idea of a ‘public good’.

A public good, as opposed to a private good, is basically a resource that is enjoyed freely by a group. A clean environment or a quiet neighborhood is a good example of this.

A private good, in this example, might be the right to play your music as hard as you want to. However, if everyone starts doing that, the public good of a quiet neighborhood will soon disappear. In this scenario, everyone in the neighborhood has to pay the cost of setting a limit on how hard and when you can play your music in order to preserve the public good of a quiet neighborhood. This idea is related to the Prisoner’s Dilemma, for those of you curious about that.

With regards to software development, a set of well-running unit tests is a public good while the act of writing a  unit test is actually a private cost. I think this is self-explanatory to most of the developers reading this,  but I’ll explain it just to be sure.

Writing a unit test that is of good quality is not advantageous to a developer writing or modifying code for the following reasons:

The productivity of a developer is measured based on 2 things:

  1. The number and quality of features the she produces
  2. The number of bugs she fixes

The first measurement, the number of features produced, is weighted more heavily: that is to say that creating a feature is, in general, seen as more productive than fixing a bug.

However, writing a good unit test does not directly contribute to either the creation of features or bug fixes.

While writing a unit test might be helpful when creating or modifying a feature, it’s not necessary. Every decent developer out there is perfectly capable of writing software without having to write a unit test. In fact, writing a unit test costs time which a developer might otherwise have spent on writing more features or fixing bugs! The better the unit test is, the more time that a developer will have needed to spend on it, making good unit tests more “expensive” than lower quality unit tests.

Thus, from an individual developer’s perspective, it does not pay to write good unit tests, especially in the short term.

Furthermore, the unfortunate thing about “quality”, is that the quality of a feature (or any piece of code) is something that can only be measured in relation to how long the code has existed. In other words, the quality of code is never immediately apparent and frequently only apparent after a significant period of time and use. Often, by the time the (lesser) quality of the code becomes apparent, no one can remember or determine exacty who created the feature and it’s not productive to search for that information.

But it does always pays to have a high quality suite of unit tests. A well-written suite of unit tests does 3 things:

  • Most importantly, it will alert the developer to any problems caused by new or changed code.
  • Because an existing unit test will obviously use the existing API, it will alert the developer to problems with backwards compatibility if the developer changes the existing API.
  • Lastly, unit tests are functional examples of code use: they document how existing code should and can be used.

All of these benefits help a developer to write better quality features (in less time) and help not only with fixing bugs, but also with preventing bugs!

But, in a situation in which there are no external pressures in how a developer writes his or her code, there are no immediate reasons for a developer to write unit tests. This is especially true in situations in which the developer will only be working on a project for a relatively short period of time.

Of course, some developers might feel that writing tests helped them develop features more quickly — or that it might help them fix bugs more quickly. However, if at a certain point they have to justify their use of time to a superior (the project lead, project manager, etc.) and they explain that they were writing unit tests instead of writing new features or fixing bugs, they will get in trouble, especially if there’s less value placed on unit tests or refactoring.

At a company, obviously, this is where a project manager, project lead or even a CTO comes in. While it may be in the interest of the developer to create new features and fix bugs as quickly as possible, it’s probably equally important to the CTO and other managers that the quality of the software created meets certain standards. Otherwise, users of the software might become so disgruntled with the software that they’ll complain, leading to a negative reputation of the software, which may lead to the company going bankrupt!

It’s in the interests of the CTO and other managers to require software developers to write unit tests that are of a certain level of quality: namely, the unit tests should be good enough to assure that the software retains a positive reputation among it’s customer base. This is often a difficult limit to quantify but luckily often easier to qualify, I think.

Open source software is, however, a different story. There is no CTO and the highest authority in an open source project is often the lead of the project.

The question, then, is what determines the quality of a suite of unit tests in an open source project? To a large degree, the answer is obviously the attitude of the lead of the project. Attitude is often very hard to measure, unfortunately: it’s easy enough to say one thing and do another. If you ask the lead what she thinks, it’s hard to say if the answer she gives you represents the attitude that she communicates to the rest of her project.

Realistically, one of the most decisive factors determining the quality of the features of an open source project is simply the example set by the lead in the code and tests that he or she writes.

Categories: Other
  1. 15 February 2013 at 22:14

    “Why it doesnt pay to write unit tests Code Snippets” was indeed a splendid blog post, cannot help but wait to look at alot more of your posts.

    Time to waste numerous time on the net lolz.

    I appreciate it -Jed

  2. 2 March 2013 at 18:07

    My company solved this issue in a very interesting way: We integrated the QA department with the engineering teams which forced the engineers to do their own QA. This also forced the engineers to manually regression test all of their systems before each deployment. This process took an engineer from each team for a full month leading up to deployment when it was first implemented, but over time the engineers automated all of those regression tests and started writing code that didn’t introduce MORE manual tests. The incentive structure changed when engineers stopped throwing code “over the wall” to QA and started having to do that manual work themselves.

  3. 5 June 2013 at 20:01

    If writing unit tests reduces the bugs I have to fix, then I have more time to write new features. So writing good unit tests would improve my overall productivity. I’m not sure why that wouldn’t come up if I had to defend my time at any given point in a project. But then again, my assumption is that writing unit tests is part of development work. If you as a developer or your managers are estimating project time and excluding the time for certain project work (unit tests), then you will always have a problem.

  4. 10 August 2013 at 06:03

    There are a number of benefits of unit testing and TDD that are not covered by this post. First, writing your tests first drives software design. You are forced to think about separations of concerns, you almost certainly will avoid bad practices such as global variables, singletons and such. Another huge benefit is that a comprehensive set of unit tests gives the developers confidence to refactor the code without being afraid to brake it.

    Test driven code tends to be easier to read, easier to change, have fewer bugs, have fewer lines of production code that can break. I simply cannot agree that it doesn’t pay to write unit tests.

  5. Pete
    19 August 2013 at 12:03

    Very interesting post but I suggest that if you have a project manager, on a development project, who doesn’t view fixing bugs as productive work and measures productivity only by the number of features that are Done you have significant incentive to write good unit tests.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: