Unit testing means quick debugging

There's been another flock of unit testing posts released to stalk the blogosphere recently and I've not found any that mention one of my top reasons for unit testing - quick debugging.

I release code that's as buggy as the next developers, well maybe not that bad, but there has been the odd occasion where my code has had scope for further stability enhancements to be proactively scheduled for imminent release. When one of these rare events occurs, my first goal is to get the problem happening repeatedly and reproducibly in a debug environment. Until you can do that you often don't really know if it is indeed a bug and certainly its hard to see the breadth and depth of its impact. Does it happen for every customer? Or only those trying to purchase 1000 or more units?

Its at this stage that I find my test suite really earns its keep. I might not (initially) care why some system call is failing on some customer machines - all I really care about right now is how my application handles the error. If I've got a tests that mock out the problem subsystem I can quickly add a new test case which has the (mock) subsystem throwing the appropriate error and see how the application logic handles this case. Given I'm looking at a bug ticket describing this very case the odds are good that the application doesn't handle it properly at all. So this is the first thing to fix. I'll decide what the appropriate action is at the application level, to retry or to ignore and log or whatever and can easily add this into my new test as a closing expectation.

Then I can move down a level to get closer to the specific problem at hand. If the faulty subsystem has a unit test suite then I'll load that up and can write new tests to quickly experiment with scenarios that may have caused the error until ultimately we've isolated the original root fault e.g. a failed syscall.

The advantage of this approach is two-fold:

One is in the speed with which you can fix bugs. It is often the case that the limiting factor in bug fixing is the speed with which you can reproduce and then understand the logic and state of the program at the time of failure. In complex systems without unit tests the only way to get to a known system state can be an elaborate process of insert fake data through some front-end, reusing old test accounts on a dummy database and hoping the left-over cruft doesn't affect today's issue etc. If the bug involve any writing to persistent storage you may well have to go through a manual clean up after each experiment. Its not unusual for a typical day bug fixing in such an environment to be dominated by this process.

In my experience reducing the time between idea and executing code is vital. This means optimising ease of actual code input (mostly competently handled by IDEs these days) or optimising compile and linking times but the biggest delay in this feedback loop and the easiest to affect is optimising the time between having a working executable or library and having the new code executing. And 9/10 the best way to do this is with a unittest framework.

The second big benefit is as a kind of 'five whys' analysis. You start off a bug fix at the most general application layer, to resolve why the particular subsystem failure was allowed to manifest as a problem serious enough for a user to actually report it to you. At this stage the specific error hardly matters and your fixes here are addressing a much broader class of errors. More importantly, any changes you make out here don't affect the lower levels of the application and so its easy to apply the process recursively until you've bottomed out at the specific root cause. Without unit testing, you start bug fixing the other way round, from most specific cause outwards - and once the specific issue is dealt with it becomes much harder to reproduce the effects on the upper levels of the application because you've fixed the original tst case and you've not got a test rig that can manipulate them in isolation.

To be honest, I'd build a test rig for these benefits alone. Faster turnaround for problems means more code gets out each milestone and testing down through the layers makes your application progressively more and more robust.

Functional programming

I agree, and I think it's evidenced by the fact that I find unit tests a lot less of an issue when I'm using a Lisp-like language. Lispy languages (clojure is my current favourite) have two advantages in this respect: lack of side-effects and the REPL.

The benefit of the REPL is obvious: you can call into any function you like, right from the command line. Lack of side-effects is equally useful, because it means that no reproduction case is ever more than one line: there's no state to set up / tear down around each test.

I found it really interesting when I was implementing the Graham scan algorithm in clojure that the REPL still works when the output is a GUI, not plain text. In fact, clojure's integration with Swing made it practical to knock up a GUI as a disposable test tool. This sort of interactive-GUI testing may be most appropriate when it's easy to judge the results by eye, but hard to write automated verification code.

Maybe the generalisation of your point is that as developers we should write manual-control rigs alongside each module that allow that module to be manipulated as flexibly as possible. The classic xUnit unit-test suite is one example of such a rig, but perhaps not the only way it can be done.