RSS LJ

January 21, 2009

Exceptions vs. error returns, quantified ()

by fluffy at 4:05 PM
Today I got into a bit of a discussion regarding embedded programmers and their tendency to avoid exceptions in C++. The argument basically boils down to three things: code size, memory usage, and runtime. So, rather than continue on without actually putting my beliefs to the test (the discussion basically centered around whether engineers are making assumptions based on how things were 15+ years ago), I decided to construct a minimal test which I think shows where the tradeoffs may be.

Okay, so, a pretty simple test case is our friend, recursive Fibonacci. It's a pretty simple algorithm which would be pretty straightforward at demonstrating the issues. So, here are two versions, using checked trivial error returns and using exceptions. (The error condition in this case is whether the operation overflowed.) Both of them use the same mechanism for determining how large the stack has grown, and the error checking version is based on how it works in real code that's based on error checking.

(Note: I realize that the stack value isn't going to be exactly precise in terms of how much is used, but it is precise in showing how much it grows based on the depth of the call tree. That's all I care about here.)

(Also, to stave off any criticisms about the error-checking version, while yes, this could have used 0 to indicate error, that's meaningless in the context of most applications; 0 could be a valid result value. In real applications, you end up always using the errorType function(input, input, input, output&) method if you want to keep things consistent and sane.)

Code size

In terms of human-readable code size, the exceptions version is slightly shorter, with 62 lines instead of 65. That doesn't seem like a big difference, but keep in mind that you generally don't define a different exception type for every function.

I compiled both versions with gcc-g++ 4.3.2 and stripped the executables. The error-return version weighs in at 5720 bytes, while the exception version is 5812 bytes - a difference of 92 bytes. How much of that is due to generated code by the exception handling and how much is due to the inclusion of std::exception and creation of the OverflowException class is unknown. (However, if I remove all STL exception stuff and have it simply throw an int, the resulting binary is 5800 bytes, which tells me that the exception class only contributed 12 bytes. Since storing the string "Overflow" requires 12 bytes, clearly there's somewhat more interdependent behavior than that going on.)

Verdict: too close to call.

Stack usage

So, the next thing to compare would be the amount of stack used. I ran each version from n=1 to 30 using seq 1 30 | xargs -n 1 ./fib-1. The error return version grew the stack exactly 12 words (48 bytes) for each additional n. The exception version grew the stack exactly 12 words (48 bytes) for each additional n. The exception version also added an overhead of 7 words (28 bytes), presumably for the try/catch block.

Verdict: too close to call.

Run time

~$ time ./fib-1 38
Starting stack: 0xbff68b2c
Ending stack: 0xbff68440 (delta=-443)
f(38) = 63245986

real	0m2.091s
user	0m2.088s
sys	0m0.000s
~$ time ./fib-2 38
Starting stack: 0xbf9a3568
f(38) = 63245986
Ending stack: 0xbf9a2e60 (delta=-450)

real	0m1.866s
user	0m1.856s
sys	0m0.004s
In other words, the exception version ran 12% faster. As trivial as this example is, I think that's a pretty obvious margin.

Verdict: exceptions.

Conclusion

For code size and memory usage, it probably depends a lot on the complexity of the application, and a trivial application won't do a very good job at capturing the difference between the two. It requires further study with a less-trivial example.

For execution time, in the (hopefully common) case that errors don't occur, exceptions are faster. This makes sense; checked error returns require checking every result, whether it's good or bad, while exceptions only have to do anything special when an error occurs. Now, given a sufficiently-complicated example (with lots of different exception types with a complex inheritance tree, for example), the error condition could very well take a lot of time to handle compared to simply checking an error return. In an environment where you'd expect there to be a lot of error conditions to handle (transitory failures, network hiccups, etc.) then error returns could very well turn out to be faster. But it seems that it'd be better to just use error returns for the circumstances which call for it, and to use exceptions everywhere else.

There is, of course, one major missing component to the memory use issue: the static data structures which have to be set up to begin with when the first try block starts. I don't know of a quick, platform-independent way to measure that. I also don't know what the impact is on a per-exception-type basis. However, those are essentially one-time costs, shared across every use of the exception type.

So, basically, my conclusion is that the only way you can be sure of which one works better for a given application is to write the application both ways, and see which one works better. However, my suspicion is that the small amount of memory overhead for exceptions is far outweighed by the time savings, reduced code complexity, and guarantee that errors are not simply ignored (unless they are explicitly ignored, and even ignoring exceptions means code was written to do so — and just putting a try{}catch(...){} at the top level will still do a lot more to "handle" an error than letting a return code drop on the floor).

Also, a larger, more meaningful comparison would do a better job of determining whether the generated code size (which is an issue for embedded systems! executables load entirely into memory on MMU-less systems) grows larger with exceptions or with error returns. I suspect that error returns will grow faster — there's more code that has to go into more places, unless you absolutely need fine-grained error checking. If you're wrapping a single line of code in a try/catch block, that would certainly be a good situation to use error returns in. But if you're dealing with larger functional units where one of several things could go wrong and abort an entire process, exceptions are probably the way to go.

Anyway, the most important thing is to not simply assume things based on "common knowledge," but to actually test them. People who have been working in a field for a long time tend to accumulate Ways of Doing Things and collections of assumptions based on how things were when they started. I used to assume exceptions were bad because that's what older, more experienced programmers than myself told me. But change can be good, and just because someone isn't as experienced in a given field doesn't mean they don't understand the issues. A fresh pair of eyes is always a good thing.

Comments

#11687 01/21/2009 08:05 pm
Counterpoint: Raymond Chen argues that error returns are much easier to write properly, and much easier to determine whether someone else wrote properly, than exception-based code.
#11688 01/21/2009 09:55 pm
Sure, you can easily see whether error returns are used properly if you're looking at the code, but tracking down where they were used improperly is very difficult. It's a lot easier to just drop an error return on the floor than an exception. You have to go out of your way to fuck up an exception. (Also, Raymond Chen's examples are pretty terrible strawmen.)

Anyway, programmer habits are somewhat of a separate issue from this - every tool can be good or bad and has its uses and counterindications. What I'm railing against here is *embedded systems* people immediately insisting that exceptions be turned off because they are somehow inherently wasteful or slow. I see it time and time again on embedded platforms doing complicated things which would benefit from exceptions in terms of high-level error condition management.

They are a tool, and can be useful, and I see a lot of counter-exception religion for what amounts to "well when I was growing up, we didn't have exceptions." I also see similar religious arguments against virtual methods on occasion (mostly from newbie game programmers who have just heard that they were "bad" somehow). It's a wonder that to-the-wire systems people have at least decided that C++ isn't the devil anymore (although oddly enough I occasionally see ones who decide to completely skip C++ and do everything in Java, and of course I have to wonder about Palm's idea of dumping compiled code entirely and just using JavaScript for their application-level runtime).
#11690 01/22/2009 01:36 am
This discussion would not be complete without a reference to the FQA.
#11691 01/22/2009 08:09 am
Yet another categorical "I've been burned by misuses of a tool so let's just stay put in 1989" rant. Hooray.

Anyway, I'd like to reiterate that the point of my writeup wasn't about whether it's good or bad programming practice, just about whether the common beliefs of them being inefficient are true or not. When porting a C++ application to an embedded platform, one of the first things that almost always comes up is, "this needs to compile without exceptions." That's the part I'm trying to address, by showing that exceptions vs. equivalent if/else logic can be faster and uses only slightly more memory (and in this case, "slightly" actually means slightly, and it's possible that in real-life cases it uses less).

The FQA, Raymond's rants, etc. set up a false dichotomy, with the implicit assertion that you'd be using one or the other exclusively. Sometimes, exceptions make more sense. Sometimes, error returns make more sense. Exceptions are great for exceptional circumstances - things which occur rarely but which you need to handle when they do. They're not so great for things where you actually expect things to "error." Like, hitting the end of a file, or not finding an element in a map: those aren't errors, those are normal circumstances that you're specifically expecting. But running out of memory, or being passed a null pointer when a null pointer is meaningless, or a database file getting corrupted out from under you: those are good cases for exceptions.

I selected the Fibonacci example specifically because it demonstrates where to use exceptions: pretty much all of the time, you're not going to overflow, but once you do overflow you need to categorically abort right away.

Exceptions should be used in exceptional circumstances. They are a useful tool. They should not be disallowed just because of some deep-seated and provably-wrong belief that they're "bloated" and "slow."
#11692 01/22/2009 10:27 am
Okay, another clarification I need to make: in this context, the discussion is about embedded programmers coming into the strange new world of pocket-size systems which aren't "embedded" in the classical sense. For example, small devices with 128MB of RAM running Linux are called "embedded," and because of that they still bring the "embedded" mindset.

This gray area in terminology appears to be where much of the debate comes from.

Obviously if you only have 32KB of RAM, you're not going to want to have anything extraneous at all, but in that case you're probably not doing a LOT of things that C++ provides (in fact you're probably not even using C++). You're also going to be very explicit about everything and are going to be keeping a watchful eye on where exactly you need to be checking error returns.

Also, since these pseudo-embedded systems often don't provide the STL at all, I tried the comparison out again using an explicit compare/assign instead of std::min, and just throwing a const char * instead of a std::exception. The resulting stripped binaries were 5720 bytes for error returns and 5800 for exceptions - so a difference of 80 bytes between the two in terms of generated code.