For What It’s Worth
Often I am asked: How much does a bug cost? I would love to say $1023.76. Each and every time. Consistently. Use this as your metrics for calculating risks and you will be fine.
Well, the truth is costs are somewhere between nothing and your business. Now, let me explain why.
Let’s start with the upper extreme. Assume you have a really, really critical software bug. Something that only happens once in a million. Unfortunately, statistics are deceiving and there seems to be a reciprocal relationship between the likelihood of a severe bug occurring in your brand new rocket scientist (aka rock star developer) designed and tested missile guiding system and the fact that it actually occurs. Call it bad luck, Murphy’s Law or simply time pressure. More often than not, when putting complex systems together, something goes wrong … most likely where you do not expect it. So if you hear a loud bang at the wrong time … bad, very bad. The echo might well be the slamming sound of your business hitting rock bottom.
Why did this happen? Everything had been well tested, all functional requirements were met, perfect design. As so often the devil is in the detail. While traditional testing is an excellent way to check for functional correctness, only so many cases can be covered in any reasonable amount of time. Moreover, functional testing checks for some given input data if it results in some expected outcome. You check it for obvious critical values such as the input of 0, maybe -1 and 1, -100 and 100 and so on. Funnily, for 68 it doesn’t work.
This is pretty mundane and unexciting you think. What is the worse that can happen? Worse case your application crashes unexpectedly or you have a security hole big enough for someone else taking over your application and in turn your business.
Mundane software bugs such as buffer overflows, null pointer dereferences or memory leaks, however, like to manifest themselves by odd combinations that cannot realistically be found with traditional testing techniques. Code reviews help a lot, so do run-time verification techniques. Ahh, simple, right? Well, if you only have some hundreds of lines of code, for sure, that is a good solution. Don’t get me wrong, I recommend these even if you have a zillion lines of code, but typically that doesn’t really cut it. You have to have herds of people looking at the code or spend months of run-time analysis to detect those bugs. Possible to do, but it comes neither cheap nor reliably.
Now, after all this scaring news there is good news, too: There are many, many applications out there where the worst case does not occur. Maybe you need an update here and there, but overall it does not cost you millions or even kill you business. In fact, no one has discovered the subtle flaws in your software … yet. You are taking chances, but that is all right. Because, what is the worse that can happen?
I will tell you later.