Good automated testing procedures (for unit and integration testing) use two metrics to gauge the overall effectiveness. The first metric is the pass rate observed when running the tests. If only 20% of your tests pass, you know there’s a problem. But a 100% pass rate doesn’t mean that you’re in the clear either.

The second metric that’s needed to ensure effective testing is something called “code coverage”. A code coverage value tells you how much of your code is actually being tested. It actually tells you what lines of code were executed during a test suite and which lines weren’t. Pretty cool eh?

So, a 100% pass rate is nice. But it’s a poor indicator if your code coverage is only at 40%. This would indicate that 40% of your code does exactly what you want it to do (or at least what you’ve written tests for). But, 60% of your code isn’t even tested. So, your application could be 60% broken and you wouldn’t even know it.

The third factor that comes into play is the quality of your tests themselves. Even if you have 100% code coverage and a 100% pass rate, there may still be use cases or user stories that you haven’t written tests to test.

As you can see, it’s a real exercise in rigor to write an effective, comprehensive test suite. In the last analysis though, it’s the only way to prove, not just be able to say, that your software works properly.

Tools are available to enable you to test effectively in all of the more common languages used today. PHP, Python, Ruby, Java, .NET, PERL and even Javascript. A quick Google will find you what you need to get started.

So, if the quality of your software matters, and it’s important to you to find bugs before your customers do (as it should always be and certainly is for us) then we encourage you to start thinking about the quality of your testing process.

We encourage you to tell us about your own experiences with automated software testing. Tell us what tools worked well for you and which didn’t. Tell us what was most valuable about having good tests. Tell us about how responsive your developers were to fixing broken tests. And any other information on what made your testing processes succeed and/or fail is very welcome.

Happy testing.