Test Coverage: Delivering the Five-Finger Death-Punch (to Bugs)
Have you ever seen the Tarantino movie "Kill Bill"? In it, the lead character Beatrix, played by Uma Thurman, learns of a legendary, secret martial arts technique that disables the opponent through five strategic punches. Wouldn’t it be great to be able to remove unwanted things during test coverage with just five easy punches?
I have recently been struggling with test coverage. Not that this is something new; test coverage is always an issue. However, every time this issue arises, I realise that I can’t properly answer what good test coverage really means.
In testing, we rarely talk about "100% test coverage", unless when unit-testing small and simple modules. There are many reasons for that. The cost is usually too high. And then there is still no guarantee that your software will be 100% error free.
From experience, and blog posts and articles on the subject, it seems to be commonly accepted that 80% test coverage is sufficient (90% or higher for critical systems). But 80% of what? Test coverage is measured on many things, such as:
- (Executed) Code lines
- Input/output combinations
- Functions and features
- Decisions paths
- Use cases
- Customer scenarios
- Risk areas
I think we can agree that you’ll need testing within several of these areas, some with high coverage, some with less. But not necessarily 100% (and never zero).
I’m still searching for the secret five punches. However, on my way, I have made some experiences that might be helpful to others who search for the same.
Get the basics covered: In most cases, the system you’re testing is supposed to serve a purpose. Functional requirements, features, functions – all the stuff that adds value. And if it doesn’t work, well to quote Metallica, then nothing else matters.
Pursue cases that are easy to identify, and easy to verify: If you can't measure it, you can't manage it. If test coverage is necessary, then you need something to measure. Don’t launch tests without a clearly defined purpose or expected result. You could and should tick off these tests early on. Done.
Structure your test-scenarios and prioritise risk: Use matrixes to crosscheck all actions with all scenarios. Focus on the areas of higher risks. Time is always crucial. Remember, testing is also to be aware of what you don't test.
Atomize your test cases: Avoid complex test cases with blurry expectations or several alternative results. Keep it simple, with few steps and clearly expected results.
Automate your tests where possible: There is a certain assurance in big numbers. Automating your tests enables you to run thousands of tests you would never have time to go through manually. We did that for a large portal solution. The scripts blasted through 18 000 tests every night, using up to 6 hours. Estimated time for manual testing would be three weeks continuously testing every second, 24 hours a day, seven days a week.
Test coverage is not guessing a number. It needs anchoring to and alignment with an overlying quality strategy with clearly defined goals. Pure math won't do either. Rather than 100% test coverage, we should talk about sufficient test coverage in relation to time, resources, criticality and expected quality.
There's no magic, no fancy tricks. If you punch the right places hard and long enough quality will improve. It's as simple as that.