Code Coverage with Test Cases?

Replies to This Discussion

Permalink Reply by Michel Kraaij on July 22, 2009 at 9:39pm

Isn't this just called requirements-based testing?

▶ Reply to This

Permalink Reply by Mark Crowther on July 22, 2009 at 11:33pm

Hiya Michel,

Yes, the last section where I describe how we might assess coverage by comparing stated requirements against the number of test cases we have is essentially RBT. I'd suggest that RBT is what testers have in mind when they talk about code coverage - which is not the same as requirements coverage and a common mistake that is made. Feel free to add the term ‘Test Coverage’ into that too, it means nothing out of context.

Sure, with RBT I'll get increased and even duplicated code coverage by virtue of writing test cases that cover requirements but the intention is not to get this increased code coverage, that's a by-product. The intention, the purpose of the tester in using RBT is to say we covered all the requirements with tests.

The punchline here is that code-coverage is one thing, requirements-coverage is another.

Mark.

▶ Reply to This

Permalink Reply by Alan Page on July 23, 2009 at 2:01am

Mark's punchinline is right on the money.

Covering requirements - or scenario coverage is beneficial. It's one way you can try to make sure you built the right thing.

Code coverage is beneficial for figuring out what you forgot to test - or determining what code the developers put into the product that isn't part of the requirements or core scenarios.

▶ Reply to This

Permalink Reply by Michel Kraaij on July 23, 2009 at 10:43am

"Code coverage is beneficial for figuring out what you forgot to test - or determining what code the developers put into the product that isn't part of the requirements or core scenarios.

Hmm, good point. So if we determine the complete code source. In addition we determine which code is covered by testing the requirements. And then subtract the last from the first. What is left is either code that isn't covered by requirements (e.g. requirements which is built, but not specified), unused code or code that also could be temporarily needed for fullfilling a requirement (e.g. storing of a temporarily value which is needed for calculations and is later on removed).

I do think this is a very interresting thought. In the light of behaviour, what kinds of behaviour could be determined after comparing the results? I can imagine performance issues (e.g. extra code, balast which is not needed, slowing down the process which is needed). Anything else?

▶ Reply to This

Permalink Reply by Pradeep Soundararajan on July 23, 2009 at 2:21pm

That's one of the coolest stories I have heard about Code coverage. Thanks Mark. My first serious learning about coverage was when I heard Michael B present it to me. He said "Coverage is the extent to which you can model the system" and I have been benefited by it by asking myself "Are there more ways I can model the system?"

I see RBT as the most and bare minimum approach that is widely practiced in the places that I have been in and heard of. Achieving a fair RBT itself is a challenge due to ambiguity and other factors.

Code coverage is one kind of modeling. Error message coverage is another. So are, time, data, operations, platform etc. Our actual coverage ( that can't be measured by a number ) depicts our ability to model and re model a system under test.

After experimenting, I have started to think about cross coverage. When I test for a timing coverage, I might have done some part of RBT, some part of data coverage and others although I can quantify only the timing coverage at that moment.

▶ Reply to This

Permalink Reply by Ravishankar Gopal on July 23, 2009 at 2:53pm

A Manual tester cannot determine from the Source code what part of the requirement he has tested and what is left out becouz he doesnot know what part of source code is performing which function
He is supposed to concentrate on Requirements coverage only
Based on Requirement specified in the document Testcases can be written and executed and its tracebility requirements can be maintained.Also different scenarios of testcases based on requirements can be done

▶ Reply to This

Permalink Reply by Pradeep Soundararajan on July 23, 2009 at 4:16pm

Ravishankar,

Thanks for sharing your views. Can you help me understand a couple of things:

a) What do you mean by manual tester?
b) Have you tried a code coverage tool?
c) Have you talked to testers who didn't have a requirements document or traceability matrix?
d) How will you test if there aren't any requirements?
e) What do you mean by test coverage?

▶ Reply to This

Permalink Reply by Ravishankar Gopal on July 24, 2009 at 12:36pm

a) Manual Tester does blackbox testing who is only aware of functionality to be tested required as part of the document Specification he doesnot have access to technical requirements specifications and the code which is implemented

b)I Have not tried code coverage tool

c)Testers who didnt hv requirements document or traceability matrix do adhoc testing and then create their own document of fucntional specification if client is asking for it

d)If ther is no documentation of requirements then i would hv one to one discussion with the business user about their requirement and also with developers about the approach of Functional Module i.e hw they will function and then do the testing accordingly

e)Testing of all the functional modules individually and testing in different scenarios

For Example
There may me Functional Modules A,B,C,D
These modules need not be tested in sequence like A after B
Then A may be tested by P
Then B may be tested by Q
Then C may be tested by R
Then D may be tested by T

Then as far as coverage is concerned
Then P may test A together with C in integerated fashion with data flowing one another
Then Q may test B together with D in integerated fashion with data flowing one another
Finally
R may test A,B,C,D together in an integerated fashion

Once all combination of integeration is tested and tester has used his brain to cover all the functionality to be tested in all possible ways which is covered in tracebility matrix is termed as 100% test coverage from a Black box Testers point of view

Correct me if iam wrong

▶ Reply to This

Permalink Reply by Pradeep Soundararajan on July 24, 2009 at 12:56pm

Ravishankar,

I am impressed about you taking time to reply to my questions in detail. I shall pay attention to your details, too.

Manual Tester does blackbox testing who is only aware of functionality to be tested required as part of the document Specification he doesnot have access to technical requirements specifications and the code which is implemented

Now, I am sure you would agree with me that idea you expressed about a manual tester is your own idea of what a manual tester is and what you have heard manual testers do. As I see you as a tester who is a little more serious than others, I want you to think of the words "Manual" and "Tester" independently.

Lets think of "Manual" - Whatever the dictionary says, we would agree to a human involvement as manual.

Lets think of "Tester" - We often refers to humans when we use the word "tester". For example, Pradeep and Ravishankar are testers. They are humans and they are testers.

So, everyone who claims to be a tester ( irrespective of manual or automated ) they are humans and anything that involves human is "Manual" approach.

The idea of Manual Tester and Automation tester is the confusion that industry is causing by giving delicious and cheese designations.

d)If ther is no documentation of requirements then i would hv one to one discussion with the business user about their requirement and also with developers about the approach of Functional Module i.e hw they will function and then do the testing accordingly

Can you list about 100 ways you would try solving the problem than the one you just mentioned above. When you just have one way to approach a problem and that fails it makes you feel stuck and lost. As a good tester you might want to practice coming out with tons of solutions so that given a problem you are practiced to think of many approaches.

Once all combination of integeration is tested and tester has used his brain to cover all the functionality to be tested in all possible ways which is covered in tracebility matrix is termed as 100% test coverage from a Black box Testers point of view

We will address this later but let's work on the above exercise. Fine?

▶ Reply to This

Permalink Reply by Mark Crowther on July 23, 2009 at 7:48pm

Hiya Pradeep,

The reply you gave feels like it echoes a bit of the post that I decided to delete. I went into a slight rant about RBT because on reflection it seemed to me simple RBT was next to useless. The idea of 'modelling the system' and 'cross-coverage' speaks to my concerns, thanks for sharing those.

Out of interest, here's what I decided not to post, I should learn to just put this stuff out there and know there are people on the same wavelength already ;]

"I have issues with the whole RBT and Requirements Coverage idea, I don’t even think it’s useful in practice. RBT is too simple in how it’s put forward as a way to drive ‘test coverage’ and so prove we tested enough. For example, does 100% coverage mean 1 requirment:1 test cases or do we need more? How many more? Of what type? However we measure coverage did we write the correct test cases? Have we considered risk and other factors and so discovered unstated requirements? Did we assess the meaning of the requirements and our tests to the user or just have in mind it’s functional aspects? The list goes on and the questions reveal a host of other practices and tester skills that have to be applied to the problem.

When we start thinking like this the wrote form of RBT quickly breaks down under the hammering and becomes a dangerous practice if depended on of itself. James Bach wrote about this 10 years ago so I’m not claiming to be breaking new ground in tester thinking by the way. My perhaps more militant stand against RBT (the misuse of terms and vocabulary for ‘coverage’ aside) is because the Behaviour Driven Testing approach I advocate more and more these days focuses on behaviour relevant to the user. Ticking off Requirements against test cases is of no use to the user. Answering the questions above and formulating testing around them AND the requirements is. "

Mark.

▶ Reply to This

Permalink Reply by Pradeep Soundararajan on July 28, 2009 at 6:17pm

Mark,

Thanks for not posting it earlier. It gave me an opportunity to get you post it.

I do exercises in my exploratory testing workshop that demonstrates that those who seem to care so much about RBT aren't actually caring about it. As Michael Bolton pointed out somewhere in Test Republic that those who profess so much about documentation themselves don't bother to read and write good documents.

A good RBT requires skills that people appear to be reluctant to build. By saying that I don't mean, "Yeah, I have built it". I have been trying to develop it as much as possible and constantly practicing it so that I am prepared for a war anytime.

My perhaps more militant stand against RBT (the misuse of terms and vocabulary for ‘coverage’ aside) is because the Behaviour Driven Testing approach I advocate more and more these days focuses on behaviour relevant to the user. Ticking off Requirements against test cases is of no use to the user. Answering the questions above and formulating testing around them AND the requirements is. "

Exactly. You would call it misuse and businessmen would call it fair-use. I am starting to realize that the scripted approach survives because there is more money out there for business people through that.

Mark, outsource work to me:

I would spend a couple of days analyzing your requirement document and bill you for X hours per person involved in my team.

I would spend a couple of days writing a test plan document ( and not refer to it ) and bill you for 2X hours per person involved in my team for preparing it.

I would spend a month or two writing test case document ( and refer only to it ) and bill you for 10X hours per person involved in my team for preparing it.

I would then again create a traceability matrix ( just to fool you and your boss about our coverage ) and bill you for 5X hours per person involved in my team for preparing it.

18X hours per person involved in the team. Assuming X is 50 hours and there are about 10 members in my team, that's 18 * 50 * 10 = 9000 hours of billing with no single bug found yet. If you are paying $20 an hour per person on an average, you would have actually given me a business of $1,80,000 without me or my team finding any bug yet.

Then comes the test case execution cycles and more billing. Why wouldn't a businessman be glad about the traditional approaches to test software?

Lets bother about the users of your product later during our maintenance billing phase ;-)

▶ Reply to This

Permalink Reply by Mark Crowther on August 1, 2009 at 4:43am

"9000 hours of billing with no single bug found yet."

That's the title of a new blog post on my blog (http://cyreath.blogspot.com). What you wrote was spot on and should form the basis of a clarion call to everyone in the profession.

The commercial reality we face is just that, a reality that we can’t escape but I hope we can generate amazing revenue while adding equally amazing value and avoiding fleecing our clients.

Too many consultancies have been sued because they didn’t (Government contracts? NHS contracts? MOD Contracts? Exchange Contracts?)– you know who you are.

Customer: "9000 hours of billing with no single bug found yet."?
Contractor: "Lets bother about the users of your product later "

▶ Reply to This

• ‹ Previous
• 1
• 2
• 3
• Next ›
• Page