Community of Software Testing Professionals
Is it possible to test a testing?
...and how to do it if it is?
I've already thought about it for some time, and today I've occasionally read a Qualificiation Barrier post, where Rob Lambert wrote in the comments something like: testing is too emotional, subjective and context based to be testable (sorry, Rob, if I misunderstood your idea).
It looked like a challenge for me: what a tester I am if I don't even try to test what is supposed to be not testable? :-)
Ok, anyone who is in test area for at least few months most likely knows that testing is a very depending process. It depends on context: context of the project, of the environment available, of the developers' speed and qualification, of testcases prepared... and it also greatly depends on tester itself. From some point of view the testing is something like an art. That's one of the reasons why I like it.
Can we test, estimate and so on such a depending process?
My point of view is: everything can be tested.
All below is not a finished idea, it is more like a trying to get that idea.
First of all, the testing is an engineering thing. It means that it has a purpose. Testing itself is obviously not a purpose no matter how we love it and how good it is. To get the purpose, we need to identify a stakeholder. Most commonly (in my practice) a stakeholder is someone who is ready to give us money for doing something for him. I.e. a Customer. Ok, Customer have a purpose - in the other case he wouldn't spend his money on us. What does he want? Looks like he wants us to develop some software. Yes? Don't think so. He doesn't need a software, he needs a problem to be solved by that magical software. That was already told maaaany times before me: software, that doesn't solve the problem, costs nothing.
So, Customer has a problem, and our team (all those, responding for the project) has a task to solve that problem as good as we can.
If we doesn't solve the problem - we won't get paid.
If we solve the problem too slowly - we won't get paid.
If we solve the problem in a way which is too expensive for Customer - we won't get paid.
Our goal is to give Customer a software which would effectively solve his problems, to do it in time and for reasonable money.
The main goal of testing team then can be defined as: to make sure that the customer's problem is solved by our product. And then there is a set of additional goals like creating support documentation, making developers to increase a project's quality, searching for features that can make a product more attractive for Customer and so on.
We cannot test a testing process from the point of view like: how many bugs were found, or: how many hours did we spend on testing, or: how many test scenarios did we write and run. We can test forever and find more and more bugs, also we cannot find all bugs - so it looks meaningless to test a testing by such parameters.
But we can at least use this point of view to tell how good a testing process is:
If a product was released to Customer and the Customer failed acceptance tests because his problem is not solved by the product - testing didn't cost its efforts (the hardest part here is, IMHO, to identify the real Customer's problem - they rarely tell it themselves).
If a product were released with minimum of bugs and in time, and has solved the problem, but testing overrun a budget significantly - testing is far from optimal, it needs to be prioritized and restructured (or just be learned to stop at correct point).
If a product was released to Customer without problems, but there was a blocking bug in the main functionality found occasionally just before the release (and fixed in a rush) - testing process certainly has problems (see above).
If customer is satisfied with the product and there were no activated by testing team's fault significant risks (like missed bugs in main functionality) - well, a testing process in a company looks like good working. Always can be better, though. :-)
Hmm... I've wrote all this and now I'm thinking if this is a correct way of thinking. Not sure if it will not be changed in future. :-)
Any ideas, objections, questions, whoever will read this?
I've already thought about it for some time, and today I've occasionally read a Qualificiation Barrier post, where Rob Lambert wrote in the comments something like: testing is too emotional, subjective and context based to be testable (sorry, Rob, if I misunderstood your idea).
It looked like a challenge for me: what a tester I am if I don't even try to test what is supposed to be not testable? :-)
Ok, anyone who is in test area for at least few months most likely knows that testing is a very depending process. It depends on context: context of the project, of the environment available, of the developers' speed and qualification, of testcases prepared... and it also greatly depends on tester itself. From some point of view the testing is something like an art. That's one of the reasons why I like it.
Can we test, estimate and so on such a depending process?
My point of view is: everything can be tested.
All below is not a finished idea, it is more like a trying to get that idea.
First of all, the testing is an engineering thing. It means that it has a purpose. Testing itself is obviously not a purpose no matter how we love it and how good it is. To get the purpose, we need to identify a stakeholder. Most commonly (in my practice) a stakeholder is someone who is ready to give us money for doing something for him. I.e. a Customer. Ok, Customer have a purpose - in the other case he wouldn't spend his money on us. What does he want? Looks like he wants us to develop some software. Yes? Don't think so. He doesn't need a software, he needs a problem to be solved by that magical software. That was already told maaaany times before me: software, that doesn't solve the problem, costs nothing.
So, Customer has a problem, and our team (all those, responding for the project) has a task to solve that problem as good as we can.
If we doesn't solve the problem - we won't get paid.
If we solve the problem too slowly - we won't get paid.
If we solve the problem in a way which is too expensive for Customer - we won't get paid.
Our goal is to give Customer a software which would effectively solve his problems, to do it in time and for reasonable money.
The main goal of testing team then can be defined as: to make sure that the customer's problem is solved by our product. And then there is a set of additional goals like creating support documentation, making developers to increase a project's quality, searching for features that can make a product more attractive for Customer and so on.
We cannot test a testing process from the point of view like: how many bugs were found, or: how many hours did we spend on testing, or: how many test scenarios did we write and run. We can test forever and find more and more bugs, also we cannot find all bugs - so it looks meaningless to test a testing by such parameters.
But we can at least use this point of view to tell how good a testing process is:
If a product was released to Customer and the Customer failed acceptance tests because his problem is not solved by the product - testing didn't cost its efforts (the hardest part here is, IMHO, to identify the real Customer's problem - they rarely tell it themselves).
If a product were released with minimum of bugs and in time, and has solved the problem, but testing overrun a budget significantly - testing is far from optimal, it needs to be prioritized and restructured (or just be learned to stop at correct point).
If a product was released to Customer without problems, but there was a blocking bug in the main functionality found occasionally just before the release (and fixed in a rush) - testing process certainly has problems (see above).
If customer is satisfied with the product and there were no activated by testing team's fault significant risks (like missed bugs in main functionality) - well, a testing process in a company looks like good working. Always can be better, though. :-)
Hmm... I've wrote all this and now I'm thinking if this is a correct way of thinking. Not sure if it will not be changed in future. :-)
Any ideas, objections, questions, whoever will read this?
Tags: ideas
Comment
-
Comment by Victoria Kuznetsova on February 14, 2010 at 5:30pm -
@Deepak Punyani
>I am strongly agree with you that before initiating testing it should be clear in each and every personal mind the idea >of time and cost.
I did not say this - actually I just supposed that you meant this in your previous comment. So you are strongly agreed with yourself here. :-)
I think that keeping in mind money/cost things is a job of a test manager. Or (if there is no manager) it should be settled on the stage of planning a test process. I do not think that tester should continuously keep in mind whether he will be paid for tests he is running or not. He should be able to test freely - the only thing that borders testing is understending WHY, WHAT and FOR WHICH AIM we are testing the program right now.
About satisfying a customer please see above in the original post: "Our goal is to give Customer a software which would effectively solve his problems, to do it in time and for reasonable money". Looks like it is the same you wrote?)
-
Comment by Deepak punyani on February 13, 2010 at 12:04pm -
victoria,
First of all thanks for replying.
I am strongly agree with you that before initiating testing it should be clear in each and every personal mind the idea of time and cost.
But there is one glitch in the above written statement What happens most of the time due to time constraint or thinking this thing in mind that client ,is not giving as much as our expectations so ,we have to do okay type of work and we are missing the things on this point the person who are supporting this statement are totally incorrect.
The aim of tester is to provide the tailor made good to there client which satisfy the need of the person.Never apply shortcuts to deliver your work. It is truly said that if your one client is satisfy with your work it automatically generate 10 others but if one is dissatisfy it automatically cut down your business.
And i am not saying that this is only one parameter for estimating quality of a testing process on which we have to focus on.
Please Suggest/Comment.
Thanks and Regards
Deepak Punyani
-
Comment by Victoria Kuznetsova on February 12, 2010 at 1:00pm
-
@ Deepak punyani
I'm not sure I have understood your comment correctly. Do you mean that one of parameters for estimating quality of a testing process is: "do members of testing team understand how much time they have and how they will be paid for the work"?
Or do you just say that it is good when testers do know it (and no relation to testing process' quality)?
-
Comment by Deepak punyani on February 10, 2010 at 11:24am
-
Is it possible to test a testing?
I agree with you but to add up I would say that the concept reasonability should be clear to the testing team. Reasonability means a fair idea of time and cost. It would also enhance a tester's ability to prioritize the work....
-
Comment by Deepak punyani on February 10, 2010 at 10:42am
- I really appreciate your thought process. Keep it up!!!!!!!1
-
Comment by Victoria Kuznetsova on May 15, 2009 at 9:44am
-
@ Pradeep Soundararajan
"Testing helps in providing information about the quality of the product that may be of help in gaining more confidence that the product is likely to do what the customers want it to do provided the customer's requirement hasn't changed over that period of time"
Oh... ok, I think, I got it now. But it seems too long, and it is not a requirement, it is more like describing (I agree with this describing though). Anurag Gupta asked for requirements against which it would be possible to test testing, so it was a quick guess from my side. Actually I don't think, testing needs formal requirements to test it - we just need to have in mind its goal and continuously think about why we are doing what we are doing and what it will give us (and customer). Risks, priorities and so on.
So testing based on the budget planned could be a bad idea in some contexts because those who did the budget had little clue about testing.
Some context always exists... but if we cannot change the budget and will not consider it while testing - we can do a lot of great work nobody pay for... I mean - maybe, customer doesn't need great quality this time, maybe he just wants a program to work and to be cheap. Then we will need to focus only on main functionality and let the other parts be buggy if there are workarounds for those bugs.
In building an aero plane the budget of the testing has to cross the cost of the aeroplane because we would need at least more than one aeroplane for tests.
I will tell it to my boss when he will reduce a testing budget next time, if you don't mind... :-)
I am concerned about thought process than the process that you might be thinking.
The better tester thinks, the better testing he do? No objections here.)
I was trying to help Anurag to understand that defect leakage is always possible no matter how many expert testers test a product. Nothing to do with whether development is great or testing is great.
Then I really made a mistake, sorry... by the way, isn't it obvious, that defect leakage always exists? If only customer understands that...)
You did fine and I like what you did with this whole post.
Glad to hear it, thanks a lot!
-
Comment by Pradeep Soundararajan on May 14, 2009 at 1:23pm -
@ Kuznetsova,
Nope, but where in the "Testing should give a confidence that tested software do what Customer wants it to do" you see anything about 100% sure? Sorry, if I used English incorrectly... By 'confidence' I didn't mean 'software absolutely perfectly solve Customer's problem'. It is more about a result: if the software doesn't solve the problem some way it is useless, imho.
It appears to me as though testing gives confidence. What does development do? What does design do?
Does the confidence occur only when the test cycle has completed or there is some confidence even before the test team takes up the product for test?
If you agree that there is some confidence even before the test team takes up a build for testing then you might want to consider re-phrasing something like, "Testing helps in providing information about the quality of the product that may be of help in gaining more confidence that the product is likely to do what the customers want it to do provided the customer's requirement hasn't changed over that period of time"
Quoting what Anurag wrote "2. Testing should be performed in time and in budget that was planned"
for which I replied And if the budget is 2 rupees. What testing would you do? to which you said The testing which I can do in the time paid by 2 rupees. If it will be meaningless - then I won't test at all until prove, that I need more time (and so - more money).
So testing based on the budget planned could be a bad idea in some contexts because those who did the budget had little clue about testing.
What would you do?
I would test the budget and the analysis that went through to be able to convince myself if I can do my tests within the budget. In building an aero plane the budget of the testing has to cross the cost of the aeroplane because we would need at least more than one aeroplane for tests.
Again, nobody talked about CMM... And I agree that without testers improving themselves there is no way to significantly improve the process. Though improving the process first will most likely make testers to improve too (and hopefully this will continue). :-)
I am concerned about thought process than the process that you might be thinking.
Well, it is clear that you have a mature point of view for testing and all your answers have a root in that point of view. Also the answers looks sarcastic in a manner 'testing is great and developers are doing bugs instead of software' (though I may be wrong here). But what is is goal of pure analyzing your answers? Analyzing a software requirements and Customer's expectation have a clear goal: to solve a problem and to get a money for it.
I was trying to help Anurag to understand that defect leakage is always possible no matter how many expert testers test a product. Nothing to do with whether development is great or testing is great.
You analyzed my answers and helped me understand your perspective of what I wrote. That's valuable to me and maybe you ( now that I have replied ). If Anurag Gupta analyzes my answers, he may offer challenges to me ( just like how you did ) and that can be of help to both of us.
You did fine and I like what you did with this whole post.
-
Comment by Victoria Kuznetsova on May 13, 2009 at 10:38pm
-
2 Pradeep Soundararajan
You have highlighted my words, so I will also answer (though I'm not Anurag Gupta)...
>By testing your blood sample, can I say your blood sample is free of any bacteria and virus?
Nope, but where in the "Testing should give a confidence that tested software do what Customer wants it to do" you see anything about 100% sure? Sorry, if I used English incorrectly... By 'confidence' I didn't mean 'software absolutely perfectly solve Customer's problem'. It is more about a result: if the software doesn't solve the problem some way it is useless, imho.
>A test reveals information and doesn't auto fix problems.
Again, nobody said about fixing. There are different ways to get the result. If we work with requirements, then with developers - to prevent bugs, - and then we test to find bugs according to risks and having a main problem in the mind - then we do give as a result some confidence: yes, this software do what it is supposed to do.
Sorry for many words used.
>And if the budget is 2 rupees. What testing would you do?
The testing which I can do in the time paid by 2 rupees. If it will be meaningless - then I won't test at all until prove, that I need more time (and so - more money).
What would you do?
>The only way testing can improve is when testers improve themselves and not by any process like CMM, ISO or Six Sigma
Again, nobody talked about CMM... And I agree that without testers improving themselves there is no way to significantly improve the process. Though improving the process first will most likely make testers to improve too (and hopefully this will continue). :-)
>Fantastic, please analyze my replies.
Well, it is clear that you have a mature point of view for testing and all your answers have a root in that point of view. Also the answers looks sarcastic in a manner 'testing is great and developers are doing bugs instead of software' (though I may be wrong here). But what is is goal of pure analyzing your answers? Analyzing a software requirements and Customer's expectation have a clear goal: to solve a problem and to get a money for it.
-
Comment by Pradeep Soundararajan on May 13, 2009 at 10:20pm
-
@ Anurag Gupta,
1. Testing should give a confidence that tested software do what Customer wants it to do;
By testing your blood sample, can I say your blood sample is free of any bacteria and virus?
A test reveals information and doesn't auto fix problems.
2. Testing should be performed in time and in budget that was planned
And if the budget is 2 rupees. What testing would you do?
3. Testing should improve all the time
The only way testing can improve is when testers improve themselves and not by any process like CMM, ISO or Six Sigma
Testing as a measurable output. Defect Leakage is a great metric for that.
Defect Leakage and not defect introduction? So people in roles of requirements, design and development add a million bugs together and few testers need to find all of them, right?
c) Improve testing by adding more brain power. A tester has to be very analytical and creative to keep on improving
Fantastic, please analyze my replies.
-
Comment by Victoria Kuznetsova on May 13, 2009 at 10:17pm
-
Oh, Mr. Wolf... but isn't he more a problem eliminator ('no man - no problem') than a problem solver? Don't remember him well, though... The movie is great, yes. :-)
MCI WorldCom? And how did they test testing? :-) For me it's just like an interesting puzzle, but having a special team is something... hmm, cannot think why did they need it.
>a) Testing as a measurable output. Defect Leakage is a great metric for that.
Sorry, I'm not quite ready to talk about metrics... never used them as something that matters. It looks logical to measure what we want to test, but there will be so many "if"s (because testing is so depending on many things process) that I just cannot think of metric to measure a quality of testing process. Maybe, except of metric 'number of satisfied customers/total number of customers'. :-)
By the way, please correct me if I am wrong, but does your 'in order to achieve that' mean you agree with such set of requirements for testing process - and with the testing process testablity?
© 2010 Created by EDISTA.
You need to be a member of Test Republic to add comments!
Join Test Republic