Well, the discussion arise about how independent is av comparatives?
Do they get paid by the av companies?
Seems they’re paid for testing the product?
They’re paid to publish the results?
If the results are good ones, well, do you need to pay more?
They’re paid to release/share the samples that were tested?
as stated in the methodology document of 2008, vendors pay for the services.
For the public main test series, every vendor that applied for inclusion (if he gets included) pays the same fee, in order that we can cover the costs of the tests etc. Of course, it does not have any influence on the results. Comodo never applied for the public main test series, as in that case results are being published in any case.
Instead, Comodo paid us several times for internal/confidential single product detection rate tests. As they are confidential, we can not publish them if Comodo does not want to. Comodo decided to keep all tests they commissioned to remain internal. If they would have wanted to use them in marketing (i.e. use the logo etc.) and to have it published together with a review, they would have had to pay additionally.
There are many wrong perceived things written (in a misleading way) by Comodo, so we may reply soon to clarify at least some of them.
Thanks for joying IBK.
Well, seems the financial relationship with av companies is now more clear.
But, in my opinion, companies should NOT pay to be reviewed in main tests. Otherwise, I can’t really think on independence or full trustfulness. Well, the results won’t be complete as some av products weren’t tested as they haven’t paid for it. Am I wrong?
Thanks for your reply IBK,
I’ve always been told that you are an independent lab and aren’t paid by companies that make they product.
Your statement sort of shoots wholes into my prior belief.
How do I really know how well any AV scores if the results depend on you receiving payment ???
Due to the complexity of tests and the work behind it, there is no testing organization which could live from nothing or user donations (I tried that at the beginning). I think that test results should be made available for free to the public; users should not pay for the external QA etc. of AV vendors.
We have a waiting list of vendors which would like to get included in our tests, but we test only ~20 products in the public main tests. As all the ones which get included (if they qualify) pay the same fee (even if they took the given opt-out option for the retrospective test), I see no way how it could influence the results. It is a very insulting to insinuate that results are influenced in any way.
Furthermore, my goal is and always was to provide independent tests (that’s also one reason why I opted for a NPO instead of a company focused on making profits), so I get quite angry if I see people trying to bring us in bad light for their own purposes.
I can’t see the reason for the opt-out methodology then.
The users are being conducted to a false sensation of completeness and security.
The results should be public and the av company should agree with this to join the test.
@IBK,
My intention was never to insult any one but simply to find out if my belief in the
impartiality of these tests was true or a myth.
Since I use some of these test results, I’d like to make sure that they are as independent as I
make them out to be.
Interesting that Comodo don’t want to have the results be published. Because I remember that before Melih was banned from Wilders, he made a statement there he had no problem with testing results to be published.
Mileh ends up with his foot in mouth again, eh? Not surprised, he is clearly the Rev. Al Sharpton of the computer security field.
I mean, come on…even if you use and love Comodo products you have to admit, Mileh’s mouth has a habit of writing checks his facts can’t support.
At the end of the day…you can either trust the documentation and methods the tester/s (in this case av-comparatives) provide you…or not.
While I still question the use of the results of such tests in such a rapidly changing threat environment, I never really had any questions about “who the tester is working for”, especially in this case. I trust av-comparatives to give me the best possible broad-based product vs. product tests.
Seems now it clearer how things work and who pays for what.
At the end of the day I won’t be sad to know that vendors pay a fee.
But I still think, if I’m not wrong, that:
Any result should be published (in an open test), liking the company or not.
The company shouldn’t have the option to not-publish a public test.
No problems on asking for a second fee if the results are published for marketing reasons.
Confidentiality agreements are to be followed.
Payments should never interfere in the results. If the user receive just part of the truth, then the trustfulness in the tester and the professionalism and the independence would be zero (at least for me).
Testers should public acknowledge their paid for the vendors to test the products (again: payments should never interfere in the results).
Tech, that list is what IBK explained, so there is no problem, no contradiction.
The only thing that was not clear (in this specific topic, but it does not mean it is not explained somewhere else) was which are the criteria for a company to get rejected from the first starting point from participating in the public tests.
It’s not that Comodo Doesn’t want to have the results published, It’s the fact that they don’t want to pay Couple Hundred Euros Extra to have AV C to publish the results.
