PreRelease Malware Testing
I would like to suggest at this point that every app that is going to be released be uploaded to our standard MultiMalware testers:
http://www.virustotal.com/en/indexx.html
http://virusscan.jotti.org/
It just does not make sense to have people posting about Viruses being detected from their AntiMalware products when it could have easily have been checked just prior to release.
I think this just make sense.
Especially with the increasing number of releases we are doing.
Now the hard part 
What to do if VirusTotal or Jotti do find something?
Should we hold off release till it's been reported and fixed?
If so, who would do the reporting?
If so, How long would we wait?
If not, Should we release with a warning that there is probable False Positive Alert by such and such product? Where, In the Announcement, in an accompanying post?
Of course it is always possible that a product might start detecting a problem after a definition update after the app is released. But it would make sense to know on the day of release that it is not triggering anything, or if it is how to deal with it.
This should probably be part of the beta testing/development/pre-release procedure so that it can be addressed if needed by the creator Before Final Release.
Opinions?
Tim
Visit the Community page
Join our forums
Subscribe to our email newsletter
Subscribe with RSS
Follow us on Facebook
Follow us on LinkedIn
Follow us on Twitter
Follow us on Mastodon
Follow us on BlueSky
Its a very good idea.
"What about Love?" - "Overrated. Biochemically no different than eating large quantities of chocolate." - Al Pacino in The Devils Advocate
Althou it seems to me a good idea, will it not throw another False Positive if the installer is changed for final release?
Not sure what your asking?
It should be tested after every significant change during testing.
If the installer is changed just before final release that should be a new release candidate, shouldn't it.
It would make no sense to have people test it, approve it, and then change it at the last moment without making sure the change was still okay for reasons other than malware detections I would think?
Maybe, I'm missing something?
Things have got to get better, they can't get worse, or can they?
I think Logan means the act of re-compiling the pre-release installer into the release installer changing the installer executable. This shouldn't matter however as the false positives are for files within the release, not the installer executable itself. Files within should not change at all.. the final pre-release essentially is the final release in all but name.
formerly rayven01
It is a good idea, but it doesn't change the fact that there are bound to be false positives.
One could upload to VT and get a clean "bill of health", and by the time the file is uploaded to SF.net and replicated to the mirrors, one or another AV vendor may start flagging it as malware. There's also the problem of VT saying it has already tested the file and just giving the old result -- a new vendor signature may warn against the file, but you won't know if all you see is the old result. (In other words, do the test once, at the last minute, just before posting the announcement.)
If a file is flagged by one of those vendors as being malware, one should not just assume it is a false positive. Do the normal things to confirm that it is clean, and submit it to the AV vendor, but in particular, I'd postpone distributing the file until it gets a clean report.
If one must distribute a file that is currently being flagged, I'd say so in the announcement somewhere, rather than waiting for an end user to get a scare...
I certainly hope John is checking releases with AV before he signs them (as well as afterward). But you are right, the developers should be doing so too on test releases and release candidates as a matter of course. Doing VT too often might be a bit of overkill though, if the development computer is secure and has up-to-date AV.
MC
Forgive the parsing, It's easier:
"doesn't change the fact that there are bound to be false positives"
Of course.
"One could upload to VT and get a clean "bill of health", and by the time the file is uploaded to SF.net and replicated to the mirrors, one or another AV vendor may start flagging it as malware."
Unlikely
"There's also the problem of VT saying it has already tested the file and just giving the old result"
Request Retest
[edit: How does it know it's tested it before, name?, use a different name, check sum?, If so the check sum would have changed with each update of the program]
" do the test once, at the last minute, just before posting the announcement"
My original thought, but if something is found only at the last minute, what do we do then?
"one should not just assume it is a false positive"
Of course. I usually try to say "Probable" False Positive.
"I'd postpone distributing the file until it gets a clean report"
I'd lean in that direction also, but I can see people getting pissed.
"If one must distribute a file that is currently being flagged, I'd say so in the announcement somewhere, rather than waiting for an end user to get a scare..."
I'd lean in that direction also, but I can see people getting pissed.
"I certainly hope John is checking releases with AV before he signs them (as well as afterward)"
Of course he is
But how many does he use. Probably 2, and not the most popular ones if I recall.
"But you are right, the developers should be doing so too on test releases and release candidates as a matter of course."
I think I'm right too
Thanks.
Tim
Things have got to get better, they can't get worse, or can they?
This may or may not help much with false positives. We have false positives crop up just before, just after, the next day, etc of a release... basically, whenever a company or group sends a bad set of definitions down.
Yes, we should be scanning each release for viruses and malware.
No, we won't let false positives dictate our release schedule.
Sometimes, the impossible can become possible, if you're awesome!
John,
You are correct that false positives can occur at any time.
My thought though is to a least be aware if they will be there on release and if possible start the correction process as soon as possible.
While I volunteered to keep an eye on certain products and certain apps I thought to expand this to all new releases on initial download. I realized with PeaZip that I did not want to have to hunt down every minor AntiMalware program to submit a report for possibly every app we release.
My thought is that if the dev of the program could make use of VirusTotal and Jotti they could handle it for their one app as needed.
I didn't say it would be easy.
I will continue to watch:
FFP, TBP, and CWP
with McAfee, Avast, and CWP
but I can't do it all, and I think it needs to be done.
Tim
Things have got to get better, they can't get worse, or can they?
Perhaps we need a tracker for false positives similar to the dev test page to allow someone to track the FPs as they occur in a central location.
We could also make it a wall of shame ala NSIS' false positives page.
Sometimes, the impossible can become possible, if you're awesome!