You are here

Please add 7z and/or zip option for much faster and more reliable download.

9 posts / 0 new
Last post
Alan_B
Offline
Last seen: 5 years 3 weeks ago
Joined: 2008-01-15 14:18
Please add 7z and/or zip option for much faster and more reliable download.

These options make the download 4 times faster, and much much more reliable.

It takes 15 Seconds to transfer WinMerge_Portable_2.8.paf.exe across the Internet, then my anti-virus (ESET NOD32 2.7) performs an in-depth investigation with the internet frozen solid (and simultaneous downloads suspended) for a further 40 Seconds before it releases its strangle-hold on the system.

My experience is that every PortableApps" takes much longer for ESET to investigate than it took for the internet transfer. n.b. Winmerge 2.8 non-portable takes the same 15 Seconds to transfer, but ESET is satisfied in less than 1 Second. There is something about the paf.exe wrapper, or the additional "portablization", which makes ESET ultra suspicious.

I got a 50 GB free on-line storage backup site, to which I uploaded three versions of WinMerge_Portable_2.8.paf.exe :-
a) unmodified *.paf.exe;
b) compressed *.zip;
c) compressed *.7z
I then downloaded each of these, one at a time.
In every case the internet transfer took 1 minute - probably coming from the far side of the world.
The *.paf.exe took a further 40 Seconds before ESET was satisfied;
The *.zip took a further 10 Seconds before ESET was satisfied;
The *.7z took less than 1 Second before ESET was satisfied;

This suggests that zip will make a download plus anti-virus scan twice as fast,
and 7z will make it 4 times as fast.

Wasting 40 Seconds on a 3 Mbyte file is no big deal,
but 20 minutes on a 90 Mbyte OpenOffice is definite aggravation.

I use Windows XP Home edition with SP2, Eset Nod32 v 2.7 anti-virus, Comodo 3 firewall, and Firefox 2.0.0.14. I believe a 7z option would benefit others with a similar system, and possibly benefit others with alternative browsers and anti-virus protection.

Simultaneous *.paf.exe downloads are unreliable.

I started an OpenOffice_Portable_3_Beta1_en.paf.exe download, during which I browsed and chose and started a download of OpenOffice_Portable_2.4_en-us.paf.exe, and they happily shared internet bandwidth.
OpenOffice_Portable_3_Beta1_en.paf.exe completed its transfer after many minutes, and ESET immediately froze internet downloading, suspending the transfer of version 2.4. whilst it spent many more minutes scrutinising the Beta version. Task Manager showed that ESET only used 60% to 70% CPU cycles, so 30% of the CPU remained available for other purposes, but ESET had additionally blocked internet activity.
After many minutes I went away to watch the television, and when that finished I returned and all downloads were completed. I then realised the suspended version 2.4 failed to resume, and was erroneously reported as being complete.

I believe that a zip download would take only a quarter of the time for an anti-virus check, significantly reducing the risk that a simultaneous download might be aborted, whilst a 7z download would have a very much faster anti-virus check and eliminate any risk of aborting any simultaneous download.

I hope I am not unique, and that others have (or will) notice a drastic hesitation after transfer whilst their anti-virus scrutinises, and will support my request.

If a 7z version of OpenOffice 3 Beta could be provided, I will be happy to test and provide feedback.

Regards
Alan

digitxp
digitxp's picture
Offline
Last seen: 12 years 9 months ago
Joined: 2007-11-03 18:33
Thanks for that very thorough report.

Yes, the reason your Anti-Virus is so suspicous over it is because NSIS techniques are prone to false positives (see here) . They are certified spyware free, but I do understand that this isn't over how there might be a virus and rather how the firewall takes time to verify. Is it possible to ask your firewall to check only by digital signature? If not, you may want to disable it while you are downloading, or download using something like the PortableApps Updater (in Alpha). Zip is considered not an option because the end user may end up installing to the desktop instead of the PortableApps folder (just the average Joe). Well, unless if you knew how to make the Zip file select the portableapps folder under a root and if not change the selected folder to a subfolder called AppName Portable.

This should be stickied as an example support question.

Insert original signature here with Greasemonkey Script.

Simeon
Simeon's picture
Offline
Last seen: 9 years 8 months ago
DeveloperTranslator
Joined: 2006-09-25 15:15
In addition

to that, todays installer do more that just unpack their content to a specific location. Sometimes they have to delete, move or modify files which is something no simple extractor can do.

"What about Love?" - "Overrated. Biochemically no different than eating large quantities of chocolate." - Al Pacino in The Devils Advocate

John T. Haller
John T. Haller's picture
Offline
Last seen: 9 hours 47 min ago
AdminDeveloperModeratorTranslator
Joined: 2005-11-28 22:21
Explanation

Howdy Alan and thanks for the detailed analysis. We only provide our apps as PortableApps.com Installers for several reasons:

  • They are significantly smaller than ZIP downloads (20% or more)
  • They work on every Windows operating system and Linux/Unix/BSD/Mac OSX operating system with wine installed without additional software (Zip only works on XP/Vista out of the box, 7z doesn't work on any OS without additional software)
  • They can be digitally signed to ensure they haven't been altered by 3rd parties and the operating system itself (XP and Vista) can verify this to the user
  • They can perform the appropriate deletion and modification of files to ensure that the upgrade works properly. Many apps need to have files from a previous version removed. Some need to have existing configuration files moved or altered. Zip/7z can't do that and would require that the user moves files around.
  • They can present an EULA before installation which is required by a few of our current apps and will be required by some upcoming freeware and commercial apps.
  • It's easier to maintain and support a single installation method that works in all instances.

As for why ESET is having issues, the installers use LZMA compression which is built into NSIS and used by the installers of many regular applications (software like OpenOffice.org, Winamp, Firefox and more use NSIS for their installers). The algorithm is open source and readily available. The EXEs and DLLs within the apps are also UPX compressed (also open source and readily available). This is more likely what is taking the additional time compared to a "regular" app which hasn't been optimized for portable use. Providing a zip or 7z package wouldn't alleviate that at all.

Weighing the pros and cons, using PortableApps.com Installers makes the most sense from a maintenance, security and user-friendliness perspective. I'm not sure why ESET is so incredibly inefficient with some of the installers but any antivirus program that takes 20 minutes just to scan an 82MB installer obviously has something up with it. Perhaps you could inquire with NOD and see if they are handling LZMAed files as well as they should or if they have issues with UPX compressed binaries. If you would like to put them in direct contact with us, we'd be happy to advise them as we can.

Kind Regards,
John

Sometimes, the impossible can become possible, if you're awesome!

haustin
Offline
Last seen: 12 years 9 months ago
Joined: 2007-09-19 17:59
Excellent explanation

But, I took the OP to mean that he simply ZIP'd and 7z'd the original paf.exe file and tested downloading the three versions (paf.exe, paf.exe.ZIP and paf.exe.7z):

I got a 50 GB free on-line storage backup site, to which I uploaded three versions of WinMerge_Portable_2.8.paf.exe :-
a) unmodified *.paf.exe;
b) compressed *.zip;
c) compressed *.7z
I then downloaded each of these, one at a time.

If I parsed correctly, this means that the virus scanner (ESET NOD32 2.7) is not only horribly inefficient but also not very thorough. Most scanners will (at least optionally) recursively inspect archives, scanning the individual files nested within. Given the improved scanning times, however, it seems that ESET performs only a cursory inspection of ZIP archives and gives a "pass" to 7-Zip archives altogether. With the unacceptable behavior of keeping "the internet frozen solid (and simultaneous downloads suspended)" while un-thoroughly grinding through a single file "before it releases its strangle-hold on the system", I would definitely be in contact with the company I purchased this nonsense from. Perhaps it and Comodo don't play well together?

-hea

Alan_B
Offline
Last seen: 5 years 3 weeks ago
Joined: 2008-01-15 14:18
Thank you everyone for the

Thank you everyone for the information.
I guess I will live with downloading only one at a time - and then doing a MD5 validation.

To clarify a few things :-

I think the Comodo Firewall is innocent - it was not using any CPU cycles, but ESET was taking at least 60% for the duration of the internet standstill;

ESET froze the internet for 10 seconds after receiving the zip file,
but when I subsequently extracted the *.paf.exe contents, ESET locked the system for another 14 Seconds before it stood down.

ESET allowed the 7z file without delay, BUT locked the system for 14 Seconds when I subsequently extracted the contents. So at the end of the day ESET gives the same protection against a 7z package as it does against a zip package.

Forgive me for a dogmatic assertion of what is a plausible belief.
I did not actually measure a 20 minute lock-up. I observed a lock-up for several minutes, very much longer than the 40 seconds for a 3 MByte file, and assumed that I was observing a process that was proportional to the length (or even the square of the length) of the file. I assumed but failed to measure because just this once I followed my son's advice to "get a life", and instead spent 1 hour watching the television, after which the 95 MByte file lock-up was finished, which shows it is not a square law feature !!!

n.b. It is possible the latest version of ESET might now be better. I briefly tried the latest version some time ago, but found an "auto-immune" disease - it not only used virus signatures but it also detected virus type activity, and it seemed to detect and attack itself when asked to scan a selected file.

Regards
Alan

Jimbo
Offline
Last seen: 4 years 4 months ago
Joined: 2007-12-17 05:43
So, what you have is

an anti-virus app that doesn't scan .7z at all (not uncommon, it is a rare format still), doesn't scan .zip files properly, and sometimes detects itself as malware.

Personally, I'd be looking for new AV software at that point.

haustin
Offline
Last seen: 12 years 9 months ago
Joined: 2007-09-19 17:59
additional thoughts

I think the Comodo Firewall is innocent - it was not using any CPU cycles, but ESET was taking at least 60% for the duration of the internet standstill;

I mentioned the firewall because the AV shouldn't actually be in a position to block internet traffic. Of course, if it's single-threaded and unresponsive plus inserts itself at the point of download, it's in a position to block downloads. If you can still view web pages (that aren't written to disk first) while the system is "frozen", then it's definitely just the lame AV.

Another question... Are you downloading to flash? If so, your flash drive is likely exacerbating the situation with single-threaded I/O (N.B. that doesn't absolve the AV from its sins!). Depending on how the AV inserts itself into the download process, there could be multiple simultaneous reads and writes going on for a single file and most flash can't read when a write is in progress.

Just a thought.  -hea

P.S. Interesting about Symantec, BuddhaChu. Yeah, even if a particular AV doesn't scan a file nested within an archive, it should scan it upon extraction if "on-access" scanning is enabled. Gotta love how those CPU cycles waft away like incense to the hardware gods...

BuddhaChu
BuddhaChu's picture
Offline
Last seen: 7 years 7 months ago
Joined: 2006-11-18 10:26
To add to what haustin said,

To add to what haustin said, we have Symantec AV installed on every machine at work (not my choice), email servers, etc, etc. They do recursively check .zip files or renamed .zip files (ex: .zzz) but they let .7z files right through. Reason being is Symantec hasn't added the LZMA algorithm to their code yet so it's not "smart" enough to extract an LZMA compressed file.

Just in case any of y'all didn't know that. Wink

Cancer Survivors -- Remember the fight, celebrate the victory!
Help control the rugrat population -- have yourself spayed or neutered!

Log in or register to post comments