New: Kanri (Oct 9, '24), Platform 29.5.3 (Jun 27, '24)
1,100+ portable packages, 1.1 billion downloads
No Ads November!, Please donate today
New: Kanri (Oct 9, '24), Platform 29.5.3 (Jun 27, '24)
1,100+ portable packages, 1.1 billion downloads
No Ads November!, Please donate today
Visit the Community page
Join our forums
Subscribe to our email newsletter
Subscribe with RSS
Follow us on Facebook
Follow us on LinkedIn
Follow us on Twitter
Follow us on Mastodon
Follow us on BlueSky
well if you do that,then people would think think Compactor by PortableApps.com? Compactor for what?... a compactor for Apps, hence the name App Compactor. it just happens that "App" is also in the PortableApps.com company name
I understand what you saying, but still it sounds a little repetetive. In my opinion PortableApps.com Compressor would be best. It's more of a development tool, anyways. And I don't think there would be much trademark confusion on that one.
I agree, it sounds somewhat repetitive and the point brought up earlier about a compressor for what doesn't seem to work for me, because its not just for compressing your PortableApps, its for compressing anything it can compress (which is usually Portable Apps but isn't always) and I like "PortableApps Compressor Kit" (PACKit) [just a suggestion though PACK is good too {and PortableApps.com App Compactor is cool too if you dont want to change it}] and if the .com needs to be there So Be It.
~3D1T0R
I just did a basic test of pre-release 4 in vista ultimate sp1, and it worked without problems, I'll try and do some more testing but so far it's looking good.
Seems to work pretty well, but I have a couple of suggestions:
2>nulafter each of your JAR/ZIPrenamelines to get rid of the screen full of "The system cannot find..."Cool app, as always. -hea
<EDIT>
I recall seeing a thread elsewhere pondering the list of possible extensions for executables, DLLs and other binaries that UPX could compress. The magic number for a WinDOS relocatable object file is ASCII
MZin position 0 of the file. That goes for EXEs, DLLs, IRCs, BINs, etc.So, instead of using BAT files to recurse the directories looking for specific extensions, the NSI could include RecFind.nsh (recursive find) and check the magic number. If it's MZ, call UPX. (Note that PY files are actually ELF/Unix rather than MZ/WinDOS, so they'd be ignored.)
Of course, it could preemptively ignore TXT/INI/CFG files, although UPX figures out pretty quickly that their internal structure is flawed.
Might be a little harder to create that pretty details window with all the brightly colored error messages, however...
Just a thought. -hea
</EDIT>
.py files are text files, not ELF binaries.
.pyd files are binary, but contain "MZ" at position zero so going by your description, they're Windows binaries.
The only files I've seen that are ELF are .so library files that are included in Windows packages sometimes.
Cancer Survivors -- Remember the fight, celebrate the victory!
Help control the rugrat population -- have yourself spayed or neutered!
Yes,
.pyusually means Python source. But check this out:App\openoffice\program\python-core-2.3.4\lib\difflib.pyBTW, ELF is the standard Unix binary format.
When I was composing my comment the other night, I recalled a thread about UPX borking OpenOffice's PY files. Hmmm. Why would it try to compress a text file? So, I go to the above directory and randomly pick a PY file to look at in Notepad++. Very odd, it seems that python.exe must be using ELF as some sort of intermediate format (why? and why not change the extension?!?). That must be why UPX is hosing OOo's PY files...
Seriously, what are the odds that I would click on the only ELF.py under the whole python tree? I've always had a knack for instinctively testing boundary conditions and error cases... I guess I still have it.
I tested UPX against a few text files and it refused to compress them. I added
MZto the beginning and got the same result.Ergo, my post. -hea
So, why would UPX try to turn an obvious text file into a self-extracting executable?!?
N.B. the first option described for
python.exe:UPX documentation, NOTES FOR LINUX section:
Linux/386 support in UPX consists of 3 different executable formats, one optimized for ELF executables ("linux/elf386"), one optimized for shell scripts ("linux/sh386"), and one generic format ("linux/386"). [ ... ] Shell scripts where the underlying shell accepts a ``-c'' argument can use the Linux/sh386 format. UPX decompresses the shell script into low memory, then maps the shell and passes the entire text of the script as an argument with a leading ``-c''. General benefits: - UPX can compress all executables, be it AOUT, ELF, libc4, libc5, libc6, Shell/Perl/Python/... scripts, standalone Java .class binaries, or whatever... All scripts and programs will work just as before.So, when UPX recognizes a platform-independent Python script, it kindly turns it into a Linux-specific self-extracting script preprocessor. The problem is that UPX is cross-platform, but doesn't provide a way to specify which platform(s) you're actually interested in right now. Seems like a
--targetoption to UPX would prevent a lot of misunderstandings.-hea
I took a stab at all of my comments in the previous post with the notable exception of #6. The resulting installer can be found at drop.io (I'm sure you can figure out which file). I took great pains to uglify the installer so that no one would mistake it for an official PortableApps.com release.
Anyway, it has a few GUI/MUI oddities but otherwise seems to work fine. To go along with the magic number processing, it has a
!definelist of file extensions to always UPX (without first checking for "MZ") and extensions to always ignore (again, without checking for "MZ"). To handle exceptions to the rules, I'm proposing an additional section for appinfo.ini like the following:Basically, for apps that can be auto-updated in situ, there might be some files or file types that are known to break after UPX. And, there might be an odd file type that can be UPX'd even though it doesn't have the Mark Zbikowski magic number in the header. These exceptions can be specified in each Portable App's appinfo.ini and App Compactor will play along. Just separate multiple entries on the same line with a pipe character.
I also added lots of redundant error checking (on top of John's) to make sure that the user's files don't get blown away until a suitable replacement is already in place. Disappearing files are still possible, though. If you kill 7za while it's compressing or testing, it's pretty easy to catch that something went wrong. Unfortunately, if you kill it while its uncompressing, my script doesn't notice. So, it won't have extracted all of the original files, but will happily recompress the partial lot and call it a day.
So, everyone, please let me know if you find the obvious stupid error that I intentionally left as an exercise for the reader.
Back to you, John.
-hea
<EDIT>
Added link to the previous post I referenced.
</EDIT>
Instead of messing with directory recognition could you possibly tell it that anything named *Portable.exe is not to be UPXed?
¿i¿Could that work?!?
~3D1T0R
Don't know how your comment slipped by without my notice... I apologize.
Instead of messing with directory recognition could you possibly tell it that anything named *Portable.exe is not to be UPXed? ¿i¿Could that work?!?
For preventing NSIS installer corruption, yes. But the main reason my script tries to find the AppInfo directory is to find the app-specific compactor config (i.e., the elsewhere-mentioned "compression manifest"). If it can't find that, it just proceeds with "reasonable" defaults. There's also the slight possibility that some user would want to use the App Compactor on a file named FooPortable.exe, when that file is just a regular bloated program and not an NSIS installer at all. (Of course, my current code would also silently refuse to compress it if it was at the topmost level of a PAF directory structure. That logic clearly needs to be tweaked.)
I actually looked to see if NSIS files had a signature/magic number, and they do. Unfortunately, I haven't found one that's at a fixed offset.
Perhaps your idea could be merged into the manifest idea -- the *Portable.exe would be listed as an ExcludeFile. Hmmm.
Thanks for the suggestion! (Have you tried #5 yet? ;-))
-hea
Sorry for two things
~3D1T0R
All .zip files with a ! in the file name will extract to the temp folder then spit out a file not found error. They fail to re compress, the temp folder does not get deleted and the files are not renamed back to .zip as a result.
Oh and a feature suggestion. You might not be allowed to re distribute some programs in compressed form due to license issues. However nothing prevents you from distributing a listing of safely compressible files for those programs. You could greatly enhance this programs usefulness by provided per application scripts that let the end user compress none redistributable programs safely.
Does this happen with version 1.0.0.3 (John's, Pre-Release 4) or 1.0.0.4 (mine, Development Test 5) or both?
John's uses BAT files to process the file list; mine does it all within the NSIS script. It might be a side effect of batch file processing.
As for your suggestion, take a look at my
appinfo.inimodification. Does this seem to address it, or would you add something like the following (where the addedIgnoreDefaults=trueline tells it not to pay attention to App Compactor's built-in default extension lists, and the addedIncludeFileline lists specific files to include)?Thanks. -hea
<EDIT>
P.S. My version is based on John's -- it simply moves the external batch file functionality into John's NSIS script. Also, because it wants to look in
appinfo.ini, mine expects you to specify the FooPortable or FooPortable\App directory, but not FooPortable\App\foo (I should add a check for this). This behavior also makes it less confusing for new users, who would naturally specify X:\PortableApps\FooPortable without knowing what "Don't compress NSIS launchers" means.</EDIT>
This is what it is doing.
I try and compress a file named "whatever! - something.zip"
It will create a temp folder named "whatever" ignoring the "!" and anything beyond it.
Now it will not be able to locate the file because the path is incorrect. It spits out a not found error and moves on to the next file leaveing the temp folder and the ._zippaCompactor file behind.
As for modifying "appinfo.ini" I was suggesting something a little more robust. Like Mozilla wont let you redistribute Firefox in modified form. So you create a script file named say "CompressFirefox.ini". That file contains a profile of how to properly and safely compress Firefox. What file can and can not be compressed. What compression mode is best used on a specific file. Etc...
So the user downloads that file opens AppCompactor, selects the compression profile, points to the location of Firefox. AppCompactor will now automatically and safely compress Firefox on the users end. So long as the end user does not re distribute it again you skirted right around Mozilla's license restrictions.
Mine doesn't use the ._zippaCompactor extension like John's (I believe the extra logic in his was a workaround for a weird problem within batch files). If the original ZIP was
foo.zip, my temp file isfoo.zip~; the original file is never renamed, only replaced if the recompress was successful.I'm not where I can look into it right now, but I certainly will later.
Re: your compression manifest idea
That's actually how I was originally planning to do it (except that the manifest would be included in the portable app's AppInfo directory instead of distributed separately). I already have all the code to read a complex configuration from similar INI files. It's a pretty small change to go back to that method if everyone thinks it would be better. My reason for switching methods was the "simpler-is-better" mantra and the notion that an auto-update of a base app would likely invalidate the list of compressible files (if they were specified by name). However, nothing says the two methods can't simply be merged.
Thanks! -hea
P.S. Although I've only used Notepad++'s Compare plugin for the files I changed, I'd recommend Ryan's WinMerge Portable to see all the differences between mine and John's (under both the App and Other folders).
I clicked the "Mine" link and thought that sent me to yer build but I just noticed it is version 4 not 5. I'll check 5 now.
Yer way of doing the compression manifest seems fine. Either method is fine. We seem to be on the same page as far as the general idea goes and thats all that matters to me
[edit]
Actually can not find release 5
[edit2]
Alright I tracked down 5. checking it out now.
[edit3]
Yer version is good to go. It picks up the "!" file names just fine. You have an issue with the progress bar however. It seems to like to continue to progress even after the job is completed. I am pretty sure you are aware of it already though.
That link points to my post about my version, but Drupal apparently drops you off at the top of the first page of a thread if you link to any pages beyond that. Of course, the referenced post is on page 2. Grr.
Yeah the progress bar thing is rather odd. I used the config from John's which fakes the user out by simply progressing along at a predetermined pace (regardless of what's really happening). Unfortunately, if you then update the bar using one of the DLL's other functions, it will update as you wish and then the stupid fake-out routine moves the bar back to where it last had it and keeps plodding along at the predetermined pace. It's supposed to go to 100% when it sees certain text appear in the window, and I thought that part was working. But, making the window stick around (so the user can actually review the details) causes some weirdness with the Next> button. That was my first time looking at NSIS' Modern User Interface config, and I'll readily admit that I didn't find it intuitive. I would consider the RealProgress DLL buggy, however, because its entry methods don't maintain a consistent state within its own progress bar object.
Do you think I should post a separate topic so that it won't be so confusing to find mine? I originally chose not to because I didn't want it to seem that I was "stealing" John's project. It's a significant enough change to call version 1.0.1, but I upped the rev number instead (while downgrading the nomenclature from Pre-Release to Development Test) for the same reason.
I actually have no idea if you're the only one who's found it.
Thanks for trying it out! -hea
I'd like to try this out as well. I didn't like the idea of external BAT files. However no one likes having their toes stepped on, so if I were you, I'd ask John how he feels. You'd then either take over development, donate your code to John to integrate, or release it as a separate app.
It's already integrated and is fully donated to be accepted, modified or rejected as John pleases. I'll gladly help keep it up to the extent of my limited abilities, but it can easily be abandoned if that's the preferred path.
Of course, John is and should be the sole arbiter of anything that carries his PortableApps.com brand.
I'm looking forward to the day when his enterprise scales to a Mozilla-esque model, where the community-at-large contributes, reviews and tests code, and a project owner decides what makes the cut and what doesn't. For non-branded projects, John can designate or accept project "owners" who make the call. Add a bug/project tracker to the SVN that's in the works, and you're talking bona fide Open Source rather than just Available Source. (Unnecessary details of cost, effort and time have been ignored for brevity. :-))
Goosebumps. -hea
My Development Test Five is available on drop.io; John's Pre-Release Four is available at the beginning of this topic.
Here's some updated Release Notes:
I haven't gotten an opinion from John yet, but a couple of users have provided valuable feedback. I haven't made any notes or touched the code since I posted it, but I've mapped out in my little noggin some changes that I'd like to make. I'm pretty likely to put it into code anyway, but I'd rather not post a follow-up release if John would rather me keep it to myself, or if there's no interest from the community.
Here's what I had in mind:
App\AppInfo; AppInfo; ..\AppInfo;<SelectedDirectory>). This will allow app-specific config files to be created for non-PAF apps (appcompactor.ini would be placed in regular app's top-level directory).There was probably something else, but I'll have to remember it or re-invent it when I'm coding.
Any thoughts on or interest in a Development Test Six? -hea
Definitely interested, with John's approval of course.
I really like the changes!
It worket like a charm untill I used the BRUTE method on FirefoxPortable beta 5, it couldn't proccess xul.dll, and I had to manually close upx.exe to get it to continue.
I waited about 5 mins or so, and all other files was proccessed quite fast.
anyway, a suggestion: Show witch method that was used for each file when using BRUTE.
Nothing here right now... I might put something here later, who knows? Or will I? I really have to think about this...
Hmmm. I haven't played with the BRUTE setting -- I guess I'll have to take a look at it.
Thanks for the feedback! -hea
<EDIT>
Oddly, UPX doesn't tell you what compression method it chose with BRUTE — even with the -v (verbose) or -tv (test verbose) options. Seems like that would've been handy (or at least interesting).
</EDIT>
I made an icon with a screen from the crystal icon pack. I can't find a good arrow, so feel free to add one.
*Edit:* Oh, whoops, forgot to put a linky.
Insert original signature here with Greasemonkey Script.
UPX 3.03 was just released on April 27; make sure to update it for your next release
Either the library computers are too screwed up work right(thank you stupid tech!) or there is something seriously wrong with the app compactor. I tried compressing all of the portable apps available for download on this site and the most I got out of it was -11kb. Wow, that sure frees up space on my pen drive(sarcasim). Also, it requires the command prompt to run, and half the computers here at the library either have it disabled, or removed entirely(how they still boot up is a mystery to me). So is there any way you can remove the need for a command prompt to open? That would make it truly portable. As for whether or not the apps survived, most of them still worked, but the launcher was broken.
The following apps would not work at all:
OpenOffice suite (a TON of error window pop up)
7-zip (not sure why)
InfraRecorder
VLC Media Player
Mplayer
The following apps gave warnings, but ran okay:
The GIMP
Pidgin
Please note that all apps were tested on library computers running Windows XP Professional, service pack unknown but probably none. I did not try PuTTY, XAMP, FileZilla or any of the other "advanced" web related apps because they don't work on these computers at all and PuTTY nearly got me banned.
EDIT!
I missed the part about being "for developers". If you don't mind me asking, since I am now confined to a pendrive, is there a tool that normal people can use to compress apps so that they will fit on and still run from a pendrive? I'm not asking for a whole lot of compression here, just 20% or 30%. I really want to use the GIMP, but that will only leave me with 20MB of space left on my pendrive, and I want to create a large site in the near future.
comparison of features
Windows: empty pockets, slow computer, crippling malware, user level frustration, inability to meet deadlines, security holes
Linux: full pockets, super fast computer, endless software, endless customization, endless potential
the reason no space was saved is because all of our apps are compressed in this way. developers use it to compress apps for this site and you can also use it to compress portable apps not from this site (it saves a ton of space without having to zip anything). as for breaking your launchers, that is a known fact and there even is a note about it in the main post; i think it's cause the launchers are already compressed with lzma. the gimp portable is already compressed as much as possible in order to run it portably; the only other thing you could do is zip it up in .7z or something and extract it to the host computer every time (which would get annoying).
The launchers break because they store a CRC checksum internally to verify the integrity of the file. UPX'ing them changes the CRC (obviously) and it fails the verification and refuses to run.
This program works well!
I like it because it compresses apps.
(Since I have a 2GB.)
EDIT:When it gets to step 4 of 4 (I think,) it always says something about
The system cannot find the file spectified(It doesn't bother me that much).
John, I just emailed you the source for my latest changes.
I implemented everything listed in my Development Test 6? wish list except for the special handling of the PortableApps root directory.
Thanks. -hea
<EDIT>
Modified link to bring up the second page (where the comment is). It sure would be nice if the server would do that for you — even the "fixed" link will break if enough comments are added to page 1 to push the linked comment to page 3.
<EDIT>
I would like to also see an option to compress files inside ZIP/JAR files before sticking them back in.
My reason for requesting this is specifically that Python, when using the "bundling" option with py2exe, sticks all the *.pyd files inside Library.zip; these can be often be compressed quite a lot, making it even smaller.
I am a Christian and a developer and moderator here.
“A soft answer turns away wrath, but a harsh word stirs up anger.” – Proverbs 15:1
...but I'm not sure how recompressing an archive of compressed files would end up much smaller than recompressing the original archive of original files. It seems that the original archive would have more redundancy (more and longer strings) for the compression method to take advantage of.
Plus you'd get hit with two decompression steps (ZIP and UPX) whenever you accessed one of the binaries in the doubly compressed archive.
So much for the theory...
How do the recompressed Library.zip archives compare in real life?
-hea
It was ~5MB instead of ~5.1-5.2MB, from memory.
Anyway, now BPBible Portable (0.3) is using a different build, with bundling and compression turned on for py2exe. It doesn't go down quite so much any more, but it's all stored in bpbible.exe. And don't rename it - that breaks it now! (py2exe issue, but I should add that as a "Known Issue")
I am a Christian and a developer and moderator here.
“A soft answer turns away wrath, but a harsh word stirs up anger.” – Proverbs 15:1
Some Apps are not able to handle newly compressed Jar-Archives, for example OpenOffice.org. Because of this we had to move the Jar-Archives first away, before we use the Compressor. Afterwards we had to move this Archives back.
It would be a big help, if the options for Zip- and Jar-Archives were divided.
Is hope there would be a way to have an Release 5 with this enhancement
Pages