R - www.r-project.org
One of the problems that I have as a consultant is not having a full suite of tools when I am off for meetings. Normally I use a Palm T5 for most work. Sometimes, clients have problems that need more sophisticated tools. One of the best open source tools is R, an open source analogue of S. It is extremely powerful (See Modern Appllied Statistics with S - Venables and Ripley) and has the bleeding edge of stats.
I had originally requested a portable implementation from John. However, you can do it yourself, just copy the foler c:/program files/R over to your USB. When you load the USB, you need to navigate to the RGUI module in the BIN folder. Firing that up gives you a very powerful statistical system. It seems to work not badly on a USB key (about 100mb needed). I am still testing it in a USB environment. I have tried it on a "cold" machine on which R has never been installed and it seems to work well and leave no obvious leftovers on the main hard drive.