Portable statistics software

You can try the r-project: http://www.r-project.org/
It's portable in the sense that once installed on a portable device (I use a USB flash disk) you can run it in any computer running windows (there are versions for Linux and MacOS-X too).
The executables are "hidden" three levels down the path of installation (in r\r240\bin) so PortableApps Menu won't recognize them automatically. You can create a shortcut in the root of the portable and launch it from there.
I've only installed it recently so I didn't have a chance to use it in my research, but I played with it a bit and it looks fine.
enjoy,
dror

You can also try Stats4U at http://www.statpages.org/miller/openstat/Stats4U.htm or OS4 at http://www.statpages.org/miller/openstat/OpenStat.htm . They're both authored by William Miller.

If you use the INNO setup, there are no registry writes.

Of course, R (mentioned above) is a more comprehensive package designed for professional analysis. However, you'll find Stats4U and OS4 to be reasonably comprehensive -- and much easier to learn than the data management procedures of R.