Portable PuTTY
I created a PuTTY plugin for one of my BartPE CDs and just tested it to work as a great portable app as well. PuTTY is the best SSH shell for Windows and my scripts basically let you load your personalized settings into the registry, run your session, save any setting changes you made and then delete all PuTTY settings from the host computer when you close PuTTY. It also lets you login with a predefined private-key file so you're not transmitting your password in clear-text either.
It can work from a read-only disk as well as a USB key. Its most secure on a ramdisk-based OS environment (BartPE CD) because none of your session will ever even touch magnetic platter. But it does delete all trace of its use on any Windows system.