I wondered if it wouldn't come in handy if the PortableApps Menu would have an eject button, instead of a close button.
This way you could easily eject the USB drive if you want to.
If you programmers really love that big closing button, I'd be satisfied with a right-click-on-the-tray-icon menu, in which the option of ejecting the USB drive can be found.
One more thing about the closing button, does it have to stop the application? I always have the intention to click it when I want to close the menu.