Two things are going wrong here:
1) When the USB-stick is unplugged prematurely, portableapps.exe does not stop itself. Instead it crashes (but remains active).
2) After that, when reinstering the USB-stick and running start.exe, it doesn't start because a ghost portableapps.exe is still running (and consuming 100% CPU!!).
For reasons of pure awesomeness (meaning NOT), a USB-stick on my pc may "unplug" all of the sudden without ever having been told. Don't ask. This is, of course, a horrific screwup from whatever outside PA.com is causing this, but the whole nature of hotpluggable devices like USB-sticks is to be, well, hotpluggable.
This may also happen when unplugging the USB-stick after going to hibernation while portableapps.exe was kept running.