Hey,
I ran into this post by the traveling librarian who used a batch file to run a Second Life Viewer, then delete the files on the local disk after the program closes (Link: http://travelinlibrarian.info/2007/01/second-life-portable/). I used that script as a template for another program (Robert Lang's Origami TreeMaker Link: http://www.langorigami.com/article/treemaker). It's open-source and relatively simple compared to GIMP. I just modified the script, so that it deleted registry files using the command REG delete. I tested it on a Windows 10 laptop and a Windows 7 desktop. I didn't find any unwanted files in %appdata%, %userprofile% or %localappdata%. It also successfully delete all the registry keys, and it didn't show up on the "Programs and Features" list.
Would this app be "truly" portable? What are the limitations of using this method?
Just doing things like deleting APPDATA or registry settings can kill the data for a locally installed version of an app. We develop our apps using the PortableApps.com Launcher which lets you specify which registry settings are used, which local directories are used, etc so the launcher can backup and restore the data. It also portablizes the paths within so that things keep working as you move PCs and the paths change. Details on use are in the Development section but it's often better to check out how an existing app works for an idea.
For the basics of registry handling, check out 7-Zip Portable, for instance.
Sometimes, the impossible can become possible, if you're awesome!