I would appreciate it if someone could help by double-checking me, but it appears this basic freeware game is quite portable when installed to a flash drive.
I installed it to my PC and then copied the files over to my flash drive.
I deleted the two uninstall files.
I then uninstalled the app from my PC, which appears to do so cleanly.
Yahtzee does create a file when you use Help, but it's created in the Yahtzee folder.
Here's the link:
http://www.rekenwonder.com/yahtzee.htm
Seems portable to me. I did just as you described, and copied over the game folder to my USB drive.
I had forgotten how fun Yahtzee was.