It seems Telegram Desktop Portable is unaware it should remove HKCU\Software\TelegramDesktop, can you add this?
Also, TelegramDesktopPortable\Other\Source\AppNamePortable.ini should be renamed to TelegramDesktopPortable.ini
Thanks!
[EDIT] Marked as bug, title clarified - mod GC