There are many apps for using one's Android phone as a remote control for desktop VLC.
However, from my research, they expect a regular install of VLC, not a portable install.
Have you figured out how to control "desktop portable VLC" via your phone?
(Worst case scenario, I'll go with using a "remote desktop" app to get my PC's screen on my phone [with a cursor], and use that).