Firefox Portable can be updated from within Firefox. However, that can easily cross major versions as well.
Is it enough to replace the loader EXE with the one from the latest portable package? Or do I really need to install the new portable version into the folder of the older application?
The best way to do it is to use the PA.c Platform which handles it automatically. The second best way is to manually download the new Firefox Portable installer and install it right over your current installation (what the PA.c Platform does automatically).
You can update within Firefox itself. It will update, but that's not the best option. First, it'll only update either the 32-bit or 64-bit version of Firefox that you're currently running. Firefox Portable has both, so you'll need to update the other on another computer unless you only use one. Second, it won't update the launcher, but you can manually download a new Firefox Portable and use 7-Zip to extract and new launcher and copy it in (in which case, better to just do a proper upgrade anyway). Third, this update process won't do any required updated on your profile when necessary as it bypasses the installer's code. This was most recently needed for the 84.0.0.2 release and later to fix an issue with update-config.json to ensure that turning off automatic updates is a remembered setting. You can manually analyze the code within PortableApps.comInstallerCustom.nsh inside FirefoxPortable\Other\Source after extracting it from each new installer if you'd like to ensure all updates are applied.
Sometimes, the impossible can become possible, if you're awesome!
Thank you, John. That explained it perfectly!