I just started up Firefox portable ESR and noticed that sync wasn't working after further investigation it seems it's respecting group policy and it's writing to the registry. If it's doing that then I don't see what the point of it it? It's not a portable app. It should only write settings locally to the same directory and never to the registry. It should not respect GPO policies, Intune, etc.
We don't change the apps we release. If the app gives us a way to redirect settings, registry, and/or other data to the app's folder, we use it.
But if the app doesn't give us a way to do that (which is a lot of apps), we handle putting the app's settings where they are supposed to go before launch, and then clean them up again after close so they always live in the app's folder when it is not running.
Are there ways to redirect/virtualise the registry? Yes, there are, but they rarely work properly, so we don't even try to use them.
The only other way to do what you are expecting would be to change each base app to not use the registry and to allow redirecting settings, which is asking for a lot more than the developer resources we have to work with would allow.