When using public key authentication, the launcher does not adapt the path to the key file. As a result, WinSCP does not find the key when the drive letter changes.
Workaround: manually remove the drive letter from the [Sessions\your_stored_session] section in winscp.ini, e.g.
It is also worth nothing that changing the "configuration storage" preferences to the registry apparently deletes all the sessions that were previously stored in the ini file, there is no going back.
Finally, for those who want to automatically start a session, just edit AdditionalParameters=your_stored_session in WinSCPPortable.ini