You can't access the Horizon without having some sort of "connection to oneself", so it stands to reason that said connection would enable you to go back to the right body.

Then again, I haven't been here long enough to be any sort of expert. Maybe you're right. Maybe people get mixed up all the time and nobody's mentioned it.