In one case history I recall (it may have been one of Dr. Stevenson's cases, but I'm not positive of that) a young child reported the following past-life and between life memories. He reported that he died near a river. After his death he stayed by the river in spirit form, and exploring up and down the river, came to a town. Still in spirit form he explored the town and found a young couple that he thought would make good parents so he hung around them for a long time until the wife became pregnant. He then took up residence in the unborn child and was, in due course, born to the couple he has choosen.
According to his account he never saw any other "spiritual realm", never encountered any "spirit guides" or advisors, and made the choice of his next incarnation entierly on his own. The whole experience from the end of one incarnation to the beginning of the next took place right here on Earth. There was no tunnel, no light, no angels, no meetings with departed loved ones, no life review. Nothing. Just hanging around on Earth looking for a new set of parents.
If we take this account seriously then there is no question that we can choose our next incarnation.
I suspect that we can basically do anything we want, including either visiting higher realms, or not visiting them, as we choose; getting help in deciding or going it on our own.
Unlike the physical realm where there is basically only one version of reality, the non-physical realms apparently bend to our desires and become whatever we want them to be. There is no unwavering single version of reality to be found there. I believe that there are no "rules" in the non-pysical realms. If you expect to sit on a cloud playing a harp for eternity you can probably do that. At least until you get bored with it and decide to expand your horizons. That's probably why there is so much disagreement about what lies on the "other side". What lies in that other realm is largely of our own making, so there can never be a single "correct" answer to questions about its nature. You will find that the true answer is whatever answer you are looking for, and that this "truth" is really only true for you.
That would explain a lot, especially, as you said, why everyone has a different view of the other side. I for instance don't believe in hierarchies so I reject the belief of a hierarchical afterlife where we ascend like how we get a promotion in a job.
So if I chose to go visit higher realms, I'd like them to be without a hierarchy but that doesn't mean that there won't be souls or spirit beings with a greater experience than mine and those could guide me or advice me if I asked for their help.
I have also read the account you mentioned, however (it was mentioned both in the works of Stevenson as well as listed as an example in one of the works of Jim Tucker), and I like that concept as well. Although I don't fully reject the existence of higher planes, as i mentioned, I'm not particularly eager to visit them unless maybe to find someone I knew who already died and wished to talk to them again or to find someone who could be a soul mate to spend a life in our next life as sisters for instance. I always wanted a sister!
But it may quite well be that I choose to roam the earth as a spirit. I could go all over the world with no nasty bureaucracy or plane ticket costs to hold me off. So I could just search until I found a family that feels right to me: parents with a more progressive view on childraising like extended breastfeeding and selfweaning, co-sleeping and unschooling. And who already have given birth to a girl who would be my sister. Or maybe they will give birth to another girl, who knows? The sister thing doesn't have to be but it would be very nice.
And I wanna be a girl to those parents.