The story of the Garden of Eden is only 3 chapters in Genesis. These should be read thoroughly, for as Walter Bruggeman said, one cannot over interpret them.
In a plain reading, however, the sense is that there was no one other than Adam and Eve. The idea that there were others has no basis and nothing to suggest itself.
Indeed, Adam was alone, and God found it good to make him a companion, per Genesis 2.
Adam had a job - namely to name the animals and steward the Garden. These were jobs that could be done alone.
Adam also had free access to God's undivided presence, which human terms (prior to the idea of omnipresence) would have implied aloneness.
Expulsion from the Garden was inherently an act of exile from the known populated to the unknown. God speaks of the ground and the brambles, but not of other people. As an angel of the Lord was placed to keep anyone from entering the Garden, one can presume the Garden itself was depopulated of humans
In contrast, when Cain goes into exile, he does fear that others would kill him, which would lead one to suspect that Adam was at least unaware of other people.
In short, while your theory is novel, it is completely unsupported by the text.