I cannot answer for those who might think "the bosom of Abraham" is literal because I believe the condition of those who have died is spiritual, not physical and, therefore, not literal. Many Christians appreciate that Jesus used the strongest words possible in Luke chapter 16 to describe things that are beyond our understanding this side of glory, in the state after physical death. He used adjectives like 'fire' and 'torment' or 'bliss' and 'comforted', with 'chasm' between the two places of torment and bliss. When it comes to Luke chapter 16, where Jesus spoke about the beggar Lazarus dying then finding himself in bliss in "Abraham's bosom" [or, side] we ought to know that we have entered another realm that is unknowable to us, except by divine revelation.
This revelation Jesus gave was in response to religious zealots who loved money. They despised Jesus for warning them that we cannot serve God and Mammon. This is significant for the account of Lazarus and the rich man in the after-life. The rich man found himself in torments. It wasn't that being rich, in itself, brought him the torments of hell any more than being poor brought Lazarus the joys of being besides Abraham. It was disregarding the suffering of the poor at his gate that was the rich man's problem, which he chose to ignore, until it was too late. Jesus was warning us all about that.
"Was Abraham believed to be the ruler of this place?" you ask. Back in the day, when Jesus spoke, we can discover what religious Jews thought about that. There were two Jewish rabbinical schools that believed in "Abraham's bosom" as a 'place' where good Jews who had died awaited in spirit form for the future Day of Resurrection when their departed spirits would be 'clothed' with resurrection bodies, for judgment before God. They believed that 'bad' Jews would be in torments in hell, awaiting that Day of Resurrection, and judgement. These were the schools of Shammai and Hillel, in existence before Jesus appeared on earth and for some time after. Jesus knew this, and spoke accordingly.
"The Life And Times of Jesus The Messiah" by Alfred Edersheim (1971) Appendix XIX, On Eternal Punishment, according to the Rabbis and the New Testament" (see vol. II Book V ch. vi):
"... For the views held at the time of Christ, whatever they were,
must have been those which the hearers of Christ entertained; and
whatever those views, Christ did not, at least directly, contradict
or, so far as we can infer, intend to correct them... the first
Rabbinic utterances come to us from the time immediately before that
of Christ, from the Schools of Shammai and Hillel. The former arranged
all mankind into three classes: the perfectly righteous, who are
'immediately written and sealed to Gehenna'; and an intermediate
class, 'who go down to Gehinnom, and moan, and come up again,'... The
careful reader will notice that this statement implies belief in
Eternal Punishment on the part of the School of Shammai...
"Substantially the same, as regards Eternity of Punishment, is the
view of the School of Hillel... In regard to sinners of Israel and of
the Gentiles it teaches, indeed, that they are tormented in Gehenna
for twelve months, after which their bodies and souls are burnt up and
scattered as dust under the feet of the righteous; but it
significantly excepts from this number certain classes of
transgressors 'who go down to Gehinnom and are punished there to ages
of ages.'...But since the Schools of Shammai and Hillel represented
the theological teaching in the time of Christ and His Apostles, it
follows that the doctrine of Eternal Punishment was that held in the
days of our Lord, however it may afterwards have been modified... The
doctrine of the Eternity of Punishments seems to have been held by the
Synagogue throughout the whole first century of our era." (1)
This is confirmed in "Josephus' Discourse to the Greeks Concerning Hades" as in "The Works of Josephus" translated by William Whiston, 1980, page 637:
"Now as to Hades, wherein the souls of the righteous and unrighteous
are detained, it is necessary to speak of it. Hades is... a
subterraneous region... angels are appointed as guardians to [souls]
who distribute to them temporary punishments."(2)
Then it goes on to describe another part called "the Bosom of Abraham" with a chaos, deep and large, fixed between it and the place of darkness. Souls are in either one part, or the other, awaiting the resurrection of all men from the dead.
Thus, when we read what Jesus said about what happened to the rich man who died, and to Lazarus (the poor beggar), it all makes perfect sense. Jesus was speaking to their beliefs about punishment after physical death, or to bliss in the Bosom of Abraham! Jesus did not say those beliefs were wrong. He went right along with them and taught his listeners the gravity of loving money and the need to believe God's word.
This is not about Abraham being 'ruler' of the place, nor about prayer to him, for he speaks of no longer being on Earth, in the flesh. The rich man speaks to Abraham as is done in another realm, a spiritual realm. Jesus is conveying a revelation about something we can have no knowledge of apart from divine revelation. Jesus uses words we know to teach us about something we have no knowledge of. We have to go beyond mere words to be taught by the Master about what awaits us - either a 'holding place' for those who will enter into the joy of their Master, or a place of torment for those who will never know the joy of the Master - Jesus. If we do not focus on Jesus as the Son of God who died to deliver repentant sinners from the torments of hell, we will miss the point, and miss the bliss of entering into the joy of the Master, the risen Christ.