One common source of knowledge about Joseph and Mary is the [Protoevangelium of James](https://en.wikipedia.org/wiki/Gospel_of_James). This non canonical gospel was actually composed some time in the 2nd century AD. It is very known in most churches, including oriental churches. Of course, if it of dubious source, but it has been well known and illustrated in whole christianity.

> 9. And Joseph, throwing away his axe, went out to meet them (= the Mary's parents); and when they had assembled, they went away to the high priest, taking with them their rods. And he, taking the rods of all of them, entered into the temple, and prayed; and having ended his prayer, he took the rods and came out, and gave them to them: but there was no sign in them, and Joseph took his rod last; and, behold, a dove came out of the rod, and flew upon Joseph's head. And the priest said to Joseph, You have been chosen by lot to take into your keeping the virgin of the Lord. But **Joseph refused, saying: I have children, and I am an old man, and she is a young girl**. I am afraid lest I become a laughing-stock to the sons of Israel. And the priest said to Joseph: Fear the Lord your God, and remember what the Lord did to Dathan, and Abiram, and Korah; how the earth opened, and they were swallowed up on account of their contradiction. And now fear, O Joseph, lest the same things happen in your house. And Joseph was afraid, and took her into his keeping. And Joseph said to Mary: Behold, I have received you from the temple of the Lord; and now I leave you in my house, and go away to build my buildings, and I shall come to you. The Lord will protect you. [Source](http://www.newadvent.org/fathers/0847.htm)

I have seen this episode in the decoration of lot of old churches, in Europe. Even if we do not have to trust it as part of the Holy Revelation - it is by no way a canonical scripture - I usually have no difficulty to follow this representation.

This is also the earliest text that explicitly claims that Joseph was a widower, with children, at the time that Mary is entrusted to his care. This feature is [mentioned in the text of Origen](http://www.ccel.org/ccel/schaff/anf09.xvi.ii.iii.xvii.html), who adduces it to demonstrate that the 'brethren of the Lord' were sons of Joseph by a former wife.