One common source of knowledge about Joseph and Mary is the Protoevangelium of James. This non canonical gospel was actually composed some time in the 2nd century AD. It is very known in most churches, including oriental churches. Of course, it is of dubious source, but it has been well known and illustrated in whole Christianity.
- And Joseph, throwing away his axe, went out to meet them (= the Mary's parents); and when they had assembled, they went away to the high priest, taking with them their rods. And he, taking the rods of all of them, entered into the temple, and prayed; and having ended his prayer, he took the rods and came out, and gave them to them: but there was no sign in them, and Joseph took his rod last; and, behold, a dove came out of the rod, and flew upon Joseph's head. And the priest said to Joseph, You have been chosen by lot to take into your keeping the virgin of the Lord. But Joseph refused, saying: I have children, and I am an old man, and she is a young girl. I am afraid lest I become a laughing-stock to the sons of Israel. And the priest said to Joseph: Fear the Lord your God, and remember what the Lord did to Dathan, and Abiram, and Korah; how the earth opened, and they were swallowed up on account of their contradiction. And now fear, O Joseph, lest the same things happen in your house. And Joseph was afraid, and took her into his keeping. And Joseph said to Mary: Behold, I have received you from the temple of the Lord; and now I leave you in my house, and go away to build my buildings, and I shall come to you. The Lord will protect you. Source
I have seen this episode in the decoration of lot of old churches, in Europe. Even if we do not have to trust it as part of the Holy Revelation - it is by no way a canonical scripture - I usually have no difficulty to follow this representation.
Also consider what "old" means. It is a relative concept. Today, we could say that someone is old when he/she reaches its 70th birthday. But in ancient times, health was much weaker, and we usually married as young as possible. At Joseph's era, the life expectancy was around 20–30! So one may see himself as an "old man", even a widower with grown children... and being 35...
This is also the earliest text that explicitly claims that Joseph was a widower, with children, at the time that Mary is entrusted to his care. This feature is mentioned in a known Origen's text, who adduces it to demonstrate that the 'brethren of the Lord' were sons of Joseph by a former wife.