This will be a lengthy answer as I will give you all the pertinent Scriptures along with my understanding of each beginning with the following:
All Scripture is quoted from the King James translation unless otherwise noted.
Luke 1:26 And in the sixth month the angel Gabriel was sent from God unto a city of Galilee, named Nazareth,
Shortly after this Mary went and spent six months with Elizabeth, before returning to Nazareth. Until her return no one in Nazareth knew she was pregnant. Shortly after her return she and Joseph were married, and so most in Nazareth probably assumed that Joseph was the natural father of the child, and had been with Mary before she left to be with Elizabeth.
We know that the wedding took place shortly after her return since she was only three months away from delivery when she returned. She delivered in Bethlehem, and that was probably not close to their marriage, since the proclamation would have been given with time for travel and such to be accomplished.
Luke 2:1 through 4 And it came to pass in those days, that there went out a decree from Caesar Augustus, that all the world should be taxed. And all went to be taxed, every one into his own city. And Joseph also went up from Galilee, out of the city of Nazareth, into Judaea, unto the city of David, which is called Bethlehem; (because he was of the house and lineage of David:)
After the Shepherds came and worshipped Jesus Luke says:
Luke 2:19 But Mary kept all these things, and pondered them in her heart.
This seems to me to indicate that this was given to the Apostles by Mary herself, and not from historical accounts.
Joseph and Mary were very devout as indicated by the following Scriptures:
Luke 2:22 And when the days of her purification according to the law of Moses were accomplished, they brought him to Jerusalem, to present him to the Lord;
Luke 2:41 Now his parents went to Jerusalem every year at the feast of the passover.
Now let's take a look at the wise men and their visit and see what we can glean from that:
Matthew 2:1 through 3 Now when Jesus was born in Bethlehem of Judaea in the days of Herod the king, behold, there came wise men from the east to Jerusalem, Saying, Where is he that is born King of the Jews? for we have seen his star in the east, and are come to worship him. When Herod the king had heard these things, he was troubled, and all Jerusalem with him.
And from this Scripture we learn a couple of things:
Matthew 2:11 And when they were come into the house, they saw the young child with Mary his mother, and fell down, and worshipped him: and when they had opened their treasures, they presented unto him gifts; gold, and frankincense, and myrrh.
By the time that the Wise men arrived in Bethlehem Jesus is no longer referred to as a baby, but a young child.
They had stayed in Bethlehem long enough for Joseph to find enough work to get them a house to live in.
They then lived for several years in Egypt awaiting the Lord telling them it was now safe to return to Israel.
Matthew 2:13 through 15 And when they were departed, behold, the angel of the Lord appeareth to Joseph in a dream, saying, Arise, and take the young child and his mother, and flee into Egypt, and be thou there until I bring thee word: for Herod will seek the young child to destroy him. When he arose, he took the young child and his mother by night, and departed into Egypt: And was there until the death of Herod: that it might be fulfilled which was spoken of the Lord by the prophet, saying, Out of Egypt have I called my son.
This would have been a period of several years, until the death of King Herod. Then God sent an Angel to Joseph again to tell him it was now safe to return to Israel.
Matthew 2:20 and 21 Saying, Arise, and take the young child and his mother, and go into the land of Israel: for they are dead which sought the young child's life. And he arose, and took the young child and his mother, and came into the land of Israel.
Upon their return Joseph decided to return to Nazareth to protect Jesus, This was probably because he felt secure in Nazareth where there were friends and relatives.
Matthew 2:22 and 23 But when he heard that Archelaus did reign in Judaea in the room of his father Herod, he was afraid to go thither: notwithstanding, being warned of God in a dream, he turned aside into the parts of Galilee: And he came and dwelt in a city called Nazareth: that it might be fulfilled which was spoken by the prophets, He shall be called a Nazarene.
And when Jesus was 12 years old he announced that he was the son of God, but was not believed.
Luke 2:48 through 51 And when they saw him, they were amazed: and his mother said unto him, Son, why hast thou thus dealt with us? behold, thy father and I have sought thee sorrowing. And he said unto them, How is it that ye sought me? wist ye not that I must be about my Father's business?
And they understood not the saying which he spake unto them. And he went down with them,
and came to Nazareth, and was subject unto them: but his mother kept all these sayings in her heart.
So we see that from the time Mary was about nine months pregnant until Jesus was several years old, they traveled quite a bit and any memories about the incident of Jesus conception would have long faded, since most would have considered her pregnancy due to Joseph anyway.