Egypt has a long history with the people of God.
There was famine in Judea, Abram went to Egypt, but was called out.
Gen. 13:1 And Abram went up out of Egypt, he, and his wife, and all that he had, and Lot with him, into the south.
Next comes Israel where again there was famine and the sons go to Egypt, which leads to the enslavement. But, they were called out.
Ex. 3:12 And he [God] said, Certainly I will be with thee; and this shall be a token unto thee, that I have sent thee: When thou hast brought forth the people out of Egypt, ye shall serve God upon this mountain.
In turn, this was the prophecy to which Matthew referred.
Hos. 11:1 When Israel was a child, then I loved him, and called my son out of Egypt.
Mat. 2:5 And [Joseph, Mary, Jesus] was there until the death of Herod: that it might be fulfilled which was spoken of the Lord by the prophet, saying, Out of Egypt have I called my son.
That was the prophecy. On a practical level, Egypt, though a part of the Roman Empire, like Judea was, was under a different command. Judea was run by Herod the Great, while Egypt was run by Gaius Turranias. Both were appointed by Caesar as prefects.
Herod the Great was appointed as "king of the Jews". He was fearful of any new king, but a king in a different area where he had no jurisdiction was hardly worth pursuing.
There were many Jews in Egypt during and before Christ's time. Alexandria was where the Septuagint was translated. There was also the only other Temple, besides the Second Temple in Jerusalem, that had valid sacrifices by Kohen. So, as a community open to a Jewish population, it made sense on a practical level that there is where Joseph, Mary, and Jesus would travel.
The fact that the Jews were once slaves and had been set free had no bearing on events 2,000 years later. From a practical point, there was a lively community of Israelites in Egypt already. From a prophetic point, the family had to go to Egypt in order for the Son to be called out of Egypt.