Historically, the answer to the race issue has been the Table of Nations in Genesis 10.
From Nigel Tomes:
The tradition that Ham was a black man developed much later. It is a Rabbinical elaboration,7 not explicitly formulated until the Babylonian Talmud of 500 AD. Hence this concept belongs in the category of Jewish “myths and unending genealogies” (1 Tim.1:4). In the Middle Ages, European scholars of the Bible picked up on the Jewish Talmud idea that the “sons of Ham” were “blackened” by their sins.8 These arguments became increasingly common during the slave trade of the 18th and 19th Centuries. A historian, Edith Sanders, concludes that the identification of Ham’s descendents as Black Africans,9 “gained currency in the sixteenth century.” Thereafter, it “persisted throughout the eighteenth century, [and] served as a rationale for slavery, using Biblical interpretations in support of its tenets. The image of the Negro deteriorated in direct proportion to the growth of the importance of slavery.” Benjamin Braude, Professor of history at Boston College, writes10 “in 18th and 19th century Euro-America, Genesis 9:18-27 became the curse of Ham, a foundation myth for collective degradation, conventionally trotted out as God’s reason for condemning generations of dark-skinned peoples from Africa to slavery.” Sadly this notion has been perpetuated through its uncritical repetition by Bible teachers and writers. However, today evangelical scholars reject this view as an out-dated remnant of folklore, masquerading as Scriptural truth. - Nigel Tomes
The text itself, as you point out only says:
Now these are the generations of the sons of Noah, Shem, Ham, and Japheth: and unto them were sons born after the flood. ... By these were the isles of the Gentiles divided in their lands; every one after his tongue, after their families, in their nations.
Of Noah's descendents:
Japheth moved east and became the father of the Asian peoples
The sons of Japheth; Gomer, and Magog, and Madai, and Javan, and Tubal, and Meshech, and Tiras. ...
Shem moved west and became the father of the White European peoples
Unto Shem also, the father of all the children of Eber, the brother of Japheth the elder, even to him were children born. The children of Shem; Elam, and Asshur, and Arphaxad, and Lud, and Aram. ...
Ham moved South and became the father of the African peoples.
And the sons of Ham; Cush, and Mizraim, and Phut, and Canaan. ... And the sons of Cush; Seba, and Havilah, and Sabtah, and Raamah, and Sabtechah: and the sons of Raamah; Sheba, and Dedan.
Whether or not this theory has any merit, it was historically understood to be the case.
The curses in Genesis 9 were thus seen to have been fulfilled:
20 And Noah began to be an husbandman, and he planted a vineyard: 21 And he drank of the wine, and was drunken; and he was uncovered within his tent. 22 And Ham, the father of Canaan, saw the nakedness of his father, and told his two brethren without. 23 And Shem and Japheth took a garment, and laid it upon both their shoulders, and went backward, and covered the nakedness of their father; and their faces were backward, and they saw not their father's nakedness. 24 And Noah awoke from his wine, and knew what his younger son had done unto him.
25 And he said, Cursed be Canaan; a servant of servants shall he be unto his brethren. 26 And he said, Blessed be the Lord God of Shem; and Canaan shall be his servant. 27 God shall enlarge Japheth, and he shall dwell in the tents of Shem; and Canaan shall be his servant.