There's some consternation about this issue, but it's clear that the Jewish writers (Philo and Josephus) confirm their existence and reason (Gen. 6:4). As well, there is some early Christian literature that confirms same.
Here's Athenagoras on the issue:
Some, free agents [angels], you will observe, such as they were created by God, continued in those things for which God had made and over which He had ordained them; but some outraged both the constitution of their nature and the government entrusted to them: namely, this ruler of matter and its various forms, and others of those who were placed about this first firmament (you know that we say nothing without witnesses, but state the things which have been declared by the prophets); these fell into impure love of virgins, and were subjugated by the flesh, and he became negligent and wicked in the management of the things entrusted to him. Of these lovers of virgins, therefore, were begotten those who are called giants.788 And if something has been said by the poets, too, about the giants, be not surprised at this: worldly wisdom and divine differ as much from each other as truth and plausibility: the one is of heaven and the other of earth; and indeed, according to the prince of matter,—
“We know we oft speak lies that look like truths.”
Another early mention was in the Clementine literature:
“But from their unhallowed intercourse spurious men sprang, much greater in stature than ordinary men, whom they afterwards called giants; not those dragon-footed giants who waged war against God, as those blasphemous myths of the Greeks do sing, but wild in manners, and greater than men in size, inasmuch as they were sprung of angels; yet less than angels, as they were born of women.
Philo had this to say:
Therefore he [God] utters no fable whatever respecting the giants; but he wishes to set this fact before your eyes, that some men are born of the earth, and some are born of heaven, and some are born of God
And Josephus mentions giants as well:
For many angels of God accompanied with women, and begat sons that proved unjust, and despisers of all that was good, on account of the confidence they had in their own strength; for the tradition is, that these men did what resembled the acts of those whom the Grecians call giants.
Commodianus wrote this about 275 CE:
Then the Highest uttered His judgment against them [angels]; and from their seed giants are said to have been born.
So, the idea of angels procreating was fairly widespread and believed early on.