If Matthew 1:25 was not speaking in a sexual sense, then those who say it was not are obliged to say in what sense it was speaking. Then they have to explain why Matthew added the word 'till".
Given that way back in the book of Genesis, the phrase "knew her" (speaking of a man knowing a woman) always meant that he knew her carnally - sexually - and this is the case right throughout the Bible - none of the earliest Church Fathers could claim that Matthew 1:25 had a non-sexual meaning with regard to Mary and Joseph. Well, not based on anything the Bible or the first century Church taught. The apostle Matthew wrote that statement, and he wrote unambiguously.
Genesis 4:1 says, "And Adam knew Eve his wife; and she conceived, and bare Cain [their firstborn]". That could not be clearer. If Joseph never "knew" his wife Mary at all - ever - till the day he died - then their marriage was not consummated, and it would therefore be invalid. Their marriage would have been a sham. It would not have been legal.
In this Catholic book, reasoning on Mary's perpetual virginity is put like this:
"Since Augustine (De Sacra Virginitate, 4, 4). theologians in
general were convinced that Lk 1:34 ["How shall this be, seeing that I
know not a man?"] meant that Mary must have made a vow of virginity.
But this traditional view has been criticized in recent years [i.e.
from the mid 20th century]. Why should Mary have let herself be
espoused if she had no intention of leading a married life? Hence many
theologians now assume that Mary resolved on a life of virginity only
at the moment of the annunciation. She then dedicated herself
exclusively and without reserve to the service of the divine plan of
salvation. Through this dedication, she conceived the Son of God in
her spirit as well as in her body." Encyclopedia of Theology, p.895,
article 'Mariology' by Michael Schumaus, Burns & Oates, 1981
Note that that does not face up to Matthew 1:25. Note that it admits to making an assumption. Note that it implies Mary contributed to the conception of the Son of God through a supposed 'dedication'. But the next page deals with her claimed perpetual virginity more clearly:
"The virginity was primarily regarded as virginitas ante partum
(Ignatius of Antioch, Justin.) [i.e. virginal before conception
and giving birth.] As regards the perpetual virginity of Mary there
was no fully general consensus before the Council of Ephesus [431]. It
was not taught by Tertullian, Origen or Jerome. But it was upheld by
Irenaeus, the apocryphal writings of Clement of Alexandria, the
Consultationes Zacchaei et Apollonii, and Gregory of Nyssa.
...Basil held that the contrary opinion was not against the faith. The
most powerful defenders of the virginity of Mary in and after the
birth were John Chrysostom, Ephraem, Ambrose and Augustine. The
conviction of the vrginity of Mary soon grew into belief in her
perpetual virginity. From the 4th century on, her perpetual virginity
is often mentioned. After the 7th century (Latern Synod of 649), the
formula of "virginity before, in and after giving birth" came into
use." (Ibid. p. 866 Bold emphases mine)
Note that that also does not face up to Matthew 1:25. There may be some Catholic writings that do, but I have not seen them in the literature I have access to. This leaves your question unanswered in a direct manner, but does answer it as to who reasoned from the 4th century onward to claim that Joseph must never have "known" Mary sexually.
