The Encyclopedia of Ancient Christianity says, in its entry on "Child":
Christians took in these exposed children to save them, though often they could do nothing more than bury them, as the tombs of many children and inscriptions in the catacombs attest (Diehl II, 142–143); in some cases consecrated virgins (Aug., Ep. 98) or benevolent wealthy families educated them.
The first reference is to E. Diehl's Inscriptiones Latinae Christianae Veteres, a highly regarded collection of ancient Christian inscriptions in Latin. The second is to Augustine's Epistle to Boniface, of which the relevant section reads:
Again, sometimes foundlings which heartless parents have exposed in order to their being cared for by any passer-by, are picked up by holy virgins, and are presented for baptism by these persons, who neither have nor desire to have children of their own (§6)
Care for abandoned children by Christians continued into the Middle Ages. The Encyclopedia continues:
After Constantine, laws began to protect these abandoned children, defending them as free persons (CJ VIII, 52,3–4). Later it was largely monasteries
that assumed the burden of educating these children, establishing conditions for their admission to a convent; orphans were accepted unconditionally (Basil, Reg. Maior, 15). [...]
At the beginning of the Middle Ages the most common way of educating exposed or abandoned children was begun: in 787 Dateus, archpresbyter of Milan, donated
a house near the church of S. Salvatore to protect them: the first brephotrophium.