I don't think there is any hot opposition to numerology directly made by the early church. The opposition is more about the so called 'mathematics' that you found Augustine complaining about. Even earlier, around AD 383, there is an interesting Canon against 'mathematics' that Bishops who assembled at Constantinople defined.
CANON XXXVI
They who are of the priesthood, or of the clergy, shall not be magicians, enchanters, mathematicians, or astrologers; nor shall they make what are called amulets, which are chains for their own souls. And those who wear such, we command to be cast out of the Church. (NICENE AND
POST-NICENE FATHERS OF THE CHRISTIAN CHURCH, P170, PHILIP SCHAFF)
To interpret the remaining of that Canon in the twelfth century Balsamin makes an interesting comment:
“Magicians” are those who for any purpose call Satan to their aid. “Enchantors” are those who sing charms or incantations, and through them draw demons to obey them. “Mathematicians” are they who hold the opinion that the celestial bodies rule the universe, and that all earthly things are ruled by their influence. “Astrologers” are they who divine by the stars through the agency of demons, and place their faith in them.(NICENE AND
POST-NICENE FATHERS OF THE CHRISTIAN CHURCH, P170, PHILIP SCHAFF)
I think this is pretty reliable and shows that the church was not opposing mathematics as we know it, or even numerology specifically (although any deeply superstitious element of numerology that assumed an influence of pagan gods would naturally be condemned) but rather the Greek and Roman quasi religious pagan beliefs in fate and destiny as a result of the planets being gods and supernatural powers. These were the mathematical religious philosophical groups that the church seems to oppose.
Actually the Catholic church at a early period starting using a lot of mystical interpretation techniques that borders on numerology depending on how we define it. Jewish groups before and after Christ also engaged in various numerology to explain scripture in mystical and sometimes plainly outrageous ways. Basically numerical mysticism was only opposed if it had a pagan sub framework but numerology derived from scripture even if to predict the future from a prophecy was okay. Predicting the future from a pagan mathematical system would have been considered an occultist divination, naturally condemned. The idea of a pure non-religious belief in numbers as having a hidden meaning probably did not exist at the time but would probably not have been opposed very much as numbers came from God who made the week to have seven days, etc. So your only numerology option in the church was a biblical based one that condemned all pagan based alternatives.
What your question does make me wonder about is how do the three wise men fit into all this as they seemed to be mathematicians and astrologers somehow finding Christ. Of course I don't support those subjects either but it does make the gospel account seem interesting that Christ could even use such methods to draw pagans to him.
