According to this answer to a question I asked on the Biblical Hermeneutics SE, the original New Testament Greek does not have a phrase like "vain repetition"; instead, the word used, βαττολογησητε ("battologesete") simply means "to babble" or "to sound like one who is stammering". The word is onomatopoietic, and the sense seems to be "using words [not necessarily repeating words] without understanding what they mean", or "using lots of words, in the hope that more words will make the prayer more effective". The New American Bible (Revised Edition), which is the standard used in English-language Masses in Catholic churches in the United States, renders the passage
In praying, do not babble like the pagans, who think that they will be heard because of their many words.
The official Catholic position is that repetition is not done in the hope of making the prayer more effective. Typically, there are two circumstances under which saying of the Hail Mary (specifically, as opposed to other prayers which might be repeated) is encouraged:
- In saying the prayer, or sequence of prayers, known as "the Rosary"
- As part of a common form of penance after Confession (for example, "For your penance, say ten Hail Marys and five Our Fathers.")
Certainly in the first case, and apparently in the second, the repetition involved is intended to allow meditation (see the Catechism of the Catholic Church, paragraphs 2705–2708) on particular events: the events of the life of Jesus, or particular events in our own life which are not always in accord with the love and will of God.
The United States Conference of Catholic Bishops has a web page entitled "How to Pray the Rosary", which comments, "The repetition in the Rosary is meant to lead one into restful and contemplative prayer related to each Mystery. The gentle repetition of the words helps us to enter into the silence of our hearts, where Christ's spirit dwells."
There is no similar teaching that I can find on the repetition of prayers as typically assigned for a penance. There is no particular specification about what penance must be offered for which sins; that's largely up to the confessor, and insofar as it is taught, it may vary from seminary to seminary, and thus from diocese to diocese. I can't find a discussion of how this practice arose; particular priests who assign this sort of penance may have different reasons for so doing.
In any case, however, the repetition is not "vain", that is, empty and without purpose. A Catholic does not say these prayers (or at any rate should not be saying these prayers) in an impatient attempt to get the ear of God. Instead, as mentioned in the bishops' web page, the intended purpose is to focus one's physical activity on the sacred, allowing one's mind, supported by the Divine, to focus on a particular subject, so as to allow us to "make our own in faith the subject considered, by confronting it with the reality of our own life." (Catechism, paragraph 2723)