To add to curiousdannii's excellent answer, I would like to add a few details from the perspective of the Catholic Church itself.
First of all, Constantine the Great (272-337, reigned 306-337) in 313 merely legalized Christianity in the Roman Empire and granted it some favors (by granting certain privileges to the clergy, and by building some basilicas), but the universal Church had been in continuous existence since Pentecost.
The kingdom of Armenia legalized Christianity before Constantine did, in 301; indeed, it went much further, making the Church the official religion.
In Constantine’s day, virtually all Christians viewed themselves as members of the universal (“Catholic”) Church, even though the first great schism—the Arian controversy—was brewing. The term “Catholic,” which comes from the Greek expression kata holou, or “according to the whole,” was actually applied to the Church long before Constantine’s day. Already in A.D. 107, St. Ignatius of Antioch (member of the church in which the followers of Jesus were first called “Christians”) writes
Wherever the bishop shall appear, there let the multitude [of the people] also be; even as, wherever Jesus Christ is, there is the Catholic Church [ἐκεῖ ἡ καθολικὴ ἐκκλησία] (Letter to the Smyrneans, ch. 8).
Moreover, the Church that St. Ignatius knew was hierarchical and sacramental like the modern Catholic Church: it had the three clerical ranks of bishop, presbyter (“priest”), and deacon, and Ignatius records that the bishops and presbyters celebrated the Eucharist. (See Chapters 7 and 8 of the same letter, for example).
Although we have no way of knowing exactly when the church of the city of Rome was founded, the epistles of St. Paul and the Acts of the Apostles record that St. Paul knew that Church (evidently, he even wrote an entire epistle to that church), and ancient extra-biblical sources record that both he and St. Peter were martyred there.
The most important reference is probably from St. Irenaeus of Lyon, who wrote his Adversus Haereses around A.D. 190:
Since, however, it would be very tedious, in such a volume as this, to reckon up the successions of all the Churches, we … [indicate] that tradition derived from the apostles, of the very great, the very ancient, and universally known Church founded and organized at Rome by the two most glorious apostles, Peter and Paul….
The blessed apostles, then, having founded and built up the Church, committed into the hands of Linus the office of the episcopate. Of this Linus, Paul makes mention in the Epistles to Timothy. To him succeeded Anacletus; and after him, in the third place from the apostles, Clement was allotted the bishopric [and Irenaeus goes on to list the bishops of Rome from the beginning up to his own time] (Adv. haer. III, iii, 2).
As can be seen, Irenaeus, in order to refute his adversaries’ claims that Christianity is a human invention, goes to great pains to show that the Church comes directly from the Apostles, and he seems to think it natural to choose the Church of Rome as the best example (even though he was from Asia Minor and serving as the bishop of Lyon).
That the authority in Rome ultimately resided in Peter, not Paul, can be seen from other writings, for example the list of bishops of Rome made by the historian Eusebius of Caeserea, written in 312:
Paul testifies that Crescens was sent to Gaul, but Linus, whom he mentions in the Second Epistle to Timothy [2 Tim. 4:21] as his companion at Rome, was Peter’s successor in the episcopate of the church there, as has already been shown. Clement also, who was appointed third bishop of the church at Rome, was, as Paul testifies, his co-laborer and fellow-soldier (Ecclesiastical History III, iv, 9-10).
Based in part on historical evidence like this, the Catholic Church claims continuity with the Church that the Apostles founded. It claims that the chief of the Apostles, St. Peter, became the first Bishop of Rome, one of the most ancient historical churches in the world (even if Christians lived in the city before he came, and even if St. Paul came to visit them before he did).
None of the later developments in doctrine and practice, the Catholic Church would say, is in contradiction with the doctrine and practice of the ancient Christians—indeed, the practice is different only in externals, and the doctrine is now better understood than it was then, but it has never fundamentally changed.
In summary, then, the Catholic Church (which comprises all the Churches in communion with the See of Peter, not just the church of Rome) would argue that it is the only Church that can claim complete historical continuity with the Church that Jesus founded; hence, in that sense, it is the “oldest” Church. (Or else, in the more technical language of the Second Vatican Council, “the one Church of Christ … subsists in the Catholic Church;” Lumen Gentium 8.)