Why did St. Jerome place Peter's letters after James's letter if Peter was the leader of the Apostles and the Church?
You are not likely to find a canonical answer to this question. My impression is that St. Jerome followed a list already pre-established. And that list is closely linked to the date they were written. With the exception of the Epistle of St. Jude that was the last one acknowledged as canonical by the Early Church.
As for the Gospels go, the reason why Matthew came before John was that the Early Church (including St. Augustine) believed that the canonical order follows the order in which the four (4) Gospels were written. Did the same reason apply to the ordering of the Catholic Epistles? The answer seems to be yes to a degree.
For the first 1,500 years of Christianity, the Church unanimously held that the Gospels were historically written in the order we find them in the canon: Matthew first, Mark second, Luke third, and John last of all.
The reason for Matthew’s priority is simple: the testimony of the ancient witnesses describe Matthew’s Gospel as first and as written in Hebrew/Aramaic. Here is Saint Augustine on the issue:
“Now, those four evangelists whose names have gained the most remarkable circulation over the whole world, and whose number has been fixed as four, …are believed to have written in the order which follows: first Matthew, then Mark, thirdly Luke, lastly John.”
“Of these four, it is true, only Matthew is reckoned to have written in the Hebrew language; the others in Greek. And however they may appear to have kept each of them a certain order of narration proper to himself, this certainly is not to be taken as if each individual writer chose to write in ignorance of what his predecessor had done.”
St. Augustine, The Harmony of the Gospels, 2.
Why Matthew is the First Gospel – and not Mark
Unlike the Gospels, the first Epistles (of St. Paul) were written in Greek, with the exception of the Epistles to the Hebrews, while the other epistle authors wrote in Aramaic (Hebrew).
Generally speaking, we may describe the so-called Catholic Epistles as Post-Pauline. We need not note here that these Epistles are not named after the addressee, as happens in the case of the Pauline Epistles, but after the inspired author. The Epistle of St. James has no final greetings; it was meant for a class, not for persons known to the writer. In I John we have a sermon rather than a letter, though its familiarity of language indicates that the readers were known to the writer. The following two Epistles of St. John are real letters in style and form. St. Peter's first Epistle supposes some familiarity with his readers on the part of the writer; this can hardly be said of II Peter or of the Epistle of Jude. What has been said sufficiently shows that Professor Deissmann's distinction between the artistic epistle and pre-literary letter cannot be applied with strict accuracy. Quite a number of the New-Testament Epistles contain those touches of intimate familiarity which are supposed to be the essential characteristics of the letter.
The Pauline epistles are usually placed between the Acts of the Apostles and the Catholic Epistles in modern editions. Most Greek manuscripts, however, place the General Epistles first, and a few minuscules (175, 325, 336, and 1424) place the Pauline epistles at the end of the New Testament.
When Paul wrote his Second Epistle to Timothy from Rome he felt that all human hope was lost (4:6); he begs his disciple to rejoin him as quickly as possible, for he is alone with Luke. We do not know if Timothy was able to reach Rome before the death of the Apostle. - St. Paul (Catholic Encyclopaedia
Wikipedia has this to add:
[The] ordering is remarkably consistent in the manuscript tradition, with very few deviations. The evident principle of organization is descending length of the Greek text, but keeping the four Pastoral epistles addressed to individuals in a separate final section. The only anomaly is that Galatians precedes the slightly longer Ephesians.
In modern editions, the formally anonymous Epistle to the Hebrews is placed at the end of Paul's letters and before the General epistles. This practice was popularized through the 4th century Vulgate by Jerome, who was aware of ancient doubts about its authorship, and is also followed in most medieval Byzantine manuscripts with hardly any exceptions.
Catholic scholars think St. Paul’s Epistles date from 62 AD to 64 AD. St. Peter’s from about 64 or later. St. John’ Epistles written while in Ephesus sometime later, towards the end of the century.
As for the Epistle of St. Jude it was only admitted to the Canon of Scriptures in the third century due to doubts of authenticity. Thus it is place last in the list of Catholic Epistles.
The Epistle of St. James was simply the first of the Epistles to be written:
Time and place of composition
The Epistle was probably written about A.D. 47. The reference to the persecutions (ii, 6) is in the present tense, and indicates a stage of suffering which has not yet receded into the past of history. Now, in A.D. 44 the Churches of Judea were exposed to the persecution inflicted by Herod Agrippa, in which James, the son of Zebedee, was murdered (Acts 12:1 sqq.). Moreover, the author could not have written after the Council of Jerusalem (A.D. 51), where James acted as president, without some allusion to his decision unanimously accepted (Acts 15:4 sqq.). Another indication also derived from indirect internal evidence, is an allusion to the hungry and naked poor (of Jerusalem, ii, 15 sqq.); they suffered probably from the famine foretold by Agabus (Acts 11:28-30), and usually identified with one mentioned by Josephus (Antiq., XX, ii, 5), A.D. 45.
Place of composition
The Epistle was probably written by St. James in Jerusalem; this we may conclude from the study of the life of the author, and this opinion finds favour with nearly all its critics.
Epistles of Saint Peter (Catholic Encyclopedia):
Date and place of composition
While those who reject the authenticity of the Epistle place it about 150, the advocates of its authenticity maintain that it was written after 63-4, the date of the First Epistle, and before 64-5, the date believed to be that of the death of St. Peter (i, 14). Like the First, it was written at Rome.
Now the Epistles of St. John
Time and place
Irenæus tells us the letter was written by St. John during his stay in Asia (Against Heresies III.1). Nothing certain can be determined in this matter. The arguments are probable in favour of Ephesus and also for the last few years of the first century.
It seems that St. Jude’s Epistle was placed last due to the doubts that originated in the Early Church of it’s authenticity.
Tradition as to the genuineness and the canonicity of the epistle
The Epistle of Jude is one of the so-called antilegomena; but, although its canonicity has been questioned in several Churches, its genuineness has never been denied. The brevity of the Epistle, the coincidences between it and II Peter, and the supposed quotation from apocryphal books, created a prejudice against it which was gradually overcome. The history of its acceptance by the Church is briefly as follows:
Some coincidences or analogies exist between Jude and the writings of the Apostolic Fathers — between Barnabas, ii, 10, and Jude, 3, 4; Clemens Romanus, Ep. xx, 12; lxv, 2, and Jude, 25; Ep. ad Polyc., iii 2; iv, 2, and Jude, 3. 20, Mart. Polyc., xx, and Jude, 24 sq. It is possible, though not certain, that the passages here noted were suggested by the text of Jude. The similarity between "Didache" ii, 7 and Jude, 22 sq., does not seem to be accidental, whilst in Athenagoras (about A.D., 177), "Leg.", xxiv, and in Theophilus of Antioch (d. about 183), "Ad Autol." II, xv, there is a clear reference to Jude, 6 and 13 respectively.
The earliest positive reference to the Epistle occurs in the Muratorian Fragment, "Epistola sane Judæ et superscriptæ Joannis duae in catholica [scil. Ecclesia] habentur." The Epistle was thus recognized as canonical and Apostolic (for it is Jude the Apostle who is here meant) in the Roman Church about 170. At the end of the second century it was also accepted as canonical and Apostolic by the Church of Alexandria (Clement of Alexandria, The Pedagogue III.8, followed by Origen), and by the African Church of Carthage (Tertullian). At the beginning of the third century the Epistle was universally accepted except in the primitive East Syrian Church, where none of the Catholic Epistles were recognized, nor the Apocalypse.
This remarkably wide acceptance, representing as it does the voice of ancient tradition, testifies to the canonicity and the genuineness of Jude. During the third and fourth centuries doubt and suspicion, based on internal evidence (especially on the supposed quotation from the Book of Henoch and the "Assumption of Moses"), arose in several Churches. However the prejudice created against the deuterocanonical Jude was soon overcome, so that the Epistle was universally accepted in the Western Church at the very beginning of the fifth century.
In the Eastern Church Eusebius of Cæsarea (260-340) placed Jude among the antilegomena or the "disputed books, which are nevertheless known and accepted by the greater number" (Church History II.23; Church History III.25); he incorporated all the Catholic Epistles in the fifty copies of the Bible which at the command of Constantine, he wrote for the Church of Constantinople. St. Athanasius (d. 387) and St. Epiphanius (d. 403) placed Jude among the canonical and Apostolic writings. Junilius and Paul of Nisibis in Constantinople (513) held it as mediæ auctoritatis. However, in the sixth century the Greek Church everywhere considered Jude as canonical.
The recognition of Jude in the Syriac Church is not clear. In Western Syria we find no trace of Jude in the fifth century. In Eastern Syria the Epistle is wanting in the oldest Syriac version, the Peshito, but it is accepted in the Philoxenian (508) and Heracleon (616) versions. Except among the Syriac Nestorians, there is no trace of any ecclesiastical contradiction from the beginning of the sixth century till the Council of Trent, which defined the canonicity of both the proto- and deutero-canonical books of the New Testament.
Date and place of composition
It is difficult to state the exact time at which St. Jude wrote his Epistle. But the doctrines against which he inveighs, and the looseness of morals or the so-called antinomismus, seem to indicate the end of the Apostolic age. Jude seems on the other hand to have written before A.D. 70; otherwise in vv. 5-7 he would have spoken of the destruction of Jerusalem. In those verses St. Jude mentions the different punishments of prevaricators, and therefore in this exhortation to Hebrew Christians he could not have passed over in silence so dire a calamity. Moreover we have shown that the Epistle of St. Jude was written before II Peter, which latter was probably written A.D. 64 (65). Therefore St. Jude must have written shortly before 64 (65).
Place of composition
Here we can only guess, but we prefer the opinion that the Epistle was written in Palestine, and probably in Jerusalem.