Augustine is certainly the major patristic source that Calvin uses, for all topics. For his use of the Fathers in general, he was not particularly invested in finding out what they believed: the witness of Scripture was paramount. He did not take the practice of the Church in that era as normative. In his use of Augustine, Calvin wished to insulate himself against the charge that the Reformation amounted to inventing new doctrine (cf. his preface to the Institutes). So he needed someone respectable who (he could say) read the Bible in the same way as he did. There was also a polemical purpose, since the dependence of the Western theological tradition on Augustine meant that Calvin's "reclaiming" of him was an attack on the scholarly integrity of Catholic teaching.
From recent textual analysis of Calvin's writing, scholars have tried to guess at which sources he had available. (He, like other contemporary writers, did not provide detailed citations, and frequently reworded or paraphrased his sources. Nonetheless, painstaking analysis of word choices and errors, plus such hints as his occasional references like "on the next page, Basil says ...", has allowed patient people to gain some degree of certainty about specific editions of the books he owned.) Aside from Augustine, his patristic library included three works by Basil, one by Ambrose, the Contra Haereticos of Irenaeus, the Recognitiones of Pseudo-Clement, and the records of the councils of Carthage, Milevis and Orange.1, Introduction This is a significantly smaller collection than was available to his opponents. (He had read more works than these, but did not necessarily possess them and so was not able to consult them in detail when writing.)
Consequently, almost all of the evidence for his extra-Augustinian reading comes from his responses to critics. For (double) predestination, Albertus Pighius wrote against him in De libero hominis arbitrio et divina gratia (1542), including a full range of references to the Fathers. Calvin's response was in two parts: after only a few months, he published his "On the bondage and liberation of the will" 1, and ten years later "On the eternal predestination of God" 2. The latter work was delayed because after the death of Pighius, Calvin did not consider it an urgent matter to refute him.
The former work contains very little on predestination to damnation. Almost the only clear reference is the assertion (in Book 3, p183) that Augustine believed that the impious are predestined to death. Although other authors are discussed, there do not seem to be any other points where there is an unambiguous mention of the doctrine, as opposed to statements that are used primarily to support predestination in general. I choose not to count the Hypognosticon, because even though it wasn't written by Augustine, Calvin thought it was. And in any case, it is not from the correct era.
In the second treatise, I did find something: an extract from Ambrose's sermons on Luke (pp30-31). The following is Calvin's Latin and French, which differ a bit from one another:
Christus quem miseratur, vocat. Item: si voluisset, ex indevotis fecisset devotus; sed Deus quos dignatur vocat, et quem vult religiosum facit.
Christ appelle à soy ceux ausquels il veut faire misericorde. Item, S'il eust voulu, il eust bien faict devots ceux qui ne l'estoyent pas, mais Dieu appelle ceux qui luy plaist et donne sa crainte à qui il veut.
The English meaning is that Christ calls those to whom he wishes to be merciful; he could make the unfaithful faithful if he wanted; but calls (only) those who he finds deserving. This is from Ambrose's Expositio Evangelii secundum Lucam, 1.10 and 7.27 (vol. 15, col. 1527 of Migne's Patrologia Latina). The first quotation is from a comment on Luke 1:3, and the second from the introduction to Luke 9:53ff. However, although this is a work by Ambrose and not Augustine, it was probably known to Calvin because Augustine used it in his De dono perseverantiae 19.49.
In summary, there is basically nobody among the Fathers, other than Augustine, who can be considered influential on Calvin's doctrine of double predestination. This does not bother him at all: at the beginning of Book 4 of the Defensio, he says:
How is the true Church to be distinguished from the false? By appeal to the multitude, says Pighius. However, I reckon judgement from the word of God, so that the Church is counted as those who follow Scripture. 3
1. Ioannis Calvini Opera Omnia. Series 4: Scripta didacta et polemica. Volume 3: Defensio sanae et orthdoxae doctrinae de servitute et liberatione humani arbitrii adversus calumnias Alberti Pighii Campensis. Geneva: Librairie Droz, 2008. Eds.: Anthony N. S. Lane and Graham I. Davies.
2. Ibid. Series 3: Scripta ecclesiastica. Volume 1: De aeterna Dei praedestinatione / De la prédestination Éternelle. Geneva: Librarie Droz, 1998. Eds.: Wilhelm H. Neuser and Olivier Fatio.
3. Veram Ecclesiam a falso quomodo discernere conveniet? Ad multitudinem, inquit Pighius. Atqui putabam ex verbo Dei iudicandum, ut quae Scripturae adhaereret, Ecclesia censeretur. Ref 1, p211.