This answer brings up an interesting verse in 1 Corinthians 7, when Paul distinguishes between his words and the Lord's.
Now to the married I command, yet not I but the Lord: A wife is not to depart from her husband. But even if she does depart, let her remain unmarried or be reconciled to her husband. And a husband is not to divorce his wife. But to the rest I, not the Lord, say: If any brother has a wife who does not believe, and she is willing to live with him, let him not divorce her.
This distinction should, I would argue, is similar to that of papal infallibality. As a human being, the Bishop of Rome is a fallen creature. As Romans 3:10 would say, "There is none righteous, not one." And, as Jesus said to the Rich young ruler, "No one is good but God alone." From a pure "perfection" standard, God's word would not allow any man's speech to be perfect. Even Roman Catholic teaching says the Pope is only infallable when speaking ex cathedra, or from God.
Clearly then not all things that the apostles said are thus "infallible," but rather only those things which are "scriptural." In other words, if they are in line with what God would have had these men say, then they are perfect. If not, they are the words of any other man. (I'm reminded of a scene from The Robe, in which Peter must remind his church that he is not perfect!)
The question, then, is whether or not the words recorded in Scripture are "perfect" or infallable. Here, I would point out the exact words of Paul in 2 Timothy 3:16:
All Scripture is breathed out by God and is profitable for teaching, for reproof, for correction, and for training in righteousness,
Notice my emphasis on profitable. Even Scripture sidesteps the question of infallability over its "usefulness" or it profitability. Regardless of whether or not the Apostles said the right thing, God uses it to be the exact right thing that is needed.
I believe wholeheartedly in the perfection of Scripture, ala Turretin, et. al - although I've stopped using the term inerrancy myself. Inerrancy says "This conforms to my standard of what is right and wrong." I say that because in order for something to be "without error" it must be judged, and "I" am the subject of that. That presupposes that "I" am in a position to judge that. That's a pretty high handed stance for a Christian who believes in the completely supremacy of The Word.
In contrast, the word telos or perfection, is a word that:
a. is in Scripture (see James)
b. keeps God in charge.
Perfect implies, as does 2 Timothy, that a thing perfectly accomplishes that which it sets out to do. It is need of no judge, because it is by definition perfect.
By that standard, the words of the Apostles, like the words of Scripture themselves, are exactly what God intends for our own reproof, correction, teaching, and training in righteousness.