Intro
(As a note, the tl;dr is at the end, in the conclusion)
I don't want to get into an argument about Sola Scriptura here, but that is in part because there were a lot of things which simply were not written down and were passed by word of mouth. This was both because it was cheaper and faster (people without books are better at memorizing), but also because a person was far less likely to be confiscated and burned (yes, Christians were arrested, but they could often go for some time before they were actually put to death and over that time they could continue teaching, documents were destroyed immediately).
The same is true for the Old Testament. Life in ancient Judah was far from stable and when the Babylonians came in and exiled the majority of the educated populous, it got far worse. In fact, at the time of Esdras, he established a school of Hebrew which enshrined the knowledge and teachings of YHWH[1]. You may know their intellectual descendants as Scribes and Pharisees. (This is also why Christ refers to them as sitting in "Moses's seat".
And the Bible also isn't everything which has been written about Judaism/Christianity, and I am not talking about what was in the Deuterocanon ether. There are apocryphal books like III Maccabees, and the books of Enoch (both fascinating, but not worth your time IMHO) from the days before Christ and then there are extra-apostolic works, some of which were written while the Apostles were still alive (and possibly by their instruction) (Immediately, I can think of the Didache).
So, what happened?
The Old Testament
Back in the day, back when Christ walked the earth, there was no canon of scripture. Some groups included Enoch, some included Maccabees, some included books like the Apocalypse of Moses. Quite frequently, different copies would have extra pieces of books or entire chapters missing (Comp. Esther with and without the Deuterocanon). Most deferred to a collection in Greek called the Septuagint. Whether Christ used that collection or not is disputable, but it is undeniable that it was quoted from and used by the Apostles (remember that quote, "a virgin shall conceive", that is the Septuagint version of the verse, the Hebrew is rendered, "A young woman shall conceive").
The Septuagint remained the primary source of Old Testament knowledge for some time.2 Predominantly, this is because it was so clearly used by the Apostles (in fact, the very fact that it was used by the Apostles gives it particularly high prominence in the Orthodox and Oriental Churches), but just as importantly, it was 1. already translated to the common tongue, and 2. it was ubiquitous. There was some questioning of this practice when the Vulgate was being written, but the practice remained in place until the Protestants switched to the Hebrew canon in the 16th to 17th century. It was at that point that the Deuterocanon was removed from the KJV (1611?) and it was "officially recognized" by the Catholics (Council of Trent).
As an interesting side note, one of the primary reasons for the exclusion, originally, was that there were no existing copies of the deuterocanonical books in Hebrew. In recent years, however, a Hebrew copy of Sirach has been found.
2. There was a rumor at one point that a council took place at Jamnia in the latter half of the first century. That is now believed to be false.
And the new
The New Testament was even less cut and dry. There a great deal of quasi-apostolic and pseudepigraphical texts (people wrote under the name of an Apostle as a tribute to them). There were also letters and instructions from the administrators the Apostles left in charge of the Church (Ignatius and Polycarp being the earliest). And then there were quasi-Christian groups (like the Gnostics) who were trying to get their documents recognized too.
A group in Asia Minor might have had the Gospel of Luke, the three letters to the Corinthians by Paul, the writings of John, and the letters of Ignatius. Another group, say from Egypt, might have had the Gospel of Matthew, the Epistles of Peter and Jude, and the entire collection of the works of St. Paul.
In order to standardize both worship and sources of theology (intrinsically linked topics), the Church realized that some things needed to be kept, and others discarded. And so it decided on the standard of, "If we can be reasonably sure that it is from the Apostles, we will include it." To be honest, they got a lot of it wrong, meaning we have things like the story of the woman caught in adultery, which may or may not have been a part of the New Testament — was that part of John, Luke, or was it not even in the canon to begin with?
And the rest?
There actually are quite a few documents out there which were not only Christian, but were used in the services and held on par with Apostolic documents for some time. And I am not talking about questionable documents like, say, the gospel of Thomas (notice the small "g"). There are some which can only be counted as true Christian treasures which all believers would benefit from. Remember, just because we know that scripture is God-breathed and profitable for teaching, that does not mean that it is the only thing which is God-breathed and the only thing profitable for teaching.
My personal favorites are the writings of Ignatius, "Episcopai" (biblical title which translates to "administrator" but is also properly rendered "bishop") of Antioch, who wrote the words, "I am God's wheat, about to be ground by the teeth of wild beasts into the bread of Christ." But there is also the letter of Clement, one of the first Roman episcopai, who wrote to the Corinthians (guess what, 20 years after Paul was dead in the grave, the Corinthians were STILL fighting). There is Polycarp, who is a great study in martyrology, and then there is Justin Martyr, whose martyrdom is rather dull, but he writes some wonderful things about the Gospel. And, of course, there is the gallic Irenaeus.
Conclusion (tl;dr)
My, I have said quite a bit, and I suppose I will end with this, if all of Jesus's actions were written down, the entire world could not hold them (last verse of Gospel of John). This means that any collection of books will necessarily be insufficient. The books we do have are books which were known to be reliable and were known to be the most essential documents of the early Christian era. And, more importantly, we have the advocate, the Paraclete who is to guide us on our way.
So, go with the Spirit. Read scripture and know it. And, if you have time, check out the other writers too, they're definitely worth something.
