A quick search reveals that the phrase "without defect" appears at least 46 times in the Old Testament. What happened to the animals that had defects? What is a defect, and were they fit for consumption by people?
What is an animal with a defect?
According to Deuteronomy 15:21 (NIV) lameness or blindness would be considered a defect. In my personal experience around cattle not having both testicles is a defect. So to protect the quality of your herd you wouldn't want to breed that animal back into the herd. It would however be o.k. for consumption. Even though I don't have anything to reference this too an illness and a defect would be two different things. Obviously you wouldn't want to eat a sick or ill animal.
What would they do with these animals with defects, and could they eat them?
According to Deuteronomy 15:22-23 (NIV) You are to eat it (the animals with the defect) in your own towns. Both the ceremonially unclean and the clean may eat it, as if it were a gazelle or a deer. 23 But you must not eat the blood; pour it on the ground like water.
So just because it wasn't acceptable for sacrifice doesn't mean the animals with defects were wasted or destroyed. Waste not want not!
I think Malachi offers a little more insight into this:
When you offer blind animals in sacrifice, is that not evil? And when you offer those that are lame or sick, is that not evil? Present that to your governor; will he accept you or show you favor? says the LORD of hosts. - Malachi 1:8 ESV
These are animals that would not be eaten or bred due to illness or other problems. The idea is that it's not really a sacrifice if it has little value to begin with.