This isn't an arbitrary grouping of Old Testament sinners, but instead a group of three men who each had the particular error of trying to corrupt the rightful worship of God. Good intentions are not enough when it comes to worship - we must not go against his instructions.
In the case of Cain, the implication is that God did not accept his sacrifice because he did not offer the best of his crops. (Genesis 4:3-5)
In the case of Balaam, Numbers 34:16 says that he not only had intended to curse the Israelites, but that he had advised the Moabites into seducing the Israelites and getting them to worship Baal, which is told in Numbers 25.
In the case of Korah, Moses and Korah actually had a worship contest! (Numbers 16) Korah was a Levite, but he wasn't one of the priests, and so he wasn't allowed to do all that Aaron was. This made him jealous, but God confirmed that only those he said could present incense at the tabernacle.
We could add to the list Nadab and Abihu, Aaron's sons, who in Leviticus 10 burned "strange fire" at the tabernacle and were burned up by God. Creativity was a dangerous thing when it came to worshipping God at the tabernacle!
Jude 10 says that these people "do whatever their instincts tell them, and so they bring about their own destruction." We must be careful not to follow our instincts blindly, but check them against the revealed word of God.