In the beginning...
Genesis 6:4 (NIV)
The Nephilim were on the earth in those days—and also afterward—when the sons of God went to the daughters of humans and had children by them. They were the heroes of old, men of renown.
The descendants of these children of Nephilim were the giants that show up later in the bible. When Moses sent the twelve spies into Canaan, the promised land (before the 40 years of wandering), they reported back of giants in the land:
We saw the Nephilim there (the descendants of Anak come from the Nephilim). We seemed like grasshoppers in our own eyes, and we looked the same to them.”
There's another reference to them in 2 Samuel:
2 Samuel 21:20
In still another battle, which took place at Gath, there was a huge man with six fingers on each hand and six toes on each foot—twenty-four in all. He also was descended from Rapha.
Goliath, himself, was a Philistine. He was also known as the "Goliath of Gath". Gath was one of the five city-states of Philistine.
1 Samuel 17:23 (NIV)
As he was talking with them, Goliath, the Philistine champion from Gath, stepped out from his lines and shouted his usual defiance, and David heard it.
Because we know that Goliath was from Gath (from 1 Samuel) and that the giants in Gath were descendants of the Nephilim (from 2 Samuel) it's pretty safe to assume he was one of those.
It's also safe to assume some possible genetic issues (due to the six fingers and toes).