Protestant Christians in general believe that 'faith' makes a person righteous. The Letter to Hebrews chapter 11 in the N.T. talks about faith at length with specific examples. But what kind of 'faith' in 'what' can actually make a sinner justified or righteous as per the teachings of the Bible?
Protestants who believe the Bible to be God's inspired word carefully study all that it says on the matter of faith, and those who have the kind of faith the Bible advocates discover what God's righteousness is all about. That protects them from the vain supposition that a person can do anything to either become righteous, or to merit righteousness from God.
The starting point has to be God's view of righteousness. Only once that becomes clear can anybody start to appreciate the kind of faith that pleases God so that they are moved from their present state of being unrighteous in the eyes and judgment of God. To discover the roots of the matter, a bit of digging has to be done, which spade-work has already been done for us in the 11th chapter of Hebrews, which you have noticed.
As the examples of saving faith are of people who lived before the time of Christ, we need to notice that they had saving faith that pleased God. Then we can compare their faith with the faith of those in the New Testament, and what do we notice? Exactly the same faith was at work in both old, and new covenants! There has been no change in God's requirement that believers in him must have a particular kind of faith. It is faith that utterly trusts him, and places total confidence in his promises and not in trying to mix that with doing things that are viewed as 'contributing' towards gaining a righteous standing.
Take Abraham's example in Hebrews 11:8-19. He heard God calling him out of his comfortable, prestigious life - to travel to an unknown destination. He obeyed the call, living as a wanderer in tents. Then God told him that although he and his wife were childless, Sarah being past the age for that, God would give him offspring that would become numerous as the stars, and that all nations would be blessed because of his faith. Abraham believed that promise - he put faith in it, despite more decades passing and no child of promise. Not until he was 100 years of age, and Sarah was 90, did the miracle finally happen. But then, when the lad was well grown, God called Abraham to sacrifice him. That's where Abraham's faith really showed, for he told his servants that he and the lad would return to them; that God would provide the sacrifice. He had faith that God would resurrect Isaac if needs be because God would never break his promise about that. Hebrews explains it:
"For he looked for a city which hath foundations whose builder and maker is God... they desire a better country, that is, an heavenly, wherefore God is not ashamed to be called their God, for he hath prepared for them a city... By faith Abraham...[who] had received the promises offered up the only begotten... accounting that God was able to raise him up, even from the dead; from when also he received him in a figure."
All those people of faith in Hebrews kept their eyes on God, in heaven, and were not lured by worldly, material 'things'. Their love of God was such that they would give their lives for him, God was that real to them. Then comes Heb.12:2 where Christians are exhorted to look "unto Jesus, the author and finisher of our faith." There is the bridge! The O.T. people of faith looked unto God, who began and completed their faith, and that becomes looking unto Jesus!
So, what did Jesus say to us? "Seek ye first the kingdom of God and HIS righteousness, and all these things shall be added to you" (Matthew 6:33). Here is also the bridge between the O.T. and the N.T. regarding righteousness:
"There is none righteous, no, not one" - Psalm 14:1-3 & Isaiah 64:6-7 & Romans 3:9-18. Then Romans 3 explains that:
"By the deeds of the law there shall no flesh be justified in his sight: for by the law is the knowledge of sin. But now the righteousness of God without the law is manifested, being witnessed by the law and the prophets; even the righteousness of God which is by faith of Jesus Christ unto all and upon all them that believe; for there is no difference, for all have sinned, and come short of the glory of God, being justified freely by his grace through the redemption that is in Christ Jesus, whom God hath set forth. Jesus Christ is propitiation through faith in his blood, to declare his righteousness for the remission of sins that are past, through the forebearance of God, to declare at this time his righteousness; that he might be just, and the justifier of him which believeth in Jesus. Where is the boasting then? It is excluded. By what law? Of works? Nay; but by the law of faith. Therefore we conclude that a man is justified by faith without the deeds of the law." Romans 3:20-28 [Emphasis mine]
If you have followed the line of faith from Abraham through to New Testament faith, you will understand the conclusion of the matter, as summed up in that quote from the book of Romans. Let no Christian give a moment's thought to his righteousness (for he has none), but bend his knees at the righteousness of God. As chapter 4 goes on to say:
"For if Abraham were justified by works, he hath whereof to glory; but not before God. For what saith the scripture? 'Abraham believed God, and it was counted unto him for righteousness'. Now to him that worketh is the reward not reckoned of grace, but of debt. But to him that worketh not, but believeth on him that justifieth the ungodly, his faith is counted for righteousness. Even as David also describeth the blessedness of the man unto whom God imputeth righteousness without works, saying, 'Blessed are they whose iniquities are forgiven, and whose sins are covered. Blessed is the man to whom the Lord will not impute sin'." (Psalm 32:1-2) [Emphasis mine]
The scriptures say it all. It is in the gift of God to bestow this standing, for our God is Righteous in his very Being. Let nobody dwell on their own supposed righteousness, for they have none. That is why Jesus said to seek first the kingdom of God and HIS righteousness. Those who do so, by faith, will enter into that, for Faith that is rooted in the Righteousness of God is the only kind of faith that God acknowledges.
The kind of 'faith' that justifies is our personal TRUST in Jesus Christ to be our Lord and Savior, not simply agreeing intellectually to a few statements about who Jesus is (see my other answer). The Bible teaches us how we should obey Christ as Lord and how we should be thankful to Christ as Savior. That much is clear.
But how do we know we have that 'faith'? I'm afraid, the only reliable answer is to examine ourselves whether we have at least the beginnings of joyfulness and gratitude in having Jesus to be our Lord and Savior. We should examine whether we actually trust Jesus as the most valuable treasure compared to anything else in the world. I know this is an indirect answer, but given that God is responsible to give us this saving faith in the first place, this is the best answer.
We cannot produce this saving faith by our efforts alone. The formula is "salvation by grace through faith"; grace by definition is a gift which produces faith in us that we can observe. Therefore, all we can do is to cry out to the right person, Jesus, "HELP ME!". Then we do our best to study the Bible, repent, love, etc., and observe God's work in producing saving faith within our heart. I highly recommend Eleonore Stump's answer to "How Does Salvation Work?" which characterizes the "HELP ME" cry as "cease resisting": admitting we need help and opening the door for God to enter us and help us.
When I was in college I used to struggle with this very question, assuming that "having the correct faith" is the key to salvation since Protestantism very loudly declares that "works" should not be a factor. Decades since then, I realize that my worry was misplaced because I was oblivious to many fine distinctions in Christian theology, not to mention that different Christian traditions have slightly different interpretations of the same Bible verses.
Here are some important distinctions you need to be aware of in your study:
(Saving) Faith in Protestantism is mainly connected with justification only, not to the whole salvation (which includes sanctification), although of course faith has a role throughout salvation, not a one time event (see What is the Christian life supposed to be like?). See the Wikipedia article sola fide to get a detailed overview of various Christian traditions' understanding of justification as well as how various Protestant denominations connects justification to the whole of salvation.
It's the prevalence of the Reformed tradition within the dominant evangelical movement that makes a lot of people mistakenly equate justification with the whole of salvation since some of them so emphasize monergism that they view sanctification as monergistic as well (see Is Sanctification Monergistic or Synergistic? A Reformed Survey).
It turns out that the most important element in faith is actually God's initiative. Depending on the Protestant theology it can even be 100% gift without any role for human free will (Reformed), but in other theology God gives Prevenient grace to prepare free will for faith (Wesleyan). Therefore, conscious human effort in the justification stage is minimal or non-existent, and we can only observe the effect of God's work in us such as:
- finding ourselves to agree with what the Bible says
- being moved to repentance
- be filled with gratitude
- be desirous for loving God and neighbors according to Jesus's commandments like forgiving and loving enemies
Bible verses that imply we need to do work (James 2:14-26) or Bible verses that imply we need to increase faith (see article What is Great Faith and Little Faith) should be interpreted with the sanctification stage of the salvation process: the increase of our TRUST in God in our Christian life AFTER conversion. This is where "faith as spiritual gift" has a shining role as well.
Another connected issue that I heard a lot in college when I asked around whether my faith is "correct" / "sufficient" was actually about Assurance of Salvation, which, again, not justification proper since it is part of the doctrine of the Perseverance of the saints. It turns out that the people around me were Reformed, so their answer was asking me back whether I have doubts whether I was saved! In their view, if I am one of the elect, faith (given through irresistible grace) will inevitably led me to produce evidences of faith in my life which includes doing good deeds with cheerful heart, peace, joy, etc. No doubt I was perplexed and even upset because I was expecting that they gave me the criteria to determine what kind of faith I should be having in the first place! In other words, they answered my question about justification with assurance, which to me at the time sounded like begging the question.
Faith that works by love and purifies the soul. Faith that takes God at His word when He said:
Jude 1:24 Now unto him that is able to keep you from falling, and to present you faultless before the presence of his glory with exceeding joy,
Faith that believes that God indeed is able to save us from the horrible pit into which we have fallen, faith that believes that the blood of Christ is all sufficient to cleanse us no matter how far we have fallen, faith that follows:
Heb 7:25 Wherefore he is able also to save them to the uttermost that come unto God by him, seeing he ever liveth to make intercession for them.
Faith that believes:
John 6:37 ... him that cometh to me I will in no wise cast out.
Faith that believes God when he said:
Rev 22:14 Blessed are they that do his commandments, that they may have right to the tree of life, and may enter in through the gates into the city.
Faith that believes that even though things seem impossible from human view, yet:
Matt 19:26 But Jesus beheld them, and said unto them, With men this is impossible; but with God all things are possible.
Faith that takes Christ at His word when He said:
Matt 5:48 Be ye therefore perfect, even as your Father which is in heaven is perfect.
And shows it in works...
Faith that says "The Lord is my Righteousness":
Jer 23:6 In his days Judah shall be saved, and Israel shall dwell safely: and this is his name whereby he shall be called, THE LORD OUR RIGHTEOUSNESS.