On what biblical basis can one argue that: "marriage is a covenant before God between a man and a woman -- and has nothing to do with the state" ?
Justification for Marriage without State
The first marriage in the Bible involved a man, a woman, and God.
Genesis 2:23 And Adam said, This is now bone of my bones, and flesh of my flesh: she shall be called Woman, because she was taken out of Man.
This might be seen as the minimum requirement. Various cultures at various times have added layers of family, friends, church, village, wedding planners, caterer, photographers, musicians, etc.
The government started to get involved when things that might be actionable in court began to come into play like property rights, inheritance, child custody, etc.
There are many people who not only do not have the government involved, but also do not want God involved. For people like this, governments often pass "common law" legislation so that rights can be adjudicated in court in case of death, divorce, or other changes in legal status.
The precept of "obey the law" could be applied to marriage unless the law extended into areas such as forcing you to marry someone or telling you you could not marry someone.
Romans 13:1 Let every soul be subject unto the higher powers. For there is no power but of God: the powers that be are ordained of God.
I can understand the desirability of being free of government influence. However, I am not aware of any Biblical basis to reject legal requirements such as taxing through licensing or registering a marriage.
With or without the State1
Marriage is a contract and is by its very nature above human law. It was instituted by God, is subject to the Divine law, and cannot for that reason be rescinded by human law. Source: Marriage instituted by God | Moral and Canonical Aspect of Marriage | New Advent.
In the same article linked above, the section under Marriage in the Christian dispensation begins
Christ revoked the dispensation granted in the Mosaic law. He promulgated the original Divine law of monogamic and indissoluble marriage; in addition, He raised marriage to the dignity of a sacrament (Genesis 2:24; Matthew 19:3 sqq.; Luke 16:15 sqq.; Mark 10:11 sqq.; 1 Corinthians 7:2 sqq.).
The Biblical references are the biblical basis.
1. This does not mean that the involvement of the State is undesirable, only that such an involvement must be subordinate to God's law. After all marriage the way God intended is a good for the individual, family, society and the State.