Preface
My answer will stick to the general position of Trinitarian Christianity (conforming to the Nicene Creed) so that it applies to all mainstream denominations including not just Protestants but also Pentecostals, Catholics, and Eastern/Oriental Orthodox, by avoiding the debates on justification, imputed vs. imparted vs. infused righteousness, role of the sacraments as means of grace, monergism vs. synergism, Pentecostal baptism in the spirit / speaking in tongues, perseverance of the saints, etc. I'm very mindful in wording this answer so the answer transcends those differences.
I am also very mindful of possible wrong readings of this answer. I'm in no way advocating obeying Jesus's New Covenant commandments (cf. Sermon on the Mount, New Commandment) to earn our salvation or to earn righteousness. No, Sir / Madam! This answer takes a firm stand in sola gratia.
My focus is on the conscious operation of our mind, will, and desires in response to the grace given to us by the Holy Spirit after we have come to faith, thus taking the assumption that we have already been born again, by framing the post-conversion life "in the Spirit" as responsibly conducting our life as a believer in union with Christ. We are not like robots being controlled by the Holy Spirit. The way grace operates in us is something that we PASSIVELY receive unconsciously (although we can pray for more grace) but at the same time we can exercise our will either to follow Jesus or to reject Jesus (cf. "take up [our] cross and follow [Him]", Matt 16:24 and "take my yoke", Matt 11:28-30). In other words, grace empowers us as we obey as a grateful response to God's gift of salvation, NOT to score "points". Otherwise how are we going to make sense of the active verbs that Jesus used in Matthew (as I quote them copiously below) and how are we supposed to understand what making Jesus our Lord means?
Life in the Spirit
Christians who have been born again receive the indwelling Holy Spirit, cf. Rom 8:9 (NIV):
You, however, are not in the realm of the flesh but are in the realm of the Spirit, if indeed the Spirit of God lives in you. And if anyone does not have the Spirit of Christ, they do not belong to Christ.
As Christians we are supposed to live with a mind governing our desires so they are aligned with what the Spirit desires (Rom 8:5-6), therefore submitting to God's law (Rom 8:7), which produces life and peace (Rom 8:6b).
"In the Spirit" in Rom 8:9 and Gal 5:16 simply means allowing the indwelling Holy Spirit to lead us in our daily living. See the GotQuestions.org article: What does it mean to walk in the Spirit? See also Calvin's commentary on Rom 8:9 and Calvin's Commentary on Gal 5:16.
Since now we have the indwelling Holy Spirit, we have another power in our heart that makes our heart tender and obedient to God's law (cf. Ezekiel 36:26) enabling us to obey (cf. "Now we can serve God ... in the new way of living in the Spirit", Rom 7:5-6). Our heart is no longer stony and rebellious like the Israelites. God promises to transform us into His Son's image in our sanctification (2 Cor 3:18). In response, we are supposed to cooperate with the Holy Spirit's work in us, part of what Paul meant by "continue to work out [our] salvation with fear and trembling, for it is God who works in you to will and to act in order to fulfill his good purpose" (sanctification) (Phil 2:12-13). Christ also commanded us to stay united with him, connected to the vine, cf. "remain in me, as I also remain in you") so we can be fruitful (John 15:1-4).
Until resurrection, we need to be on our guard by cooperating with this new power (the indwelling Holy Spirit) so we don't let ourselves to be controlled by the old power that tries actively to derail us (Rom 7:23). To help us "stand firm against all strategies of the devil" who tries to enslave us again, God provides us with an armor that we can "put on", Eph 6:10-17. We can also protect ourselves by praying "in the spirit" (Eph 6:18) which means praying according to the Spirit's leading so the Holy Spirit helps us in our weakness (Rom 8:26-28). See GotQuestions.org article which answers what is praying in the Spirit?
New covenant fulfillment of the OT life guidance
Walking "in the Spirit" can be seen as the New Covenant fulfillment of the OT life guidance. This represents an NT reading of the OT wisdom literature since OT is still relevant to Christians! Otherwise, how do we make sense of the verbs in the books of Psalms and Proverbs?
Make God's word to become a lamp for our feet and a light to our path (Ps 119:105). Also "Trust in the LORD with all your heart and lean not on your own understanding; in all your ways submit to him, and he will make your paths straight" (Prov 3:5-6). In the New Covenant we follow Jesus, the "light of the world" (John 8:12), "the way, and the truth, and the life" (John 14:6). "In the Spirit" means letting the Holy Spirit help us govern our mind and will to become Salt and light in the world (cf. Matt 5:13-16), responding to the moral situations of our lives to produce a "path" of purpose-driven life that Jesus would have approved (cf. "narrow the road that leads to life", Matt 7:14 and "seek first his kingdom and his righteousness", Matt 6:33), culminating in Jesus's praise when we die "Well done, good and faithful servant!" (Matt 25:23) instead of Jesus's rebuke "I never new you" (Matt 7:21-23). In other words, walking "in the Spirit" means taking up our cross as our yoke (Matt 16:24, Matt 11:28-30) as we follow Jesus, our Lord (Rom 10:9).
Seeing straight ahead and walking the straight path to keep from evil; evil symbolized by "the crooked way": exiting the straight path by turning right or left (Prov 4:20:27). This is about making the right choices as we construct the "path of our lives" to follow God's plan for our lives (Prov 16:1-9). "In the Spirit" means having our mind cooperate with the prompting of the Holy Spirit to discern which of the two paths to choose: the way of the flesh (the "crooked path", allow sinful desires to shape our words/thoughts/actions, cf. Gal 5:19-21) or the way of the Spirit (the "straight path", choose words/thoughts/actions that produce the fruits of the Spirit, cf. Gal 5:22-23). The "straight path" / the "way of the Spirit" is the same as doing the greatest commandments "Love the LORD your God with all your heart and with all your soul and with all your mind. etc." (Matt 22:37-40) which Jesus reiterated with his disciples on the night of his betrayal: "So now I am giving you a new commandment: Love each other. Just as I have loved you, you should love each other" (John 13:34).
When we become fearful or discouraged (our wilderness), trust that God is in front of us as well as our rear guard ("You go before me and follow me", cf. Deut 31:8, Ps 139:5, Isa 52:12). The Israelites had a vivid memory of the visible pillars of cloud and fire (Ex 13:21) when God led them in the wilderness after the same cloud protected them from Pharaoh's army (Ex 14:19-20). "In the Spirit" means trusting what Jesus said: "even the very hairs of your head are all numbered" (Matt 10:29-31) and what Paul said in Rom 8:38-39: "For I am convinced that neither death nor life, neither angels nor demons, neither the present nor the future, nor any powers, neither height nor depth, nor anything else in all creation, will be able to separate us from the love of God that is in Christ Jesus our Lord." The opposite of "In the Spirit" is quitting the sometimes inconvenient/costly discipleship by "going back to Egypt" (backsliding, or even apostatizing) and thus becoming "slave to sin" again.
Other verses you mentioned
"In the spirit" in Rom 2:17-29 is part of Paul's discussion of how the Jews cannot rely on physical circumcision alone. They need "circumcision of the heart, by the Spirit" (cf. Rom 2:29, NIV), which God provided in the New Covenant (cf. Jer 31:33). See Calvin's Commentary on Rom 2:29.
"In the Spirit" in Acts 19:21 was Paul's application to trust the Holy Spirit in his missionary journey. NIV translates this verse as "Paul decided to go to Jerusalem". As the Israelites followed the pillars of cloud and fire in their journey to the promised land, this was Paul and other apostles following the leading of the Spirit to open doors in the mission field. This is supposed to be our model as well for our life's journeys. See Calvin's commentary on Acts 19:21.
