The process is called convalidation
The diocese of Detroit has a primer on this, which you can read here, but the basics remain the same.
The presumption of any marriage is that it is valid unless challenged. (Code of Canon Law).
Catholics are obliged to, when they get married, take the effort to ensure that the marriage is sacramental. (Code of Canon Law)
Each case has it's own details and nuances. That is a critical point to understand: the pastor, and in some cases the diocese, investigates a marriage that is claimed to not be sacramental and will uncover what defects, if any, are present (Code of Canon Law).
In some cases a decree of nullity of a previous marriage is first needed, and in other cases (as with a friend of mine who was Catholic, and his wife was Catholic, but they had a civil marriage) a course of instruction and formation is called for (in his and her case, taking the RCIA class through to completion) and then having their marriage convalidated.
There are a lot of points in between.
Insofar as "theologians" go: the Catholic Church holds that the man and woman confer the sacrament upon each other (Catechism of the Catholic Church)1 and the clergy is present (as well as two witnesses) for a variety of reasons - one of which is so that the Church can attest to the truth of the sacrament having been conferred.
About your "hypothetical" case:
This is a situation: A non-practicing Catholic couple "divorces" and re-marry by state.
I presume they get re-married to other people?
They end up having families, let's say with 3 or more kids with
another husband/wife. According to the Catholic teaching, they live in adultery.
Strictly speaking, yes, unless a decree of nullity is arrived at with a favoarble finding.
Now, they start to be interested in spiritual life and undergo what we call a conversion. But they are stuck because they can't just split and leave their kids or whatever dependencies they have.
The simple answer is (even though it is not simple to do):
Convince their spouse to become Catholic. Apply for a decree of nullity. Get the new marriage convalidated. It only takes (from months to years) to go through this.
I know the ideal would be to get an annulment and solve this problem. But let's say they aren't entitled to an annulment.
Entitlement isn't the issue. Each case is investigated for its own particulars. Nobody is entitled to a decree of nullity; such a decree is the finding of a tribunal. The answer can be yes or it can be no. From your stated case, I am not convinced that the two couples have attempted to get decrees of nullity. There may indeed have been a defect that could show that the first marriage was non sacramental. You can't know that until you bother to go through the process.
My question is what does the church offer to such couples as a
solution?
I presume that you mean the two new couples? You do not specify that all four people are Catholic.
Suggesting to live like a brother and sister isn't going to work unless they stop living in the same household but that might not be ideal for kids (I guess) or it can still cause a scandal.
For some couples it may work, for others it will not. I've seen both cases ... the outcome varies.
Is splitting couples really the only way to go if they want to live a sacramental life? I know this happened to St. Augustine but his situation wasn't that complicated. Do we have other examples from the past where families needed to separate in order to return back to living in the state of grace?
There is no need to split up the couple, and for sure In My Experience the Church will not promote splitting up the family.
This one hits close to home.
I know a couple (I worked with the husband for seven years) who are still married, in this exact situation, and they come to our church regularly. The two adults do not receive communion, but in all other ways are active members of our parish and are raising their kids to be Catholic. That is a solution. I have encouraged him to pursue convalidation but between the two of them they have not yet chosen that path for their own reasons.
I challenge the frame of this question: this isn't a matter of what theologians write about, this is a matter of sacramental administration under the Code of Canon Law.
How do I know this?
I spent about 6 years in the RCIA ministry and part of that time was spent in helping people through the convalidation process, and the often difficult process of determining whether or not a decree of nullity for a previous marraige was needed. Each case was addressed on its own merits.
1 Man and woman confer on each other ...
{CCC} II. THE CELEBRATION OF MARRIAGE
1621 In the Latin Rite the celebration of marriage between two Catholic faithful normally takes place during Holy Mass, because of the connection of all the sacraments with the Paschal mystery of Christ. In the Eucharist the memorial of the New Covenant is realized, the New Covenant in which Christ has united himself for ever to the Church, his beloved bride for whom he gave himself up. It is
therefore fitting that the spouses should seal their consent to give themselves to each other through the offering of their own lives by uniting it to the offering of Christ for his Church made present in the Eucharistic sacrifice, and by receiving the Eucharist so that, communicating in the same Body and the same Blood of Christ, they may form but "one body" in Christ.
1622 "Inasmuch as it is a sacramental action of sanctification, the liturgical celebration of marriage . . . must be, per se, valid, worthy, and fruitful." It is therefore appropriate for the bride and groom to prepare themselves for the celebration of their marriage by receiving the sacrament of penance.
1623 According to Latin tradition, the spouses as ministers of Christ's grace mutually confer upon each other the sacrament of Matrimony by expressing their consent before the Church. In the tradition of the Eastern Churches, the priests (bishops or presbyters) are witnesses to the mutual consent given by the spouses, but for the validity of the sacrament their blessing is also necessary.
1624 The various liturgies abound in prayers of blessing and epiclesis asking God's grace and blessing on the new couple, especially the bride. In the epiclesis of this sacrament the spouses receive the Holy Spirit as the communion of love of Christ and the Church. The Holy Spirit is the seal of their covenant, the ever available source of their love and the strength to renew their fidelity.