I have not read Boswell's book, but I have found a number of reviews of it online (1 2 3).
Based on the reviews, Boswell's argument seems to be threefold.
First, he points out that the definition of marriage has changed over the centuries. Even today there are different kinds of marriage. A couple can get a civil marriage (performed by a judge) or a religious marriage (performed by ordained clergy). In some countries a clergy person is authorized to sign the legal paperwork so that one ceremony can satisfy both God and the law. In some locations, common-law marriage (a tradition going back to ancient times) is an option allowing a couple to get married without a ceremony.
In many places today a married person cannot marry anyone else without breaking their current contract (and even then, some churches would not allow a second marriage), but in ancient times polygamy was much more widely practiced. Solomon was said to have 700 wives (and also 300 concubines, which was somewhat akin to marriage but was a different type of relationship.)
Yet another type of marital relationship in ancient times was that of Greek soldiers, who were encouraged to pair up--the reasoning being that when they went into battle they would be fighting alongside their lovers, and would therefore fight more valiantly.
On the other end of the spectrum, many early Christians took a view (following Paul's logic in 1 Corinthians 7 to its extreme conclusion) that abstinence was preferable to sexual activity even within a marriage, as far as it was possible.
So Boswell cautions that we can't necessarily take 20th century (well, 21st century, now) preconceptions of marriage and assume that marriage was always like this.
Second, Boswell argues that the word adelphopoiesis (literally, "brother-making") was a euphemism for a ceremony analogous to a heterosexual marriage. He notes similarities; for example, the adelphopoiesis ceremony often included hand holding and a kiss, and concluded with a shared communion celebration. (The reviewers also note that there are differences, but don't elaborate.)
Certainly, the term is not intended literally. As one reviewer notes, only your parents can literally make a brother for you. But how it was generally understood by those who took part in adelphopoiesis ceremonies, we can't be entirely sure.
Finally, Boswell notes that both the church and the state began to take a stronger stance against homosexual behavior beginning in the 13th century. Boswell quotes 16th century philosopher Michel de Montaigne, who witnessed an adelphopoiesis ceremony: "...the Church of Saint John of the Latin Gate, in which some Portuguese some years before had entered into a strange 'brotherhood.' Two males married each other at Mass, with the same ceremonies we use for our marriages, taking Communion together, using the same nuptial Scripture, after which they slept and ate together." Civil authorities, alerted by Montaigne's essay, arrested and punished participants in the ceremony.
By the end of that century adelphopoiesis was banned throughout Europe. The doucments describing it were locked away, and in at least one case, the pages containing the adelphopoiesis ceremony were ripped from the book.
In summary, marriage has had many different meanings. So, too, "brotherhood" has another meaning. (But what is it?) Finally, adelphopoiesis was eventually banned by authorities.
Most scholars don't believe Boswell's case is very strong. It does appear that the adelphopoiesis ceremony may have, in the eyes of some couples, been seen as akin to marriage, but there's no clear evidence it was ever viewed that way by the church.