First, there is the argument from the ancient languages of Aramaic and Greek and how the words were used:
First it is important to note that the Bible does not say that these "brothers and sisters" of Jesus were children of Mary.
Second, the word for brother (or sister), adelphos (adelpha) in Greek, denotes a brother or sister, or near kinsman. Aramaic and other semitic languages could not distinguish between a blood brother or sister and a cousin, for example. Hence, John the Baptist, a cousin of Jesus (the son of Elizabeth, cousin of Mary) would be called "a brother (adelphos) of Jesus." In the plural, the word means a community based on identity of origin or life. Additionally, the word adelphos is used for (1) male children of the same parents (Mt 1:2); (2) male descendants of the same parents (Acts 7:23); (3) male children of the same mother (Gal 1:19); (4) people of the same nationality (Acts 3:17); (5) any man, a neighbor (Lk 10:29); (6) persons united by a common interest (Mt 5:47); (7) persons united by a common calling (Rev 22:9); (8) mankind (Mt 25:40); (9) the disciples (Mt 23:8); and (10) believers (Mt 23:8). (From Vine's Expository Dictionary of Biblical Words, Thomas Nelson, Publisher.)
The point being, just because the word "brother" means something specific in English does not mean that when a translation uses that word it necessarily matches the same idea.
Another reason cited by Catholics in justifying that Mary was a perpetual Virgin is John 19:
When Jesus saw his mother and the disciple there whom he loved, he said to his mother, “Woman, behold, your son.”
Then he said to the disciple, “Behold, your mother.” And from that hour the disciple took her into his home.
This passage tells us:
"The disciple" (John) was not Mary's son, because Mary did not live in John's home before but did after
Mary either had no sons, or at the very least had no living sons, because otherwise it would be unusual for Mary to live with John instead of these sons.
Another data point: An article on Catholic Answers also addresses this question. It states, for example, that James the brother of Jesus was an Apostle, but that of the two apostles named James, one had Alphaeus as his father, the other was the son of Zebedee. One can conclude from this that "brother" at least for James meant something other than sharing biological parents.