My understanding is that murder is a sin because I don't have rights on another person's life in the same way that another person doesn't have rights over my life. Likewise stealing would be a sin because I don't have rights over another person's property nor would I like anybody else to take what is mine. On the other hand smiling would not be a sin because I don't encroach on another person's right when smiling and they may actually like being smiled upon.
In the Old Testament David (considered a prophet) did not sin when he engaged with concubines and slave girls who were not legally wives. If adultery is defined as having sex with another man's wife, David has not broken the law. Or did he? However today if two (unmarried) consenting adults have sex, that's adultery by Biblical standards. From my understanding its encroaching on another person's right that seems to define sin. Here I don't see an encroachment of the rights of any person.
If a married person is involved in extra marital sex he sins, because he has broken his marital vows. Why is the sexual act a sin when two people have fun by mutual consent and not encroaching on each others freedom?