The phrase "he that is without sin" is a translation of a single Greek word, anamartetos, which means sinless or faultless or without flaw.
It does not refer only to fornication, Jesus is effectively saying let the perfect one among you throw the first stone.
What happened next is that the scribes and pharisees went away. It is not clear whether they ever had any intention of stoning the woman in the first place. The reason they brought her to Jesus was to test Him, to see what reply He would give. If He said she should not be stoned they could accuse Him of going against the law, but if He said she should be stoned He would lose His reputation for gentleness. St Augustine, in Tractate 33 on St John's gospel said
if he shall approve her being stoned, he will not show his gentleness; if he consent to let her go, he will not keep righteousness. But, say they, that he may not lose the reputation of gentleness, for which he has become an object of love to the people, without doubt he will say that she must be let go. Hence we find an opportunity of accusing him, and we charge him as being a transgressor of the law: saying to him, You are an enemy to the law; you answer against Moses, nay, against Him who gave the law through Moses;
So Jesus in allowing he who is without sin to cast the first stone meant he who was perfect, not simply anyone who has never committed adultery or fornication. In going away the scribes and pharisees were not making a public admission of adultery - if that had been Christ's meaning the result could have been different.