From the Code of Canon Law:
Can. 751 Heresy is the obstinate denial or obstinate doubt after the reception of baptism of some truth which is to be believed by divine and Catholic faith; apostasy is the total repudiation of the Christian faith; schism is the refusal of submission to the Supreme Pontiff or of communion with the members of the Church subject to him.
This defines what heresy, apostasy and schism are. Now, the Church cannot, and could never, enact civil or criminal punishment for anything it describes as a contravention of its own laws. All punishment she is allowed to mete out is canonic in nature. And thus, further on, we have:
Can. 1364 §1. Without prejudice to the prescript of can. 194, §1, n. 2, an apostate from the faith, a heretic, or a schismatic incurs a latae sententiae excommunication; in addition, a cleric can be punished with the penalties mentioned in can. 1336, §1, nn. 1, 2, and 3.
§2. If contumacy of long duration or the gravity of scandal demands it, other penalties can be added, including dismissal from the clerical state.
So, heresy, apostasy and schism incur a latae sententiae (that is, automatic) excommunication. This makes sense, since the person, by making a choice (αἵρεσις hairesis) to believe something different from what the Church believes, places themself outside of her communion. Additionally, a priest can suffer other penalties, according to Canon 1336:
Can. 1336 §1. In addition to other penalties which the law may have established, the following are expiatory penalties which can affect an offender either perpetually, for a prescribed time, or for an indeterminate time:
1/ a prohibition or an order concerning residence in a certain place or territory;
2/ privation of a power, office, function, right, privilege, faculty, favor, title, or insignia, even merely honorary;
3/ a prohibition against exercising those things listed under n. 2, or a prohibition against exercising them in a certain place or outside a certain place; these prohibitions are never under pain of nullity;
4/ a penal transfer to another office;
5/ dismissal from the clerical state.
§2. Only those expiatory penalties listed in §1, n. 3 can be latae sententiae.
So a heretic priest can be ordered to be removed from his parish, lose an office, and/or be forbidden from exercising certain activities such as teaching.
However, outside of the merely administrative sanctions the Church can take against a heretic (forbidding them from teaching at a Catholic institution, ordering them to move to a certain place — this last one mostly in case of religious), she does not have any power to constrain them to accept her rulings, and has merely a power of authority.
This was true in the Middle Ages, too: whenever someone was tortured or executed for heresy, it was never by the Church but always by the State — in those days, heresy was often a criminal offense, too. This is true both for Catholic States like Spain and Protestant ones like Prussia and England.
So the Inquisition tried the defendant, and either found them guilty or innocent, and then turned them over to the civil authorities in the former case.