Say for example a Pope infallibly declares that Mary was not a virgin, and is currently rotting in Hell. This obviously contradicts established Catholic Dogma. What would happen in this case? Would the cardinals get together and kick the Pope out of his office? Is a heretical ex cathedra definition cause for an automatic loss of the Papal position which would lead to a sedevacantism situation?
What would happen if a pope were to be a formal heretic?
Formal heresy is obstinately denying Catholic truths, even after being warned.
If a pope were to define a "dogma" ex cathedra that contradicts Catholic teaching, this "dogma" would not be binding nor be protected by papal infallibility as defined at Vatican I. Definable Catholic dogmas must first be material dogmas (cf. Pope Pius XII's letter to all bishops regarding to the definability of the Dogma of the Assumption), i.e., truths connected to Revelation and believed by the Church in all times and space, either explicitly or implicitly. Even Popes are bound to Tradition.
Does the Pope lose his office?
The common opinion is yes, but when is another question:
There are two major camps among theologians regarding when. One (that of St. Robert Bellarmine and Suarez) says that he would immediately lose his office, even before being warned and given an opportunity to correct his error. The other (that of John of St. Thomas and Cajetan) says God would take the office away from him after the Church convicts him and declares him a heretic (or "vitandus" = "to be avoided").
If so, how is that formalised?
It is divine law that the Church must separate herself from heretics (cf. Gal. 1:8-9), so the Church has a duty to warn the pope, try him, and (if it comes to this) declare him a heretic and proceed to elect a valid pope.
Do the cardinals have the power to kick him out?
No, not even all the world's bishops have the power to depose a valid pope. To think that would be the heresy of Conciliarism, that Councils of bishops (sans the pope) have equal or greater authority than the pope.
An excellent recent work that covers all these issues (whether a pope can be a heretic; whether he can be elected if he was previously a heretic; whether a pope loses his office by proclaiming heresy; who can declare him a heretic; when he loses his office; how is he deposed, considering he has no superior on earth; etc.; etc.) is:
- True or False Pope? by John Salza & Robert Siscoe (St. Thomas Aquinas Seminary Press, Aug. 2015)
Another good one is:
- Can the Pope Go Bad?: A Call for the Resumption of a Debate Not Taken Seriously Since the 17th Century by the Brazilian intellectual A. V. Xavier da Silveira (Catholic Research Institute, Sep. 1999)
The question in the title is far broader than the question in the text.
To the question in the title "Can a pope be a heretic?", the answer is "Yes, except for the particular case addressed below.", and that hypothetical situation would not entail an automatic loss of office.
To the question in the text, rephrased as "What would happen if a pope defines a heretical doctrine on faith or morals by a definition which formally satisfies the requirements of infallibility (*) set by the Constitution "Pastor Aeternus" of Ecumenical Council Vatican I?" the answer is "That cannot really happen, and the only valid inference of it apparently happening is that the person in question was not really the pope at the moment of issuing the heretical definition."
(*) "we teach and define that it is a dogma divinely revealed: that the Roman Pontiff, when he speaks ex Cathedra, that is,
- when in exercise of the office of pastor and doctor of all Christians,
- by virtue of his supreme Apostolic authority,
- he defines a doctrine regarding faith or morals
- to be held by the universal Church,
by the divine assistance promised to him in blessed Peter, possesses that infallibility with which the divine Redeemer willed that his Church should be endowed for defining doctrine regarding faith or morals;" (bullet structure added for clarity)