To answer the O.P.'s question directly, the maximum penalty for breaking monastic vows is essentially expulsion from the monastery (which entails a dispensation from the vows).
For the benefit of readers, in the Catholic Church, “monks” are those men who live in a monastic community, which entails a certain separation from the world and a dedication to prayer and work—the so called “contemplative” vocation—as opposed to active apostolate. The equivalent for women are called “nuns.” For example, the Benedictines, Cistercians, and Trappists are orders of monks and nuns. There are, however, other entities in the Church that share with monasticism a special consecration to God, but entail a more active ministry; these groups include religious congregations (for example, the Jesuits), secular institutes of consecrated life, and societies of apostolic life (for example, the Paulist Fathers).
As a practical matter, however, membership in all these groups entails certain promises or vows, which are dealt with in essentially the same way.
By “monastic vows” the Church means the solemn vows of poverty, chastity, and obedience that monks make when they formally enter their monastery. The Code of Canon Law (CIC) describes vows as follows:
A vow, that is, a deliberate and free promise made to God about a possible and better good, must be fulfilled by reason of the virtue of religion (Can. 1191 §1).
In the case of monks, they take public and solemn vows, as described by Can. 1192:
§1. A vow is public if a legitimate superior accepts it in the name of the Church; otherwise, it is private.
§2. A vow is solemn if the Church has recognized it as such [as in the case of monastic orders]; otherwise, it is simple.
The taking of these vows constitutes what is called religious profession, described in Canon Law as follows:
By religious profession, members assume the observance of the three evangelical counsels [i.e., poverty, chastity, and obedience] by public vow, are consecrated to God through the ministry of the Church, and are incorporated into the institute with the rights and duties defined by law (Can. 654).
Note that religious profession takes places in two stages: temporary profession and perpetual profession:
Temporary profession is to be made for a period defined in proper law; it is not to be less than three years nor longer than six (Can. 655).
When the period for which profession was made has elapsed, a religious [here, referring to anyone who takes these kinds of vows: monks, nuns, members of religious orders, and so on] who freely petitions and is judged suitable is to be admitted to renewal of profession or to perpetual profession; otherwise, the religious is to depart (Can. 657 §1.).
What happens to a religious or consecrated person when he commits a serious fault against his vows depends on whether he has only made his temporary profession, and also on the seriousness of the offense.
The general practice nowadays, in the face of such a violation, would be simply to dismiss a monk or religious who has made only his temporary profession, or else simply not admit him to perpetual profession. (In the former case, the superior general is authorized to dispense his subjects from their temporary vows; see Can. 688.)
For monks and religious who have made their perpetual profession, there is generally more of an attempt to try to rectify the situation before proceeding to dismissal, because only the gravest of causes should lead to an indult of departure from an institute (Can. 691 §1).
In very extreme cases (spelled out in Canons 694-704), monks and religious can be expelled from their institute.
Offenses that lead to dismissal by their very commission
- Notoriously defecting from the Catholic Faith (Can. 694).
- Attempting marriage (Can. 694).
Offenses that generally require dismissal
Other grave offenses against the sixth commandment, such as concubinage, abuse of minors, and so on (Can. 1395).
Committing homicide (Can. 1397).
Procuring [i.e., directly helping someone to have] an abortion (Can. 1398).
Other possible reasons for dismissal
Canon 696 §1 details other reasons for dismissal:
A member can also be dismissed for other causes provided that they are grave, external, imputable, and juridically proven such as: habitual neglect of the obligations of consecrated life; repeated violations of the sacred bonds; stubborn disobedience to the legitimate prescripts of superiors in a grave matter; grave scandal arising from the culpable behavior of the member; stubborn upholding or diffusion of doctrines condemned by the magisterium of the Church; public adherence to ideologies infected by materialism or atheism; the illegitimate absence mentioned in ⇒ can. 665, §2, lasting six months; other causes of similar gravity which the proper law of the institute may determine.
In summary, monks (and other religious, consecrated persons, and members of societies of apostolic life) face in essence a maximum penalty of expulsion from their institute or society (and, if they are priests or deacons, also reduction to the lay state) for grave offenses against their vows.
They would not be obliged to copy manuscripts as a penance, and would certainly not be subject to cruel punishments (such as being walled up). If such things occurred in the past, they are now entirely forbidden by Canon Law.