St. Alphonsus of Liguori, Theologia Moralis lib. 6, 942, p. 690:
A husband or wife rendering himself or herself impotent, even by licit means (e.g., fasts, etc.), sins gravely. If one cannot otherwise perform the [marriage] debt, it is licit to not observe the fasts of the Church.
Peccat graviter vir vel mulier, si se impotentem reddat, etiam mediis alias licitis, v.gr. ieiuniis etc. invita comparte: imo si aliter non posssit reddere debitum, licite non observat ieiunia ecclesiæ.
cf. ibid. lib. 3, 1034 §4, p. 768:
But if the man observing the fasts of the Church, and other moderate [fasts] from devotion, is rendered less potent to render the [marriage] debt, he is not bound to refrain from them; because he is not bound to make himself more potent by lacking spiritual [goods].
Si autem vir servans ecclesiæ ieiunia, et alia moderata ex devotione, redderetur minus potens ad debitum reddendum, non teneretur ab eis abstinere; quia non tenetur cum spirituali iactura illa omittere ut potentiorem se reddat
Impotency ≠ infertility, but they are related. An impotent husband is infertile, but an infertile husband isn't necessarily impotent.
Impotence (impotentia coeundi) means the inability to perform the marital act.
A husband or wife invalidly marry if they know they cannot perform the marital act; impotency is diriment impediment to marriage.
Infertility or sterility (impotentia generandi) means a child does not result from a marital act.
Now, if one intentionally eats a diet to render himself impotent to the extent he becomes infertile, but is still able to perform the marital act, that is no different than artificial contraception, condemned in Casti Connubii §56:
any use whatsoever of matrimony exercised in such a way that the act is deliberately frustrated in its natural power to generate life is an offense against the law of God and of nature, and those who indulge in such are branded with the guilt of a grave sin.
quemlibet matrimonii usum, in quo exercendo, actus, de industria hominum, naturali sua vitæ procreandæ vi destituatur, Dei et naturæ legem infringere, et eos qui tale quid commiserint gravis noxæ labe commaculari.