St. Alphonsus of Liguori, [*Theologia Moralis* lib. 6, 942, p. 690][1]:

>A husband or wife rendering himself or herself impotent, even by licit means (e.g., fasts, etc.), sins gravely. If one cannot otherwise perform the [[marriage] debt][2], it is licit to not observe the fasts of the Church.<br><sub>Peccat graviter vir vel mulier, si se impotentem reddat, etiam mediis alias licitis, v.gr. ieiuniis etc. invita comparte: imo si aliter non posssit reddere debitum, licite non observat ieiunia ecclesiæ.</sub>

cf. [*ibid.* lib. 3, 1034 §4, p. 768][3]: 

>But if the man observing the fasts of the Church, and other moderate [fasts] from devotion, is rendered less potent to render the [marriage] debt, he is not bound to refrain from them; because he is not bound to make himself more potent by lacking spiritual [goods].<br><sub>Si autem vir servans ecclesiæ ieiunia, et alia moderata ex devotione, redderetur minus potens ad debitum reddendum, non teneretur ab eis abstinere; quia non tenetur cum spirituali iactura illa omittere ut potentiorem se reddat</sub>

Impotency ≠ infertility, but they are related. An impotent husband is infertile, but an infertile husband isn't necessarily impotent.

 - [Impotence][4] (*impotentia coeundi*) means the inability to perform the marital act.<br><sup>A husband or wife invalidly marry if they know they cannot perform the marital act; impotency is [diriment impediment][5] to marriage.</sup>

 - Infertility or sterility (*impotentia generandi*) means a child does not result from a marital act.

Now, if one intentionally eats a diet to render himself impotent to the extent he becomes infertile, but is still able to perform the marital act, that is no different than *artificial* contraception, condemned in [*Casti Connubii*][6] §56:

>any use whatsoever of matrimony exercised in such a way that the act is **deliberately frustrated** in its natural power to generate life is an offense against the law of God and of nature, and those who indulge in such are branded with the guilt of a grave sin.<br><sub>quemlibet matrimonii usum, in quo exercendo, actus, **de industria hominum**, naturali sua vitæ procreandæ vi **destituatur**, Dei et naturæ legem infringere, et eos qui tale quid commiserint gravis noxæ labe commaculari.</sub>


  [1]: https://archive.org/details/theologiamoralis02ligu_0/page/690/mode/2up
  [2]: https://www.catholicculture.org/culture/library/dictionary/index.cfm?id=32969
  [3]: https://archive.org/details/theologiamoralis01ligu_0/page/768/mode/2up
  [4]: https://www.catholicculture.org/culture/library/dictionary/index.cfm?id=34126
  [5]: https://www.catholicculture.org/culture/library/dictionary/index.cfm?id=33109
  [6]: https://www.vatican.va/content/pius-xi/en/encyclicals/documents/hf_p-xi_enc_19301231_casti-connubii.html