St. Alphonsus of Liguori, Theologia Moralis lib. 6, 942, p. 690:
A husband or wife rendering himself or herself impotent, even by licit means (e.g., fasts, etc.), sins gravely. If one cannot otherwise perform the [marriage] debt, it is licit to not observe the fasts of the Church.
Peccat graviter vir vel mulier, si se impotentem reddat, etiam mediis alias licitis, v.gr. ieiuniis etc. invita comparte: imo si aliter non posssit reddere debitum, licite non observat ieiunia ecclesiæ.
Impotency ≠ infertility, but they are related. An impotent husband is infertile, but an infertile husband isn't necessarily impotent.
Impotence means the inability to perform the marital act.
A husband or wife invalidly marry if they know they cannot perform the marital act; impotency is diriment impediment to marriage.Infertility (sterility) means a child does not result from a marital act.
Now, if one intentionally eats a diet to render himself impotent to the extent he becomes infertile, but is still able to perform the marital act, that is no different than artificial contraception, condemned in Casti Connubii §56:
any use whatsoever of matrimony exercised in such a way that the act is deliberately frustrated in its natural power to generate life is an offense against the law of God and of nature, and those who indulge in such are branded with the guilt of a grave sin.
quemlibet matrimonii usum, in quo exercendo, actus, de industria hominum, naturali sua vitæ procreandae vi destituatur, Dei et naturae legem infringere, et eos qui tale quid commiserint gravis noxae labe commaculari.