The key passages on this are:
James 5:12 (ESV):
But above all, my brothers, do not swear, either by heaven or by earth
or by any other oath, but let your “yes” be yes and your “no” be no,
so that you may not fall under condemnation.
and
Matthew 23:16-22 (ESV):
“Woe to you, blind guides, who say, ‘If anyone swears by the temple,
it is nothing, but if anyone swears by the gold of the temple, he is
bound by his oath.’ 17 You blind fools! For which is greater, the gold
or the temple that has made the gold sacred? 18 And you say, ‘If
anyone swears by the altar, it is nothing, but if anyone swears by the
gift that is on the altar, he is bound by his oath.’ 19 You blind men!
For which is greater, the gift or the altar that makes the gift
sacred? 20 So whoever swears by the altar swears by it and by
everything on it. 21 And whoever swears by the temple swears by it and
by him who dwells in it. 22 And whoever swears by heaven swears by the
throne of God and by him who sits upon it.
It seems that the issue here is that everything we say should be true. We should not need to claim a distinct truthfulness for a particular statement that contrasts to falsehoods in our normal dialogue. All of our statements should carry the same level of honesty and truthfulness.
In English, I don't think we really use the word "swear" in this way. Oftentimes it's just an expression.
The Pharisees were distinguishing between the object on which someone was swearing, when it shouldn't have mattered if there were any swearing at all.
So, if the person is sensitive to this particular phrase, you could simply explain that you are using the word as an idiom and not swearing by the temple or the gold of the temple or anything like that. Hopefully, the person can understand that.
It is actually quite dangerous to just accept church teaching without question. Knowing what without knowing why can lead to significant problems. This is where the question of Sola Scriptura and Church Authority collide. Acts 17:11 commends the Bereans for being more noble than the Thessalonians. The commendation was based on the fact that they didn't just accept what Paul taught, but "examined the Scriptures every day to see if what Paul said was true." If the Bereans were commended for this, we would do well to follow that pattern.