It's not so much that sandals or slippers are considered to be unholy, it's that they're dirty, and removing them is a sign of respect similar to removing one's hat when entering a building, or perhaps removing a nose ring when entering a strict parent's house.
From the United church of God's article on the subject:
Taking off your sandals was like the old custom of a man taking off
his hat when entering a building or greeting a lady—it was a token of
respect.
The ground was holy because of God's presence. People were to approach
God with solemnity and humility. Taking off their sandals expressed an
inward reverence through an outward behavior in their worship. Showing
such respect avoids anything casual, sloppy or rude.
Some Eastern religions today still require bare feet when entering
their temples. Anciently the Greeks, in the worship of Diana and
Jupiter, required worshippers to take off their shoes (Adam Clarke's
Commentary, Exodus 3:5 ). A common custom in the Orient and for many
in North America is to take off your shoes when entering a person's
home. God Has a High Standard for Approaching Him
As to why...
From Jamieson-Fausset-Brown Bible Commentary:
put off thy shoes—The direction was in conformity with a usage which
was well known to Moses, for the Egyptian priests observed it in their
temples, and it is observed in all Eastern countries where the people
take off their shoes or sandals, as we do our hats. But the Eastern
idea is not precisely the same as the Western. With us, the removal of
the hat is an expression of reverence for the place we enter, or
rather of Him who is worshipped there. With them the removal of the
shoes is a confession of personal defilement and conscious
unworthiness to stand in the presence of unspotted holiness.
Also, from the Pulpit Commentary (available at the link in the preceding section)
Rather, "thy sandals." Shoes were not worn commonly, even by the
Egyptians, until a late period, and would certainly not be known in
the land of Midian at this time. The practice of putting them off
before entering a temple, a palace, or even the private apartments of
a house, was, and is, universal in the East - the rationale of it
being that the shoes or sandals have dust or dirt attaching to them.