The yoke that Christ is talking about is in direct contradiction to the yoke and burden of the Pharisees.
In Matthew 23:4, Jesus speaks of the Pharisees like this:
They tie up heavy burdens, hard to bear, and lay them on people's
shoulders, but they themselves are not willing to move them with their
That is to say, they ask so many things of the people, and they demand so much of the people who follow them, and yet they're not themselves willing to help those who try to follow their demands.
Attempting to follow the demands is exactly what wears people out. Any time we try to follow all the rules that are thrown at us, we begin to be weary of the rules that we're carrying. We realize that the burden is more than we can bear, and we begin to resent those who give us the rules and the burdens. Those who are weary in verse 28 are those who are drowning in the rules of the Pharisees. They're those who are tired of trying so hard.
In contrast, Jesus yoke is easy because He Himself carried the yoke. The demands that he places on us, he himself fulfills, and then goes on to pay the penalty for those of us who cannot fulfill the demands. It's as if he's saying, "I tie up heavy burdens, hard to bear, and I lay them on my own shoulder." Thus, for us his yoke is easy and his burden is light, because he himself carries it, and in this we can find rest for our souls.