Christ gave the power of "binding and loosing" to Peter alone with this saying; however, a little while later He also told all the apostles:
Amen, I say to you, whatever you bind on earth shall be bound in heaven, and whatever you loose on earth shall be loosed in heaven.
(Matthew 18:18)
Similarly, Paul states that
All this is from God, who has reconciled us to himself through Christ and given us the ministry of reconciliation.
(2 Corinthians 5:18; emphasis added)
For these reasons, the Church believes,
our Lord placed Simon alone as the rock and the bearer of the keys of the Church, and made him shepherd of the whole flock; it is evident, however, that the power of binding and loosing, which was given to Peter, was granted also to the college of apostles, joined with their head.
(Lumen Gentium, Section 22)
The Catechism adds,
Christ has willed that in her prayer and life and action his whole Church should be the sign and instrument of the forgiveness and reconciliation that he acquired for us at the price of his blood. But he entrusted the exercise of the power of absolution to the apostolic ministry which he charged with the "ministry of reconciliation." The apostle is sent out "on behalf of Christ" with "God making his appeal" through him and pleading: "Be reconciled to God."
So what allows us to believe that this power was passed on from the apostles to their successors? The same thing that allows us to believe that their successors were intended to take their place and exercise exactly their powers with those whose spiritual welfare they were intended to oversee:
In order that the mission entrusted to them might be continued after their death, [the apostles] consigned, by will and testament, as it were, to their immediate collaborators the duty of completing and consolidating the work they had begun, urging them to tend to the whole flock, in which the Holy Spirit had appointed them to shepherd the Church of God. They accordingly designated such men and then made the ruling that likewise on their death other proven men should take over their ministry. ... Just as the office which the Lord confided to Peter alone, as first of the apostles, destined to be transmitted to his successors, is a permanent one, so also endures the office, which the apostles received, of shepherding the Church, a charge destined to be exercised without interruption by the sacred order of bishops.
(Lumen Gentium section 20, quoted in Catechism of the Catholic Church, paragraphs 861–62)
So where do the priests get that power? Priests are appointed to assist bishops, and share in their power as it is delegated to them:
The function of the bishops’ ministry was handed over in a subordinate degree to priests so that they might be appointed in the order of the priesthood and be co-workers of the episcopal order for the proper fulfillment of the apostolic mission that had been entrusted to it by Christ.
(Presbyterorum Ordinis, paragraph 2 section 2; quoted in Catechism of the Catholic Church paragraph 1562)
This then is the answer: Jesus Christ gave this power to all his apostles, and then in order to safeguard the Church, gave them (through the Holy Spirit) the ability to appoint successors who carried on all their powers and responsibilities. These men in turn appointed delegates (priests) who were able to exercise some of those powers where the bishop might not be available to.