The short answer: to our fellow Christians, our brothers and sisters in Christ.
Allow me to develop my answer by first contextualizing the verse. Notice in each of the preceding parables in Matthew 24 and 25, there is a pronouncement and/or an act by someone in authority:
a master (24:45-51) who in light of his slave's faithfulness while his master was away on a trip put him in charge of all his possessions
a lord (25:1-13, especially v.12), probably the father of the bridegroom, who says, "Truly I say to you, I do not know you."
another master (25:14-30) who said to two of his slaves, "Well done, good and faithful slave. You were faithful with a few things, I will put you in charge of many things; enter into the joy of your master," but to the third slave who was unfaithful as a steward of the master's money, the master said, "'You wicked, lazy slave, you knew that I reap where I did not sow and gather where I scattered no seed. Then you ought to have put my money in the bank, and on my arrival I would have received my money back with interest. Therefore take away the talent from him, and give it to the one who has the ten talents.'"
a king (25:31-46) who says to the sheep on his right, "'Come, you who are blessed of My Father, inherit the kingdom prepared for you from the foundation of the world.'" To the goats on his left, however, he says, "'Depart from Me, accursed ones, into the eternal fire which has been prepared for the devil and his angels.'"
What these parables have in common is a judgment rendered by a person in authority. That judgment determines who is rewarded and who is punished; who is let in and who is excluded; who is praised and who is censured.
I suggest the two categories of people are the faithful, who are the true believers in Jesus who are gathered to his right, and unbelievers, who are gathered to his left. The common theme in all the parables concerns faithfulness in doing what the person in charge expects to be done.
The first "work" of the true believers in Jesus' day (and by extension to all true believers in every generation since then) was to believe in him. Jesus said to a crowd of people whom he had just fed miraculously, and whom he suspected of following him simply because he fed them,
"'Do not work for the food which perishes, but for the food which endures to eternal life, which the Son of Man will give to you, for on Him the Father, God, has set His seal.' Therefore they said to Him, 'What shall we do, so that we may work the works of God?' 29 Jesus answered and said to them, 'This is the work of God, that you believe in Him whom He has sent'" (John 6:27-29 NAS, my emphasis).
The second work for the true believers in Jesus' day (and for us today) was to do things consistent with repentance. In other words, Jesus expected them to demonstrate their faith by their works, a concept which Jesus' half-brother James developed quite thoroughly, concluding "faith without works is dead"! (see James 2:26). Here is how Matthew had Jesus addressing the religious leaders of his day:
"But when [Jesus] saw many of the Pharisees and Sadducees coming for baptism, he said to them, 'You brood of vipers, who warned you to flee from the wrath to come? Therefore bear fruit in keeping with repentance and do not suppose that you can say to yourselves, 'We have Abraham for our father'; for I say to you that from these stones God is able to raise up children to Abraham'" (Matthew 3:7-9; cf. Luke 3:8, my emphasis).
All this to say, the words of the king to which you refer in your question come from the mouth of the Son of Man (v.31) at a great judgment. It is a winnowing judgment at which "sheep" and "goats" are segregated. God then pronounces both a blessing and a curse.
His blessing (v.34) is directed to his true believers who demonstrated faithfully their love for "the least" of his (and their) brothers and sisters, all of whom are gathered together on the right of the Son of Man. They did so by
feeding the hungry among them
giving a drink to the thirsty among them
clothing the naked among them
showing hospitality to those strangers among them
visiting the sick and imprisoned among them
His curse (v.41) is directed to unbelievers who failed to demonstrate faithfully their love for "the least of these" (v.45). Notice that this phrase "the least of these" does not refer to "the least of these brothers of mine" (i.e., the king's) in verse 40, but simply "the least of these [goats]" (i.e., the unbelievers).
Could this winnowing, or separating, be part of the "Great White Throne Judgment" John wrote about in Revelation, chapter 20, verses 11-15? Quite possibly. Regardless of whether it is or isn't, there is a common thread in all four parables; namely, there will be a day of reckoning for all humankind, with there being only two classes of people (or "nations," as John puts it) gathered before God:
the blessed (Matthew 25:34), who are the elect of God who have been faithful in serving Christ by serving their brothers and sisters in Christ in need
the cursed (v.41) who have proved they are not among the elect by not faithfully serving the needy among them
Notice that the judge does not use the phrase "one of these brothers of mine, even to the least of them" when addressing the goats. He addresses them instead as follows:
"'Truly I say to you, to the extent that you did not do it to one of the least of these, you did not do it to me'" (v.45, my emphasis).
In other words, the people whom the judge refers to as "the least" are not fellow believers (i.e., the judge's "least" brothers) but fellow unbelievers among the goats on the left.
I could go on at length about how Jesus will judge unbelievers not only by whether their names are written in the Lamb's book of life (Revelation 20:12), which they are not, but also by their works. In my view, there will be degrees of punishment in the "eternal fire" of which the king in Matthew 25 speaks. The more heinous the deeds done, the more severe the punishment will be.
Suffice it to say, in conclusion, that while we believers are alive and in the world, "working out our own salvation" (Philippians 2:12-13), we also,
"while we have opportunity . . . [are to] do good to all people, and especially to those who are of the household of the faith" (Galatians 6:10 NAS, my emphasis).
By doing so we prove we are the elect, the sheep belonging to the "good shepherd" who gave his life for the sheep (John 10:11 and 14).