I think translation is partly the issue. A 1902 biography by Father Bacci, The Life of Saint Philip Neri: Apostle of Rome, and Founder of the Congregation of the Oratory Vol. 1 Chapter XVII (Of Philip's Humility) ends with this quote containing a Latin version:
... Lastly, he used to say that to arrive at the perfection of humility, four things are necessary, -- to despise the world, to despise no one, to despise self, to despise being despised by others -- spernere mundum, spernere nullum, spernere se ipsum, spernere se sperni.
But more importantly, we can understand the meaning from the life context of St. Philip Neri himself, whose life is characterized by reaching out to forgotten poor people of Rome ignored by indifferent clergy (thus "despised"), but who were religiously hungry. He famously refused the offer of being made cardinal, so in other words, he "despised" the worldly ambition of corrupt church officials. In contrast, his newly founded Congregation of the Oratory has the charism of being bound by no formal vows, but only with the bond of charity in reaching out those "despised" people. He even got in trouble for hearing confessions in the streets (outside church buildings) in order to bring the church to them! I highly recommend watching the 2010 movie Saint Philip Neri: I Prefer Heaven which is available through FORMED subscription, which many Catholic parishes offer for free to their parish members.
For interpretation of this maxim, I quote from the blog of Fr. Joseph Illo who interprets "despise being despised" as "to despise sadness" in the sense of insisting on being joyful despite being despised by the world.
Quote from his St. Philip Neri Homily 2015 blog entry celebrating the St. Philip's Feast Day:
To despise the world
St. Philip “despised” the world. He refused to let it dazzle him, holding always man’s final end before his mind. “What will you do in life?” he asked a young nobleman. “I plan to study law and become an attorney.” St. Philip pressed him: “And then?” he asked. “And then,” the young man replied, “I will be a successful lawyer.” “And then?” Philip again asked his young friend. “And then people will speak well of me and I’ll have a good reputation. “And then?” came the question once again. “And then I shall lead an easy life and be happy.” St. Philip asked a final time: “And then?” Across the centuries Good St. Philip asks us: where do you want to be at the end of it all? For what are you making such a tremendous exercise of your beautiful human spirit if not for God and heaven?
To despise no man
St. Philip despised no person. Like Mother Teresa of our own time, he saw Christ before him in every man, each made in the image of God. Those who take faith seriously in a secular society are tempted to hold aloof from irreligious people; what have we to do with them, and how can we even hold a meaningful conversation? St. Philip took a keen interest in every man and woman, holding no one in contempt. This was a grace, but a grace to which he responded, sometimes at great personal cost.
To despise myself
St. Philip despised himself. He did not despise the image of God in himself, nor discount the great good that he was to all of humanity. But he knew every man needs to offset the egocentrism inherent in all who inhabit their own skins. So he did his best not to take himself too seriously, often using an odd humor to deprecate his own and others’ egos. People shook their heads, some considering him completely loony, at the crazy stunts he pulled off among Rome’s gentile classes.
To despise sadness
And finally, St. Philip despised being despised. I’m not exactly sure what that means, but I think he is taking a stick to overly pious and fraudulent asceticism. Plenty of us think that our sufferings, of themselves, our being despised by the world in itself, is a sure sign of our true holiness. Not so, says St. Philip. In the end, it is joy, not moroseness, which manifests true holiness. This last “requirement for true humility,” to despise being despised, puts humility back into God’s hands. We cannot manufacture our own humility, or our one sanctity. We can only commend ourselves, with loving trust and true joy, into God’s hands. It is this carefree joy that marks St. Philip’s life, and, let us hope, his sons in the Oratory he founded.