Why is this? It seems a bit arrogant, especially as Jesus loved the other disciples as well.
There are two ways to read that. Reading it as an arrogant statement is certainly one, but I think it can also be read exactly the opposite way -- as a sign of humility. Not wanting to name himself in a "me, John, I was there see" way, he simply refers to himself based on his identity in relation to Christ.
As a Christian I think this is great way to think of ourselves. Rather than thinking about myself and my accomplishments, if I think about my identity as the way Christ sees me -- that I am simply a recipient of undeserved love -- my life is more likely to reflect him.
It doesn't have to be a boastful statement.
Textually, to say that the Evangelist is one and the same with the disciple whom Jesus loved is an assumption. The Evangelist doesn't make that claim explicitly. That doesn't mean it's wrong. It's possible that refraining from that claim is a sign of humility.
This Gospel made the transition from (presumably) oral history to the written word well over a generation after the Resurrection. The original hearers of this Gospel were a community in exile, living on the margin of the Judean community. From their point of view in exile it was likely a great comfort to them to know that their guy (John) enjoyed a special place in Jesus's favor.
One of the most significant words of Jesus to this disciple is when he said "Mother, behold your son. Son, behold your mother," while he was dying at Golgotha. For each of us, to have a special place in Jesus's heart is to take PERSONALLY his commandment to "love one another as I have loved you."
Your place, and my place, in Jesus's heart are his gift to us. To hold one another in our hearts are our response to that gift.
The answer lies in the history of the gospels. All the New Testament gospels were originally anonymous until attributed to Matthew, Mark, Luke and John later in the second century. The Church Fathers looked for clues that might help them decide who probably wrote each of the gospels.
In the case of the fourth gospel, the Church Fathers noticed that it never mentioned the apostle John, but that it did mention a 'disciple whom Jesus loved'. They decided that the beloved disciple must be John. They then decided that John must have been the author but that his modesty prevented him from using his own name in this account.
The New Testament scholar, Elaine Pagels, was the first to realise why this disciple was described in this way. She noticed that the beloved disciple, in all but one case, was contrasted with the apostle Peter and always came out the more worthy apostle. Even the name 'disciple whom Jesus loved' suggests to us that Jesus saw him as more worthy than Peter (or any other disciple, but it is only Peter he is compared with). It seems that the author of John sought to tone down what he saw as excessive veneration of St Peter early in the second century.
The disciple whom Jesus loved, was loved more than the others because Jesus trusted Him with his greatest secrets or teachings.
Jesus put His Mother into the care of this disciple for her lifetime. Now consider the text.
The author of The Gospel according to John, was not at the last supper. He knew the story well, through oral tradition from the disciples, perhaps from John of Zebedee himself. The Gospel itself was pen'd at the end of the century, and by a Greek theologian, perhaps the greatest in that century. The Hebrew who was lying on Jesus's breast was one of the twelve, and in comparison with the author, theologically rather naive. If the disciple whom Jesus loved was this John, He called him the son of thunder. “And James the son of Zebedee, and John the brother of James; and he surnamed them Boanerges, which is, The sons of thunder:” Mark 3:17, KJV. Here was the disciple who took care of Mary. What great love must Jesus have had for John.
“When Jesus therefore saw his mother, and the disciple standing by, whom he loved, he saith unto his mother, Woman, behold thy son!” John 19:26, KJV. And this disciple lived with Mary until her death (or ascension.)
Some of the Fathers think it was out of modesty (Augustine, and Chrysostom) and that the author and the disciple were the same.
There is another consideration: Peter beckoned to the disciple, and he moved from the Master's bosom (or perhaps leaned away,) to hear the question Peter was to put to him. Then the disciple whom Jesus loved returned to lie on His breast to ask "Lord who is it?"
Here Peter, the Chief Apostle, beckons to that disciple because of the special place he had in the Master's heart. Peter could expect an answer, but he did not hear what the disciple heard, nor did the disciple speak.
Peter saw the Master dip the sop, and only heard "what thou doest, do quickly."
The Master had taken the disciple whom he loved into His confidence, and the disciple did not betray the confidence by rising up and calling a warning to the others. This was out of the love between the disciple and the Master, which was stronger than with any other disciple. The others, even Peter, could not be trusted to keep quiet about Judas. Any one of them would have shouted out a warning to the others, had he known. Wouldn't we do the same? But the disciple whom the Master loved, did not betray His Master.
He also did not understand what was going to happen to the Master - and of the resurrection to come. “Then went in also that other disciple, which came first to the sepulchre, and he saw, and believed. For as yet they knew not the scripture, that he must rise again from the dead.” John 20:8, 9, KJV.
Though Judas must have known, for he had been chosen by God to do what was necessary to fulfill the scripture:“While I was with them in the world, I kept them in thy name: those that thou gavest me I have kept, and none of them is lost, but the son of perdition; that the scripture might be fulfilled.” John 17:12, KJV.
And the Master allowed Judas in the garden to kiss him, for love. The Master does not mention anyone kissing Him but Mary Magdalene. “... he turned to the woman, and said unto Simon, Seest thou this woman? I entered into thine house, thou gavest me no water for my feet: but she hath washed my feet with tears, and wiped them with the hairs of her head. Thou gavest me no kiss: but this woman since the time I came in hath not ceased to kiss my feet.” Luke 7:43-45, KJV.
“But Jesus said unto him, Judas, betrayest thou the Son of man with a kiss?” Luke 22:48, KJV.
How great is the love of the Master for those who trust Him and are obedient to Him?
If you read it in a way where "The Disciple whom Jesus Loved" = "All Disciples whom Jesus Loves" (meaning you and me) then a few of the last words of Jesus take on a much deeper meaning, especially for Catholics who strongly love the Blessed Virgin Mary.
Take for example the footnotes from the John 19:26-27 (NABRE)
This scene has been interpreted literally, of Jesus’ concern for his mother; and symbolically, e.g., in the light of the Cana story in Jn 2 (the presence of the mother of Jesus, the address woman, and the mention of the hour) and of the upper room in Jn 13 (the presence of the beloved disciple; the hour). Now that the hour has come (Jn 19:28), Mary (a symbol of the church?) is given a role as the mother of Christians (personified by the beloved disciple); or, as a representative of those seeking salvation, she is supported by the disciple who interprets Jesus’ revelation; or Jewish and Gentile Christianity (or Israel and the Christian community) are reconciled.
Add on the idea that the woman in John's Revelation is Mary and the woman is also the Church (not the manmade stuff, just the mystical stuff) the disciple is us. So we take Mary into our home, and we are loved by Jesus as His brother.