If you've ever witnessed a birth, it's not exactly a silent affair. Add to this that it was a birth in a stable where there were animals presumably making all kinds of noises, you have quite the noisy affair.
So no, it's not a reference to anything beyond the idea that the carol "Silent Night" presents a rather idealized version of the nativity that just isn't particularly realistic.
This sermon covers this idea pretty well.
And yet, it’s absolutely appropriate that our Christmas Eve, that our Silent Night should be filled with sound and turmoil. It’s appropriate because that was probably the case on that first Christmas nightt. Think about it. Bethlehem was a little town with a huge heritage. All of the descendants of Jesse, the father of King David, had to return to their ancestral home for the census to be taken, and Jesse had one big family. David was one of eight sons and that was just part of one generation. Now hundreds of years later, all those descendants were coming to this little farming village of Bethlehem. It was anything but a silent night. The village overflowed with business travelers staying in strange environments. There was no more room in the inns or motels. Restaurants are booked to capacity. Energetic entrepreneurs quickly set up activities for entertainment and profit. There was a little gambling here and there. A gala party event there. Everyone looking for something to do like a family at the beach on a rainy day. Children whining. Babies crying. The teens and adults getting out of hand every now and then. We don't know everything that was going on, but I think that we can assume that it was not a silent night.
It goes on to talk about the things I mentioned in the intro, animal sounds, birthing sounds, baby jesus' cries etc.
However, this doesn't mean the song is wrong. From the same sermon:
Yet, at the same time, our favorite Christmas hymn is also right – right in every respect. On that night, Jesus did, indeed, sleep in heavenly peace. The glories did stream from heaven afar, and the heavenly hosts did sing, “Alleluia! Christ the Savior is born!” The Christ child who laid in that manger and who cuddled in His mother’s arms was – and still is – Lord at His birth.