Of course the cross was of pagan origin and you are probably correct that there are "references that agree that prior to 3rd or 4th century only non Christian groups ever used the symbol of the cross in their worship."
First of all let us remember that it was only in 313 with the Edict of Milan that peace was officially given to the Church and crucifixion was put away with as a form of capital punishment.
In antiquity crucifixion was considered one of the most brutal and shameful modes of death. Probably originating with the Assyrians and Babylonians, it was used systematically by the Persians in the 6th century BC. Alexander the Great brought it from there to the eastern Mediterranean countries in the 4th century BC, and the Phoenicians introduced it to Rome in the 3rd century BC. It was virtually never used in pre-Hellenic Greece. The Romans perfected crucifion for 500 years until it was abolished by Constantine I in the 4th century AD. Crucifixion in Roman times was applied mostly to slaves, disgraced soldiers, Christians and foreigners--only very rarely to Roman citizens. Death, usually after 6 hours--4 days, was due to multifactorial pathology: after-effects of compulsory scourging and maiming, haemorrhage and dehydration causing hypovolaemic shock and pain, but the most important factor was progressive asphyxia caused by impairment of respiratory movement. Resultant anoxaemia exaggerated hypovolaemic shock. Death was probably commonly precipitated by cardiac arrest, caused by vasovagal reflexes, initiated inter alia by severe anoxaemia, severe pain, body blows and breaking of the large bones. The attending Roman guards could only leave the site after the victim had died, and were known to precipitate death by means of deliberate fracturing of the tibia and/or fibula, spear stab wounds into the heart, sharp blows to the front of the chest, or a smoking fire built at the foot of the cross to asphyxiate the victim. - The history and pathology of crucifixion.
Thus it would be normal that the faithful in the Early Church would not openly keep the sign of the cross as a public expression of faith, since it was still an instrument of torture.
It is from this original Christian worship of the cross that arose the custom of making on one's forehead the sign of the cross. Tertullian says: "Frontem crucis signaculo terimus" (De Cor. mil. iii), i.e. "We Christians wear out our foreheads with the sign of the cross." The practice was so general about the year 200, according to the same writer, that the Christians of his time were wont to sign themselves with the cross before undertaking any action. He says that it is not commanded in Holy Scripture, but is a matter of Christian tradition, like certain other practices that are confirmed by long usage and the spirit of faith in which they are kept.
The punishment of the cross remained in force throughout the Roman Empire until the first half of the fourth century. In the early part of his reign Constantine continued to inflict the penalty of the cross (affigere patibulo) on slaves guilty of delatio domini, i.e. of denouncing their masters (Cod. Th. ad leg. Jul. magist.). Later on he abolished this infamous punishment, in memory and in honour of the Passion of Jesus Christ (Eusebius, Church History I.8).
The so-called Constantinian monogram prevailed during the whole of the fourth century, assuming various forms, and combining with the apocalyptic letters Alpha and Omega, but ever approaching more and more closely to the form of the cross pure and simple. In the latter part of that century what is known as the "monogrammatic cross" makes its appearance; it closely resembles the plain cross, and foreshadows its complete triumph in Christian art. The early years of the fifth century are of the highest importance in this development, because it was then that the undisguised cross first appears. As we have seen, such was the diffidence induced, and the habit of caution enforced, by three centuries of persecution, that the faithful had hesitated all that time to display the sign of Redemption openly and publicly. Constantine by the Edict of Milan had given definitive peace to the Church; yet, for another century the faithful did not judge it opportune to abandon the use of the Constantinian monogram in one or other of its many forms But the fifth century marks the period when Christian art broke away from old fears, and, secure in its triumph, displayed before the world, now become Christian also, the sign of its redemption.
Seeing that the cross was the symbol of an ignominious death, the repugnance of the early Christians to any representation of Christ's torments and ignominy is easily understood.
Since by His holy sacrificial death upon the Cross Christ sanctified this former instrument of shame and ignominy, it must have very soon become in the eyes of the faithful a sacred symbol of the Passion, consequently a sign of protection and defence (St. Paulinus of Nola, "Carm. in Natal. S. Felicis", XI, 612; Prudent., "Adv. Symm.", I, 486). It is not, therefore, altogether strange or inconceivable that, from the beginning of the new religion, the cross should have appeared in Christian homes as an object of religious veneration, although no such monument of the earliest Christian art has been preserved. Catholic Encyclopedia
Let us not forget that Our Lord shed His sacred blood upon the Cross for our salvation!