Prior to modern scientific consensus regarding the age of mankind, were there any theologians who argued that the chronology in Genesis 1-11 could not be used to calculate the date of Adam's creation?
Between AD 389-417 Augustine wrote three commentaries on Genesis and discussed the early chapters of Genesis in The City of God. Augustine believed the genealogies given in Genesis to be literal chronologies and that the pre-Flood patriarchs lived to be around 900 years. He stated:
“Unbelievers are also deceived by false documents which ascribe to history many thousand years, although we can calculate from Sacred Scripture that not 6,000 years have passed since the creation of man.”
Augustine believed that the original creation happened in an instant of time and that Genesis 6–8 describes a literal, global Flood. Augustine clearly had old-earth views to contend with in his day—from the Greeks and from other pagans—but he did not accept them and did not try to fit those ideas into Genesis.
Although Augustine reckoned it was possible to calculate the time from the creation of Adam based on biblical chronology and genealogies, he was not an orthodox young-earth creationist, rejecting the seven-day creation beliefs of Ambrose, who was instrumental in Augustine’s conversion to Christianity.
On the patriarchs’ ages see Augustine, The City of God, Book 15, Chapters 11–12, pp. 436–440.
Augustine. The City of God, translated by G. G. Walsh and G. Monahan (1952), Book 12, Chapter 11, p. 263. Washington, D.C.: Catholic University of America Press.
Augustine. The City of God, Book 12, Chapter 10, p. 263; Chapter 11, pp. 264–265.
Primeval Chronology by Dr. William Henry Green (1825-1900) Professor of Old Testament at Princeton Theological Seminary Published in Bibliotheca Sacra April, 1890 (pp. 285-303):
“On these various grounds we conclude that the Scriptures furnish no data for a chronological computation prior to the life of Abraham; and that the Mosaic records do not fix and were not intended to fix the precise date either of the Flood or of the creation of the world.”
While doing research into this subject, I came across an article about the concept of a pre-Adamic race. The idea is that God created a race of humans who lived on the Earth before He created Adam, the first man. This hypothesis has been promoted by various scholars at various times throughout history. Roman Emperor Julian the Apostate (circa A.D. 331–363) and Calvinist theologian Isaac de La Peyrère (1596-1676) are two notable examples.
It is perhaps not directly related to your specific question, but allow me to leave you the link to the article: https://www.gotquestions.org/pre-Adamic-race.html
Although I am unable to do justice to your question, I have learned new things while looking into it so my time has not been wasted! I hope someone else will come forward with insights into the views of theologians prior to Ussher regarding the biblical genealogies being used to fix a date for the creation of Adam.