God, nature, reason, all scripture, all law, all authors, all doctors, yea all councils are against usury. Philosophers, Greeks, Latins, Divines, Catholics, Heretics, all tongues, all nations, have thought usury as bad as a thief.
-Roger Fenton, 1612
There are a few different reasons why usury has historically been considered a bad practice.
First, it is unvirtuous. It is a lazy, unproductive, unskillful way to make a living. That is, the usurer is not contributing to the society with a particular skill set or mode of labor. (This argument commonly comes from religions, but is also present in ethics books, such as Aristotle's Nicomachean Ethics)
Second it is contrary to the nature of money, the purpose of which is exchange. Money is meant to be a stand-in for goods or services and is meant to provide a way to exchange one good or service for another. Usury contradicts the very purpose of money and is therefore wrong. (This is a largely non-religious argument that can be found in places like Aristotle's Politics)
Third, usury is unjust because it is a manipulation whereby one sells that which does not exist. That is, the usurer is selling the use of money, but there is no true difference between money and the use of money since money has no purpose apart from use. By definition, for money to be used is for money to be consumed. I could loan someone a car and charge them for using it, and then receive the car back when they are done with it, but money does not work this way. If someone uses the money then it is gone/consumed. No one would ever take out a loan without using the money. This argument was more plausible before complex financial systems were erected and ROI and opportunity cost were more or less guaranteed. (This is a Medieval argument that can be found in places like Thomas Aquinas' Summa Theologica)
A more modern argument against usury is that it creates "fake wealth" and results in problems like the "housing bubble." For example, suppose Joe lends Fred $100 at 10% interest. Fred does the same to Bob and Bob does the same to Joe. After the three-part transaction all three parties are entitled to an extra $10. But this $30 is not connected to any real thing in the world. Fake wealth has been created.