When God said he should go with the men this was the second embassy of men from Balak to request Balaam's help. On the first occasion we see Balaam thinking he was a 'god dealer' trying to manipulate God and Balak, hoping to make the God of Israel give up on the Israelites and entice Him into league with Balak and at the same time get money from Balak for this service. Balaam sees God's working with him as recognition of his 'magic powers' and he is therefore acting as a mediator between God and Balak with the end goal of removing God from His allegiance with Israel. We must remember that in paganism the gods are often motivated by the same kinds of sinful desires that we have and are not omnipotent but influenceable.
Now we see with the first embassy that God gave Balak a message but Balak only passed on parts of it, hoping to bring both parties together.
But God said to Balaam, “Do not go with them. You must not put a curse on those people, because they are blessed.” The next morning Balaam got up and said to Balak’s officials, “Go back to your own country, for the Lord has refused to let me go with you.” (Numbers 22:12-13, NIV)
We see Balak purposefully left out most of God's message and had no plans of 'saying what the Lord had told him'. One can't properly imagine the type of perverse mind Balaam had. Balaams version of God's message implied an ungrounded arbitrariness on the part of God and confirmed Balak to hope for better results under more favorable circumstances. Possibly when God was not so moody, God would reconsider?
It was under these circumstances that a second embassy from Balak came again to ask Balaam 'to curse this people'. Refusing in the first embassy to yield himself willingly to God, he would now be made the unwilling instrument of exalting God. And thus God gave him leave to do that on which he had set his heart. However because ofmhis blinded self-satisfaction , in which next morning he accompanied the ambassadors of Balak, 'God’s anger was kindled because he went'.
In other words the idea is God gave Balaam over to his lusts in such a way that God would manipulate him, while he thought he was manipulating God. Yet due to the insane contradiction of reality as he should be going in obedience but his actual going was in deluded belief that he was controlling the situation, God let's him know that he was 'dumber than an ass', before sending him fully on his way. Yes God for I send you, but realize you are a dumb ass who knows not what you are doing when you go.
Balaam's actions was as dumb as Simon Magus when he offered the apostles money to give him the power of imparting the Holy Ghost. Therefore Balaam goes down in history renown for his greed:
They have left the straight way and wandered off to follow the way of Balaam son of Bezer, who loved the wages of wickedness. (2 Peter 2:15, NIV)