Jonah 1:2 says
"Go to the great city of Nineveh and preach against it, because its wickedness has come up before me." (NIV)
So, we can say from there that they had sinned greatly in the eyes of God, and had they not repented, they would be destroyed (Jonah 3:4 says "...Nineveh will be overthrown"). Of course, your question was "why Nineveh in particular?" I'll elaborate on that.
First, a correction: Nineveh wasn't a Babylonian city, it was one of the largest cities of Assyria, another one of Israel's enemies. I want to say it was also the capital during Jonah's time; at the very least there is a mention of a "king of Nineveh" (Jonah 3:6). Sennacherib lived in Nineveh (see 2 Kings 19:36), though he is a little bit after Jonah's time.1
Using your analogy then, I wouldn't equate Nineveh with New York City, I'd equate it with Washington, D.C., though the Bible does explicitly mention that the city is big
...Now Nineveh was an exceedingly great city, three days' journey in breadth (Jonah 3:3, ESV)
As far as strategy goes, there's also this to consider: Assyria was the one that conquered Israel (the ten tribes) not long after Jonah went to preach to Nineveh.2
In short, you could single out Nineveh since it was a very important city for one of Israel's enemies, and posed a very real threat to Israel and Judah.
1 My bible has a table of dates putting Jonah in 793-753 BC. Sennacherib attacked Judah during Hezekiah's reign (2 Kings 18:13-19:36), which according to the same table is 727-698 BC. This is also consistent with Wikipedia's pegging of Sennacherib's reign at 705-681 BC.
2 The same table lists the conquering of Israel by Assyria at 722 BC. See also 2 Kings 17:6.