The Hebrew letters roughly corresponding to YHWH are the name of the Jewish god. Literally, this is their god's name, just like my name is Kyralessa and your name is Rachel.
However, in order to avoid using God's name in vain, the Jews did not pronounce this name. Instead, they substituted the Hebrew word adonai, which means "lord".
In the actual Hebrew text of the Old Testament, the consonants YHWH for the name of God are pointed with the vowels for the word adonai. (You can see what Hebrew vowels look like here: Hebrew Nikkudot) This means that whenever you're reading aloud, and you come to the name of God, you instead read the word adonai.
This isn't the only case of this kind of substitution, though it's by far the most common. You can find more qere/ketiv cases here: Qere and Ketiv
Reading the vowels of adonai with the consonants YHWH gives one "Yehovah", or "Jehovah". This isn't the name of God, however. Despite its historical popularity, it's simply a nonsensical mixture of two different words. We don't know what vowels the word YHWH originally had.
Writing the word LORD in the Old Testament is a way of indicating that in that place, the name of God (YHWH) is present in the original Hebrew. adonai translates as "Lord", so this is a way in English of simulating the Hebrew practice of saying "Lord" in place of God's name.
There are a few cases in Hebrew where adonai YHWH occurs. To avoid the awkwardness of "Lord LORD" in the text, Bibles generally render this as "Lord GOD" instead.
Incidentally, the capital letters in English don't have any connection with capital letters in Hebrew, because Hebrew doesn't have capital and lower-case letters.
Now, what about the New Testament? The New Testament doesn't contain the name of God anywhere. The word "Lord" in the New Testament is κύριος kyrios, and the word "God" is θεός theos. It's quite possible that in some cases of the New Testament, kyrios is meant to stand in for the name of God (with the adonai substition). But since it's technically not the name of God, the "Lord"-in-small-caps convention isn't used in the New Testament.