Why is the square a symbol of the Melchizedek Priesthood?
Short answer is because it's on the altar cloth with Melchizedek on the mosaic. I've been unable to find any ancient reference to the eight pointed star referred to or necessarily associated with Melchizedek.
Henry Pelham Holmes Bromwell, a mason, in his book Restorations of Masonic Geometry and Symbolry Being a Dissertation on the Lost Knowlege of the Lodges apparently identifies the eight pointed star the "Signet of Melchizedek" in 1905. 1
The LDS also commonly refer to it as the seal of Melchizedek because of Michael Lyon's caption, The white altar cloth is decorated with ... the so-called ‘seal of Melchizedek,’ two interlocked squares, to a picture in Hugh Nibley's book Temple and Cosmos (the picture and caption are not included in the online version, but is similar to the first in the OP question) in 1992.
What ancient documents detail this?
None that I can find, though there are several architectural/artworks that have this symbol as part of it, though not necessarily connected to Melchizedek.
In Islam, the symbol is called a Rub el Hizb, or quarter group. 2 The usage of this symbol by Muslims, however, is a moot point since their faith sprung up in the early 7th century AD, and they appear to have adapted this symbol from Byzantine art. CR Clifford mentions:
The combination of two squares gives us the Mohammedan eight-pointed star, and the five-pointed star is supposed to be a Christian symbol. Yet if we go back to the period of Christian enthusiasm in the Byzantine Empire we will find the Brahman and Mohammedan eight-pointed star in universal use. 3
In Hinduism, the eight-pointed star is called Star of Lakshmi.
If no ancient documents, then are there Jewish or early Christian graves that somehow convey that the square represents the Melchizedek Priesthood?
I have been unable to find graves that convey the squares or any connection the squares have to Melchizedek.
A few other articles on the seal of Melchizedek (most are from the LDS perspective):
2 http://symboldictionary.net/?p=3106 and http://ldsmag.com/the-seal-of-melchizedek-history-meanings-and-mysteries
3 http://lds-studies.blogspot.com/2011/05/seal-of-melchizedek-eight-pointed-star.html and https://books.google.com/books?id=4BroAAAAMAAJ&pg=PA12&lpg=PA12&dq=Mohammedan+eight-pointed+star+-lds&source=bl&ots=-iQByaLw0z&sig=ogg2EZjLnTOxg2UzWhdbqcF9-5I&hl=en&sa=X&ved=0ahUKEwi4rdq0p8fSAhWISCYKHcmNBcsQ6AEIGjAA#v=onepage&q=Mohammedan%20eight-pointed%20star%20-lds&f=false