The Nation of Israel at the time Jesus was preaching, was well versed in the Pentateuch, the Prophets and the Psalms, and from those books we find the following Scriptures:
Deuteronomy 26:15 KJV Look down from thy holy habitation, from heaven, and bless thy people Israel, and the land which thou hast given us, as thou swarest unto our fathers, a land that floweth with milk and honey.
Psalms 11:4 KJV The Lord [is] in his holy temple, the Lord's throne [is] in heaven: his eyes behold, his eyelids try, the children of men.
Isaiah 66:1 KJV Thus saith the Lord, The heaven [is] my throne, and the earth [is] my footstool: where [is] the house that ye build unto me? and where [is] the place of my rest?
From these Scriptures it would cause one to believe that Heaven was up, as seemingly most people believe it to be today. The Israelites on the other hand might have believed that God's repose was in the Temple based on:
1st. Kings 8:10 and 11 KJV And it came to pass, when the priests were come out of the holy place, that the cloud filled the house of the LORD, 11 So that the priests could not stand to minister because of the cloud: for the glory of the LORD had filled the house of the LORD.
And even though the Temple had been destroyed it had been rebuilt by Ezra and Nehemiah. However the presence of the LORD is more connected to the Mercy seat which is the lid to the ark of the covenant, and that disappeared at about the time of the Captivity of Israel by Nebuchadnezzar according to this article extracted from http://www.allaboutarchaeology.org/when-did-the-ark-of-the-covenant-disappear-faq.htm:
When did the Ark of the Covenant disappear? The last historical mention of the Ark in Scripture is in Second Chronicles chapter 35, where King Josiah (who reigned in Judah c. 640-609 BC) asked the Levites to return the Ark to the Temple where Solomon had originally housed it after completing and dedicating the Temple sometime in the 10th century BC (Second Chronicles, chapter 5). There is no mention as to why the Levites had removed the Ark in the first place, nor is there any indication as to whether or not the Levites acquiesced to King Josiah's request.
In the apocryphal book (that is, noncanonical book) of Second Maccabees, chapter 2 (verses 1-8), we read that the prophet Jeremiah "following a divine revelation, ordered that the tent [i.e. the tabernacle of meeting] and the ark [i.e. the ark of the covenant] should accompany him and how he went off to the mountain which Moses climbed to see God's inheritance [i.e. Mt. Nebo; see Deuteronomy 31:1-4]. When Jeremiah arrived there, he found a room in a cave in which he put the tent, the ark, and the altar of incense; then he blocked up the entrance." (vv. 4-5) There is some debate as to whether or not this second-hand account (see v. 1) is trustworthy.
- See more at: http://www.allaboutarchaeology.org/when-did-the-ark-of-the-covenant-disappear-faq.htm#sthash.MIz6AGn2.dpuf
Since it appears to be generally accepted that Heaven is the abode of God, Heaven would not se a specific place, but would be wherever God is, and not where he goes to.
And since we have no direct information, as to what was the general attitude of the Israelites at that time, any speculation as to their concept of where Heaven actually exists would be just that speculation, unless there is some artifact which can further enlighten us.
Hope this helps.
