The “holy place” is a thoroughly biblical notion. Jerusalem was and is a holy city:

Matthew 4:5 Then the devil took him to the holy city, and set him on the pinnacle of the temple, (cf. Neh 11:1, 18; Is 48:2)

Matthew 27:53 and coming out of the tombs after his resurrection they went into the holy city and appeared to many.

The temple in Jerusalem (destroyed by the Romans in A.D. 70) was a particularly holy place:

1 Chronicles 29:3 Moreover, in addition to all that I have provided for the holy house, I have a treasure of my own of gold and silver, and because of my devotion to the house of my God I give it to the house of my God:

Psalm 5:7 But I through the abundance of thy steadfast love will enter thy house, I will worship toward thy holy temple in the fear of thee. (cf. 138:2)

Isaiah 56:7 these I will bring to my holy mountain, and make them joyful in my house of prayer; their burnt offerings and their sacrifices will be accepted on my altar; for my house shall be called a house of prayer for all peoples.  (cf. Mt 21:13; Mk 11:17; Lk 19:46)

The inner sanctum of the temple, called the Holy of Holies (Lev 16), was the ultimate holy place in the world, and in Judaism. Originally the wooden, treasure chest-like ark of the covenant was in the Holy of Holies, and God was thought to be present there in a very special way, on what was called the mercy seat, in between two carved cherubim:

Exodus 25:21-22 And you shall put the mercy seat on the top of the ark; and in the ark you shall put the testimony that I shall give you. [22] There I will meet with you, and from above the mercy seat, from between the two cherubim that are upon the ark of the testimony, I will speak with you of all that I will give you in commandment for the people of Israel. 

It was so holy it could not be touched, and hence it was transported with poles that ran through rings on its side (Ex 25:13-15). In fact, on one occasion, when it was about to fall over while being moved, after the oxen stumbled, one Uzziah merely reached out to steady it and was immediately struck dead:

2 Samuel 6:7 And the anger of the LORD was kindled against Uzzah; and God smote him there because he put forth his hand to the ark; and he died there beside the ark of God.

The men of Beth-shemesh also died when they simply looked inside the ark:

1 Samuel 6:19 And he slew some of the men of Beth-she'mesh, because they looked into the ark of the LORD; he slew seventy men of them, and the people mourned because the LORD had made a great slaughter among the people. (cf. Ex 33:20).

It was not that God was “cruel.” He had to get through to stubborn, sinful men, the notion that some things are sanctified or holy: so much so that they have to be approached with the utmost reverence and solemnity, and are not even to be touched.

It's a bit like the child touching the hot oven. They learn not to, when they get hurt by it. Once a few Israelites were killed by touching holy things, the people learned very quickly. God is a transcendent “Other” and completely holy. To get too close to Him is as frightening as it is exhilarating, as the Bible illustrates in many places.

Mt. Sinai was holy due to God's tangible presence there, in the burning bush (Ex 3:5). Just before the Hebrews were to receive the Ten Commandments, God charged the people to not even touch the mountain, or its “border,” on pain of death (19:12-13). Even animals were included in the restriction!

God's special presence – considered apart from the fact that He is also omnipresent – imparts holiness:

Deuteronomy 7:6 For you are a people holy to the LORD your God; the LORD your God has chosen you to be a people for his own possession, out of all the peoples that are on the face of the earth. (cf. 26:19; Jer 2:3)

Angels are called “holy ones” in Scripture precisely because of their proximity to God:

Psalm 89:6-7 For who in the skies can be compared to the LORD? Who among the heavenly beings is like the LORD, [7] a God feared in the council of the holy ones, great and terrible above all that are round about him?

This is a bare summary of what the Bible teaches about holy places. The New Testament continues to refer to Jerusalem as the “holy city” (Mt 4:5; 27:53 above), and Jesus spoke of the Holy of Holies as “the holy place” (Mt 24:15; cf. Heb 9:2, 12, 25).

St. Peter calls the Mount of Transfiguration “the holy mountain” (2 Pet 1:17-18; cf. Mt 17:1-6). Protestants widely use the terms “Holy Bible” and “Holy Land”. In so doing, they tacitly acknowledge the notion of “holy things”: even though, if pressed, they may argue against it, as I once did myself.

My standard argument to shock Protestants of this sort into the realization that they themselves – down deep – accept the existence of holy places and things, is to ask them whether they would bulldoze the spot they believe to be the location of Jesus' crucifixion, and put up a fast food restaurant. Most would be outraged by the very notion, and would consider it an insulting or ridiculous question. Why? Well, because they do believe in holy places, after all.

The difference for the Catholic or Orthodox Christian, however, is that the holy thing literally conveys grace, as the sacraments do (though in a somewhat different sense, similar to what is called a sacramental: holy water, crucifixes, scapulars, etc.).