The first castle of Visegrád was built on the Sibrik Hill under the reign of St. Stephen. This building was destroyed during the Mongol invasion of 1241-42. As part of the reconstruction of the country's infrastructure, from the dowry of his wife, Maria Laskarina, King Béla IV commissioned the building of a new, dual fortification system. She also wanted the new castle to be a safe place for her daughter, Margit, who was living in a Dominican Monastery, in case of a new wave of attacks. By the end of the 1250's, the Upper Castle (Citadel) was completed on the top of the 328 meters tall hill. The Lower Castle, at the foot of the hill, was made up of barbicans, guard towers, fortified walls and a large residential tower. With its enormous size, strength and pomp, and because the royal family occasionally used as a residence, the Citadel had a special importance among the forts of Hungary.

The most important event in its history happened in November 1335 when the a two-month congress was held here with the participation of the Hungarian king, Charles, the Bohemian king, John of Luxembourg, and the Polish king, Casimir III. The Holy Crown of Hungary was kept in the castle from 1323 until the Ottoman conquest. The castle was fortified many times from the 13th to the 15th century but during the Ottoman siege it fell to ruin.

Its reconstruction began in the 1870's and it is still ongoing. Today visitors can see exhibitions about the Holy Crown, the history of the castle, the summit of kings, and hunting customs of noblemen. Visiting and walking through the castle and the exhibition takes about an hour.