Following the initial conquest of the Welsh kingdom of Gwent by William Fitz Osbern, Lord of Breteuil in Calvados, between 1067-75, the Normans built a triangle of castles at Grosmont, Skenfrith and White Castle to control their newly won lands. These late C11 defences would have been of earth and timber. At White Castle the great outer ditch of that ringwork castle still survives, the perimeter of which would originally have been protected by a wooden palisade.
The earliest masonry structure at White Castle, built probably in the early C12, was a small rectangular keep- tower now demolished which stood within the inner ward. During the reign of Henry II, the surviving Exchequer Accounts record an expenditure of £128. 16s. on work at White Castle. This substantial expenditure must relate to the construction of the great six-sided stone curtain wall which encloses the inner ward.
In July 1201, King John granted the three castles to Hubert de Burgh, who was later was captured whilst fighting for King John in France. Whilst he was still prisoner John gave the castles to William de Braose of Abergavenny, in royal favour at the time. On Hubert's release, a prolonged dispute ensued over the ownership of the castles, but in December 1218 this was settled by the King's Court in Hubert's favour. After Hubert fell from power in 1232, he was forced to surrender the three castles which shortly afterwards reverted to the Crown and were placed in the charge of a royal officer, Waleran the German.
In 1244 Waleran built a new hall, buttery and pantry at White Castle. Shortly afterwards, in 1254, the three castles were granted to Henry III's elder son, later King Edward I.
White Castle played an important strategic role during the rising of Llwyelyn ap Gruffydd, Prince of Gwynedd in 1262. The Welsh conquered many of the English-held castles, but Abergavenny and White Castle, which did not fall, became important frontier fortresses.
As a defence against Llywelyn, White Castle was refortified in the military mode of the late C13. Great round towers were added at the angles of the curtain wall of the inner ward, and the earlier C12 keep was demolished and replaced by a new inner gatehouse. At the same time, the outer ward was enclosed by a formidable new curtain wall with massive corner towers and a new outer gateway. These works were carried out sometime after 1263, probably by Edmund, Earl of Lancaster, the younger brother of Edward I.
With the subjugation of Wales by Edward I, the strategic importance of White Castle declined and by the C16 the castle had fallen into ruin. In 1825 the Duchy of Lancaster sold the ruins to the Duke of Beaufort and in 1922 they were given to the state by Mr Henry Mather Jackson.