Remnant of C12-13 castle, principally the shell of the great keep with two towers. The castle originally extended much further to both SW and NW, traces of buildings in a wider area have been revealed, for instance behind the SW side of Castle Street, and to the NW, near the NW end of the former County Gaol.
The castle is first mentioned in 1188 by Gerald of Wales, when it was held by Richard Fitz Tancred, whose father, from Flanders, probably founded the castle c. 1110. Robert Fitz Richard was ejected by King John in 1210 and the castle was granted to William Marshal of Pembroke. It was probably rebuilt then. It resisted an attack by forces of Llewelyn the Great in 1220. From the Marshals the castle passed to the de Bohuns, though taken from them 1265-74 for their part in the De Montfort uprising. In 1289 it was exchanged by the de Bohuns with Queen Eleanor, and large sums were spent by her in 1289-90, which may include much of what remains. It was a royal castle thereafter, though granted for life to Aylmer de Valence of Pembroke, 1308-24, to Isabella of France 1331-58, to Thomas de Felton c. 1370-80, John de Clanvowe 1383-91 and Thomas Earl of Worcester 1393-1403. It was attacked but not taken in the Glyndwr uprising 1405. There are records of a tower rebuilt, a new tower at the gate, and a new stable at this time. The pattern of grants to nobles continued in the C15, including William Lord Herbert 1462-69, and Jasper Tudor 1488-95, both Earls of Pembroke. Small works are recorded in 1472 and 1477. By 1577 the castle was in ruins. The remains were slighted in 1648, and the ruins slowly robbed of stone. They are shown more extensive in the 1740 Buck engraving. In 1780 a prison was built within the inner ward, enlarged with a debtors' prison 1816. French prisoners were held here in 1797. These buildings have gone, mostly cleared in the later C20. The surviving former County Gaol in the outer ward dates from 1820.