The seat successively of the Wogan, Dwnn and Philipps families in a uniquely unbroken line, the descent to Dwnn in c.1420 and Philipps in c.1490 having occurred by marriage with heiresses. The castle is believed never to have been besieged until the Civil War.
The castle replaced an earlier motte and bailey, at a site of strategic value to the English colony based on Pembroke. It is thought to have been built by Sir John Wogan, Judiciar of Ireland in the years 1295 to 1313. It was probably in existence in 1302, when Sir John was described as Dominus de Pyketon. No architectural detail survives to confirm this, but narrow trefoil-headed lights of appropriate apparent date are recorded in Buck's C18 engraving.
The original building was of unusual design, but planned on the concentric principle. It consisted of a hall raised above an undercroft and fortified by five main semicircular turrets, plus (but without any intervening open space) two narrower bastions flanking a portcullised entrance on the E side. A surrounding curtain wall gave a first line of defence. The earliest illustration, a small E view by Dineley (1684), shows the original E entrance to the undercroft, from which a staircase led up to the great hall. The main bastions were semicircular projections, two on each side elevation and one centrally on the W end elevation. The two nearest to the E end were the largest. The hall corresponded with two storeys of the bastions, and its traceried high windows are recorded in the Buck brothers' engraving. The hall had an E gallery.
In 1697 Sir John Philipps, the 4th baronet, demolished most of the outer curtain wall and formed a raised causeway leading to a new entrance at the level of the hall.
The second Sir John, the 6th baronet, undertook a complete remodelling of the interior in the mid C18. The architect of the remodelling is unidentified, but could have been James Gibbs; Sir John was in communication with a person of that name in 1752. As a Commissioner for the Fifty London Churches he was well acquainted with leading London architects. However, the letter from Gibbs, of unknown contents, is late in the project (which seems to have been under way by 1749). There was also a letter from the joiner James Rich, to whom the new gallery stairs and chapel interior are credited; by this he is known to have provided a plan for the chapel in 1753, and might have been in control of the whole design, except that his involvement, also, may only have been late in the project. Henry Cheere (later Sir Henry), mason, provided a number of fireplaces. Sir John was in communication with him on many occasions from 1749 onwards. All the plasterwork, panelling and joinery was renewed, floors relaid, sash windows substituted, and at the same time the grounds were re-landscaped.
The next radical change was the demolition of the W bastion and substitution of a large four-storey Regency W wing by Lord Milford. Fenton gives the date of this extension as 'about ten years ago', i.e. c.1801. He refers to the 'two magnificent rooms' provided on the hall level. The dining room fireplace is by Cheere, and may have been installed originally in the W bastion.
Lord Milford's successor, R B P Philipps, employed the architect Thomas Rowlands of Haverfordwest in the late 1820s to carry out further improvements in a Norman style (presumably in the view that the castle was of that period). The E entrance was given a Norman doorway and the chapel above it was given round-headed windows. Norman window heads were also added to the main S facing windows of the Regency wing. The old narrow causeway to the entrance was enlarged to a broad terrace with sweeping staircases down to the N yard and the S lawn. Estate improvements were carried out at the same period.
A surviving C19 undercroft plan shows the service layout of that time. There was a conservatory and a kitchen in N extensions, which have since been removed. The Butler's pantry was under the Regency wing, with other rooms; the four main bastions contained a salting room, a beer cellar, a servants' hall, and the Cook's room.
In 1960-63 some late attics were removed and the hall re-roofed under the directions of Donald Insall.