The castle courtyard is entered at the south-west side by a small gateway between blind rounded piers across which a high arch spans. The gateway contains a four-centred arch with sunk spandrels in local Sutton stone and above that a square sunken armorial panel (Mansel quartered with Penrice and Scurlage) including the initials of Rice Mansel, in oolitic limestone. The high arch is slightly chamfered at the front and there is a machicolation-slot at its rear and a parapet walk as if for defence of the gate. To the right of the entrance is a semicircular tower, with two small Tudor windows; a similar tower probably existed originally to the left of the gate to complete the symmetry. The masonry apart from the carved work is local rubble or axe-dressed work, with many surviving putlog holes. The courtyard is square, with a low modern wall to the north-west side returning to the south-west side up to the entrance. Only part of the courtyard is paved. The more impressive part is the north-east range, surviving to almost eaves height on the side facing the courtyard but reduced to the level of the undercroft vaults on the outer side and internally. Local limestone rubble with sandstone stone dressings to openings. In front is the exposed foundation of the staircase building which gave access to the Great Hall. Five windows of the Long Gallery survive partially at the top, occupying the full length of the range. One window of the hall remains, its rear walled up; it consists of three tiers of diminishing lights, with thicker and thinner mullions dividing it horizontally into three groups of two lights. Above it is a relieving arch in the common masonry. To the right of this is an area of walling with a similar relieving arch, and the surviving jamb of a similar window but of four tiers in height, the latter window fully walled up. Beneath these to left is the hall doorway, now walled up, with a segmental arch. To the left is an area of four common storeys (not counting the Long Gallery) with three surviving two-light Tudor windows on the front elevation (one walled up). Ten small windows in what remains of the return north-west elevation, vertically staggered as some were lights to staircase landings. There is a Tudor arched undercroft entrance to the left of the site of the stairs block in the yard elevation, and two segmental-headed undercroft windows to the right, one walled up; square-headed doorway adjacent to the south block, slightly angled. There were three tower-like projections on the north-east side. To the south-east side of the courtyard is the lower range, consisting now of a two-storey four-window range, slate roofed. Restored Tudor windows above, three of four lights and one of two; label moulds with saltire terminations. One similar two-light Tudor window to the ground storey, but with no label, at the right; four-centred door arch beside this; three sash windows and a modern entrance with porch. The sash windows are in old openings, over which there are label moulds with saltire terminations. This block has a gable elevation to the south-west showing a two-light Tudor window with label in the attic, and a sash window below at first storey. Lateral chimneys and small rear wing (formerly a bakehouse) with end-chimney. The south-east elevation of the east tower has six storeys of Tudor windows with label moulds, of one or two lights with mullions and in some cases transoms. The architectural dress of this sun-lit elevation shows that it was to be seen to good effect from Oxwich Bay. The north-east elevation of the latter block has similar windows but without labels.