A C16 manor house built on the site of a medieval manor, greatly enlarged between 1682 and 1699 by the addition of an imposing classically-designed main range by Henry Somerset, 1st duke of Beaufort of Badminton, Gloucestershire, for his son and heir, the marquis of Worcester on his marriage. This new building further re-inforced the Beaufort influence over Monmouth, where the 1st duke had built Great Castle House in 1673.
The C16 house had been the property of the Herbert family, descended from an illegitimate son of the 1st earl of Pembroke. At the end of the C16 this was sold to William Somerset, 3rd earl of Worcester, whose successor Edward Somerset, 6th earl and 2nd marquis of Worcester, soon afterwards bought the manor of Great Badminton, to which Troy subsequently became subordinate. Edward Somerset's son Henry, president of the Council of Wales, was created 1st duke of Beaufort in 1682; and, Raglan being in ruins since the Civil War, built the present mansion at Troy as the family's main residence in Wales. It remained in the Somerset family until 1901, when the whole Troy estate, amounting to some 1,670 acres (676.35 hectares), was auctioned. Although the house itself was not sold at that time, in 1904 the Sisters of the Good Shepherd took possession of the house for use as a convent school; and subsequently, with the financial assistance of the Home Office from 1935, as an Approved School. That school having closed, the house is currently occupied by a caretaker.