Lord Anglesey's widow, Elizabeth married in 1641 Hon. Benjamin Weston one of ten children of the financially astute Lord Treasurer to King James and King Charles, Richard Weston, 1st Earl of Portland; they continued to live there until she died in 1662. Multiple owners or tenants of short duration followed before it was bought by Field Marshal Viscount Shannon in 1718, who purchased from the estate of Sir Richard Pyne, retired Lord Chief Justice of Ireland (1695-1709); he bought the manorial rights which were until then held separately, and he died at the house, 20 December 1740. Lord Anglesey and Lord Shannon are buried and memorialized in the parish church of Walton-on-Thames. Lord Shannon's heir and only child Grace became by marriage The Countess of Middlesex (d.1763). By her Will and by virtue of her husband dying like her childless in 1769, the estate passed to her cousin Colonel John Stephenson and thereafter his sisters.
The house was inherited by a cousin of these sisters: Sir Henry Fletcher, Member of Parliament for Cumberland in 1786 and stayed in his family until it was bought by Sassoon David Sassoon shortly before his death in 1867; his grandson Sassoon Joseph Sassoon and young family were the final owners of the house.
Ashley Park Golf Club appeared in the 1890s but ceased to exist prior to the First World War. The house itself remained in the Sassoon family until it was demolished in 1920.
