Rutherfurd was born and raised in New York City, New York. She is the daughter of Winthrop Rutherfurd, Jr. and Mary Spratt (née Kernan) Rutherfurd, himself the grandson of Winthrop Rutherfurd (1862–1944), a well known New York Society figure. Her paternal uncle is Lewis Polk Rutherfurd (b. 1944) who was married to Janet Jennings Auchincloss (1945–1985), the half-sister of former First Lady Jacqueline Kennedy Onassis, from 1966 until her death in 1985.