Whit Stillman was born in 1952 in Washington, DC, to Margaret Drinker (née Riley), from Philadelphia, Pennsylvania, and a Democratic Politician, John Sterling Stillman, an assistant secretary of commerce under President John F. Kennedy (a classmate of Stillman's father at Harvard), from Washington, DC. His great-grandfather was businessman James Stillman and his great-great-grandfather, Charles Stillman, was the founder of Brownsville, Texas. Stillman grew up in Cornwall, New York and experienced depression during puberty. "I was very depressed when I was 11 or 12," he told The Wall Street Journal. "I was sent to the leading Freudian child Psychologist in Washington, D.C. It was heck. The last thing I needed to talk about was guilt about sex." However, when his parents separated, he found that his depression ceased: "I actually felt healthier."