Kluge was married four times. His first wife was Theodora Thomson Townsend, his second was Yolanda Galardo Zucco, the third was Patricia Maureen Rose, and his fourth was Maria Tussi Kuttner. Patricia Kluge has since filed for bankruptcy after taking on too much leverage during the recession. Kluge had three children, Joseph (whom he adopted), Samantha (with Zucco) and John Jr (adopted, with Rose). He also had three step children who remained a part of his life until his death: Diane, Jeanette Brophy and Peter Lockwood Townsend. He had homes in New Rochelle, New York, Virginia and Palm Beach, Florida with his fourth wife, Maria Tussi Kluge, at the time of his death in 2010.