Kahn had three children with three women. With his wife, Esther (1905-1996), whom he married in 1930, he had a daughter, Sue Ann. With Anne Tyng, who began her working collaboration and personal relationship with Kahn in 1945, he also had a daughter, Alexandra. When Tyng became pregnant in 1953, to mitigate the scandal, she went to Rome, Italy, for the birth of their daughter. With Harriet Pattison, he had a son, Nathaniel Kahn.