Frère is married and has three children. His first wife was Nelly Poplimont, and they had a son, Gérald. His second wife is Christine Henning, and they have had two children, Ségolène, and Charles-Albert, who died in 1999, in a car accident at the age of 19.