Born in Clinton, Massachusetts, the son of Larkin Harry Brown (1866-1942) a cotton manufacturer and Katherine Ann Brown (née Gaw) (1865-1954), Brown moved to Tennessee when he was 11 years old. He attended Knoxville High School and the University of Tennessee, both in Knoxville, Tennessee, graduating from the university at the age of 19 with two degrees in engineering. An early fascination in automobiles led Brown to a job with the Stevens-Duryea Company, then to his own Brown Motor Car Company in Alabama. He later abandoned the car dealership after developing an interest in motion pictures around 1913. He was hired by the Peerless Studio at Fort Lee, New Jersey, and became an assistant to the French-born Director Maurice Tourneur.