Brown was born on July 28, 1891, in Holgate, Ohio, near Toledo, into a large family largely of Welsh descent. He spent most of his childhood in Toledo. In 1902, at the age of ten, he joined a troupe of circus tumblers known as the Five Marvelous Ashtons, who toured the country on both the circus and vaudeville circuits. Later he became a professional baseball player. Despite his skill, he declined an opportunity to sign with the New York Yankees to pursue his career as an entertainer. After three seasons he returned to the circus, then went into Vaudeville and finally starred on Broadway. He gradually added comedy to his act, and transformed himself into a Comedian. He moved to Broadway in the 1920s, first appearing in the musical comedy Jim Jam Jems.