Miller was born in The Bronx on April 14, 1917, and grew up in Flatbush, Brooklyn, rooting for the Brooklyn Dodgers. His father, Alexander, was a salesman for a clothing company on the Lower East Side in Manhattan; and, as a youngster, Marvin walked a picket line in a union organizing drive. His mother, Gertrude Wald Miller, who taught elementary school, was a member of the New York City teachers union, now the United Federation of Teachers.