Born in Paris to Felix Passy, a veteran of Waterloo, and Marie-Louis-Pauline Salleron, Passy's uncle was Hippolyte Passy, a cabinet minister for both Louis Philippe and Louis Napoleon. Passy studied law and practised for a short time before accepting a position as an accountant in the State Council (Conseil de Droit) from 1846 to 1849. However, under his uncle's influence he gave up this post after three years and returned to study economics. True to his republican principles, he withdrew from politics after the coup d'état of Louis Napoleon and refused to be reconciled to the Second Empire; he was therefore ineligible for any government post. He became a professional Economist in 1857, and in 1860 he began to teach political economy both in Paris and in the provinces.