Harilal was married to Gulab Gandhi and they had five children, two daughters, Rani and Manu, and three sons, Kantilal, Rasiklal and Shantilal. Rasiklal and Shantilal died at an early age. He had four grandchildren (Anushrya, Prabodh, Neelam and Navmalika.) via Rani, two (Shanti and Pradeep) via Kanti, and one (Urmi) via Manu. He became detached from his children after Gulab’s death in a flu epidemic.