Walters did not play in 1954 and 1955, however in 1956 he played for the Augusta Tigers and Charleston Senators, hitting a combined .251 with 20 home runs in 124 games. For the Birmingham Barons in 1957, he hit .275 with 17 home runs in 138 games, and in 1958 he hit .255 with eight home runs in 123 games for Charleston. He played for the Fort Worth Cats in 1959, hitting .291 with 21 home runs in 152 games. On December 5, 1959, he was traded to the Phillies with minor league Alex Cosmidis and Ted Lepcio for Chico Fernandez and Ray Semproch.