Ronald "The Crab" Carabbia, 73, was released from Chillicothe Correctional Institution to his wife and son at 10:10 a.m., said Andrea Dean, spokeswoman for the Ohio Department of Rehabilitation and Corrections.
Dean said Carabbia will live with his wife in Columbus and have his first meeting with a parole officer Wednesday.
Carabbia was convicted by a Cuyahoga County jury in May 1978 following a 79-day trial for detonating the bomb that killed rival gangster Danny Greene in one of Cleveland's most infamous slayings.
The Ohio Parole Board voted 7-2 in July to recommend that Carabbia be released.
Carabbia was paroled despite objections from some law enforcement officials who say he still has criminal ties and could return to lead the Youngstown mob.
His attorney has said Carabbia is a frail man weighing less than 140 pounds, and there is nothing to indicate he would return to criminal activity.