I would say that without a doubt it's Jim Brown.
Some basic facts.
James left Cleveland for Miami. Yes, he returned, but he also chose to leave.
Paul Brown went on to build yet another team down south.
While his situation was much different that James, both of these people aren't strictly tied to Cleveland sports alone. They have other sports ties that lie outside of Cleveland.
While baseball at one time was America's pastime, it no longer holds that title. While some may debate these points, it is my reasoning for holding jim Brown above all others.