After graduating from high school in Franklin, Mass, Crawford was accepted by Harvard College where he enrolled. However, after only three weeks at Harvard he dropped out to work as a stevedore on the New York docks.