Spence was born in Brownwood, Texas, to Roy Milam Spence Sr. and Ruth Griffin. He attended Brownwood High School, where he was a member of the 1965 Class 3A state championship football team. He then went on to enroll at the University of Texas at Austin, and went on to found GSD&M in 1971.