Scott was born June 6, 1932, at Randolph Field (for which he received his middle name) near San Antonio, Texas. His father was Tom William Scott, a fighter pilot in the United States Army Air Corps who would rise to the rank of brigadier general; his mother was the former Marian Davis. Scott lived his earliest years at Randolph Field, where his father was stationed, before moving to an air base in Indiana, and then in 1936 to Manila in the Philippines, then under U. S. rule. David remembered his father as a strict disciplinarian. The family returned to the United States in December 1939. By the time of the attack on Pearl Harbor in 1941, the family was living in San Antonio again; shortly thereafter Tom Scott was deployed overseas.