Perceiving rapidly approaching sounding objects can be critical for survival. In studies of “auditory looming” perception, listeners consistently perceive sound sources as closer than they actually are, resulting in an underestimation of arrival time (Neuhoff, Planisek, and Seifritz, 2009; Rosenblum, Carello, and Pastore, 1987). This effect has been argued to provide an evolutionary advantage by allowing more time to prepare for the source. However, critical to this argument is the timely engagement of motor behaviors. Here, we tested the hypothesis that listeners with lower levels of physical fitness would have a larger anticipatory bias in perceived auditory arrival time, and thus a larger margin of safety in response to looming sounds. Listeners judged the arrival time of a three‐dimensional looming sound. Physical fitness was measured using recovering heart rate after exercise and grip strength. Results show that the anticipatory bias in perceiving looming sounds is negatively correlated with physical fitness (r=−0.41). Those least prepared physically to interact with a looming sound source have a greater perceptual margin of safety. The findings are consistent with an evolutionary explanation of the anticipatory bias for looming sounds and provide evidence for fitness‐based perception‐action links between the auditory and motor systems.