Voluntary social distancing plays a vital role in containing the spread of the disease during a pandemic. As a public good, it should be more commonplace in more homogeneous and altruistic societies. However, for healthy people, observing social distancing has private benefits, too. If sick individuals are more likely to stay home, healthy ones have fewer incentives to do so, especially if the asymptomatic transmission is perceived to be unlikely. Theoretically, we show that this interplay may lead to a stricter observance of social distancing in more diverse and less altruistic societies. Empirically, we find that, consistent with the model, the reduction in mobility following the first local case of COVID-19 was stronger in Russian cities with higher ethnic fractionalization and cities with higher levels of xenophobia. For identification, we predict the timing of the first case using pre-existing patterns of internal migration to Moscow. Using SafeGraph data on mobility patterns, we confirm that mobility reduction in the United States was also higher in counties with higher ethnic fractionalization. Our findings highlight the importance of strategic incentives of different population groups for the effectiveness of public policy.