The aim of this study is to examine the extreme return spillovers among the US stock market sectors in the light of the COVID-19 outbreak. To this end, we extend the now-traditional Diebold-Yilmaz spillover index to the quantiles domain by building networks of generalized forecast error variance decomposition of a quantile vector autoregressive model specifically for extreme returns. Notably, we control for common movements by using the overall stock market index as a common factor for all sectors and uncover the effect of the COVID-19 outbreak on the dynamics of the network. The results show that the network structure and spillovers differ considerably with respect to the market state. During stable times, the network shows a nice sectoral clustering structure which, however, changes dramatically for both adverse and beneficial market conditions constituting a highly connected network structure. The pandemic period itself shows an interesting restructuring of the network as the dominant clusters become more tightly connected while the rest of the network remains well separated. The sectoral topology thus has not collapsed into a unified market during the pandemic.