SARS-CoV-2 infects less than 1% of cells in the human body, yet it can cause severe damage in a variety of organs. Thus, deciphering the non-cell-autonomous effects of SARS-CoV-2 infection is imperative for understanding the cellular and molecular disruption it elicits. Neurological and cognitive defects are among the least understood symptoms of COVID-19 patients, with olfactory dysfunction being their most common sensory deficit. Here, we show that both in humans and hamsters, SARS-CoV-2 infection causes widespread downregulation of olfactory receptors (ORs) and of their signaling components. This non-cell-autonomous effect is preceded by a dramatic reorganization of the neuronal nuclear architecture, which results in dissipation of genomic compartments harboring OR genes. Our data provide a potential mechanism by which SARS-CoV-2 infection alters the cellular morphology and the transcriptome of cells it cannot infect, offering insight to its systemic effects in olfaction and beyond.