It seems to me that the popular sentiment, "yeah, there are rights, but they have limits and exceptions", essentially turns rights into mere privileges to be withheld or taken away if X conditions are(n't) met. I think the more accurate/waterproof way of thinking is "there are rights that always exist, but when they conflict, they are hierarchical", so for example life > mobility.
The crux of the issue of when rights conflict is that the more indirect, potential in nature, and subject to interpretation the threshold of being considered a rights-conflict is, the more room there is for abuse of arbitrary curtailing of rights.
For example, holding a gun to someone's head = direct and immediate threat to one's right to life VS going outside and interacting voluntarily with others who might interact voluntarily with you who might pick up a virus which has a death rate of .7 to 3.8 = highly indirect, highly potential, and highly subject to interpretation.
If the latter example's level is the threshold of where rights may be legitimately considered in-conflict and in need of curtailing, then the floodgates of arbitrary abuse of rights have been opened wide.
I'm not saying that the fire of government limiting of individuals' rights is never useful, but I am reminding that it is fire and shouldn't be played with, but used only when strict predefined conditions are met, contained in a fire pit, and with a water hose in hand. Fire only shrinks by itself when it has nothing left to consume.