Duplicate Management - Avoid Record Reappearing in Duplicate Manager

This would mean that while I spent endless hours purging contacts from the Duplicate manager, deeming them as NOT to be merged (endless reasons) the system might re-offer these records again for me to AGAIN reject them.
We have a 1M user database. I thought I was slowly digging to the end of this merging tiring dynamic, 10K records at the times…but given the answer above…I will never finish.
This should NOT be allowed. IF I reject a merge, this should NOT be offered again. I can’t put more hours into this work when I now think I might be working over and over on the same records. Just NUTS.