When the diopter is set at the "neutral" poing (-1 diopter), the virtual distance of the focusing screen is one meter from your eye. For most people, that requires "distance" correction, if correction is necessary.
I'm extremely nearsighted (-8.5 diopter correction) so I have to wear glasses to see clearly anything more than 8 inches from my face, including through the viewfinder. That's been for 40 years of shooting now.
I did wear contacts for a long time, but I found the problem then was that my eyelashes kept depositing oil on the viewfinder eyepiece. I'm not sure whether I was more aggravated by that or by not easily seeing the entire viewfinder with my glasses on.
I'm an old guy, so I'm also presbyopic (farsighted), which means I need bifocals. I do very well with progressive bifocals, so that's okay. I tried bifocal contacts, but I discovered bifocal contacts are for people who want to look good, not for people who want to see well. All the designs tend to obscure sharp vision.
One advantage to being very nearsighted has always been that my close vision is good--I never needed a loupe in my film days, and I don't need one for Live View. I just peek over the top of my glasses.
I can't imagine constantly flipping my glasses on and off for any shooting other than maybe landscapes.