I have a belt case for a lens and I don't like to use it. Maybe it's just me, but I find it uncomfortable and it tends to pull my belt (and pants!) down. May I suggest:
1. Try it our first. Borrow or buy a cheap one and see if you really like it.
2. Use a separate belt. That might work better.
3. Take a look at small sling type bags. That may be a better choice. It might not even need to be a photo type bag. Check some outdoor stores and see what is available.
If I'm walking for any distance, I prefer a vest unless it's really hot. I carry my gear, including back-ups, in a rather small but efficient back pack (Canon). If I'm traveling light but need more than one lens, I have a small shoulder bag that I can wear as a sling but I'm going to replace it with an even smaller sling bag. At weddings I put a few extra things in my coat pockets (I always wear a suit) and keep my back pack close by. Just a few thoughts. I think I've tried it all when it comes to carrying gear. Thankfully digital doesn't require the amount of gear that film, especially 4x5, does so I travel much lighter today.