But this seems to defeat the purpose of manual mode
Choosing shutter speed and aperture is "manual mode".
A camera could exist that let you select these two parameters, but has no ISO at all, and you just move a slider later on and make it as bright as you want it. ISO is only there because of historical shortcomings of recording media. It is not a significant photographic parameter.