I own the EF 70-200mm f4L USM lens. I found it to be very good at IQ. I shoot it many times at high ISO, and shutter speeds over 1/200 sec. It is much lighter than both the 2.8 with IS, and f4 with IS, so shooting birds on a perch is easy. I've even had good luck shooting BIF, but at almost max IOS, and max shutter speeds. You do have to be more aware of the light when using it. I think if I had to do it again, I'd get a copy of the 2.8 with IS. I shoot a good bit in the evening hours, and sometimes the herons are just to dark for the f4. So, I'll be looking for a good used 70-200 f2.8L USM sometime soon, and will sell the f4 non-IS.
In any event you will see a greatly improved IQ from any "L" lens you may purchase. The EF-S high performer is the EF-S 10-22mm USM is as good as any "L" lens could be on a cropped sensor body. My EF-S 28-135mm IS STM lens is also pretty sharp, but I use mainly for street photography. The 100mm 2.8L IS USM is tack sharp, but I hear the non "L" version is just the same. Both with or without IS, both lenses make good portrait lenses. The 40mm STM is also a great prime lens for simi-wide angle shots, and portraits. You cannot go wrong by adding "L" glass, beware it is expensive, but there are some very good EF lenses that are just as good as "L" glass. The secret is the word "glass". Those really good EF lenses just don't have the same water seals, and dust seals as the "L" quality lenses, but have the same optics.