I had the IS version first, then returned it because I felt it was too pricey for what it was. The I bought the non-IS version before I came across a good deal on a used IS.
IQ-wise they are near indistinguishable. The differences you are paying for are just weather sealing and IS, nothing more.
I would not advise buying the f4 IS new; look for a nice used copy.
I found that the lack of IS was not particularly limiting for what I use it for, but I wanted the weather sealing, and IS "just in case".
Both are very good lenses. The non-IS is definitely more value for money, but the IS is the more versatile.
If you intend to shoot static subjects in low light without a tripod, then save up for the IS. If you shoot moving subjects most of the time or have a tripod with you always, you can save yourself the money and go non-IS.