I made the move from the 1.8 to the 1.4 and I was very happy. That being said, I don't know if the 1.4 is 3-4x as good. The 1.4 definately didn't have 3-4x better IQ than my 1.8 (I have a flickr set comparing the images)
However, the build was a huge upgrade. FTM was also another huge reason I switched.
I assumed the USM AF on the 1.4 would be blazingly fast, but I was very disappointed. The 1.4 micro USM was WAY slower than the ring USM on my 28mm 1.8. I'm sure the AF speed on the 1.4 is a bit better than the 1.8, but the real difference is the AF sound. 1.8 is loud, 1.4 is quiet. Another huge plus.
All that being said, I think the 50mm 1.8 is the best bang for the lens canon makes.