I have had an Em-1 for a while now. I really do love it. It is one of the best overall cameras I have used. Everything is so responsive and quick. I have had a wandering eye and tried a NEX and X100, but I keep coming back to Olympus. The sensor size is not really as issue for me. I love how small the lenses are. The 45 and 17 are tiny. You can stick them in a coat pocket and cover about everything you would want to shoot of you are just wandering about.
I think m4/3 represents the best compromise right now in terms of size and image quality for me. Fuji is coming out with some beautiful stuff, but I think I would miss Olympus's autofocus. It is reliable and fast. An in-focus image is indefinitely sharper than an out of focus shot.
Fuji and Sony are saying that theirs is the fastest, and it might be, but they have made those claims before and missed in my experience.