I had an SL1 for a few months and never really warmed up to it, mainly because it didn't have an articulated LCD. I've also owned nearly every Rebel from the T1i to the T5i which my daughter now owns. I liked all of them and still do. I've also owned 40/50 and 60D's, which I liked as well, more then the Rebels, in fact, but were all on the big side.
I've switched back and forth to Olympus Mu4/3's bodies more then once, first with the OM-D EM-5 and recently to an EM-10 strictly for the sake of size and portability.
An EM-10 with a 20mm pancake weights about 570 grams, with a much smaller footprint. the difference in body size isn't huge, but the main difference is the size of the lenses.
Most Mu4/3 primes are tiny, fast and have great IQ, so I can have a great kit of fast primes that takes up about as much space as a 24-105. For travel it just can't be beat. I really hated hauling around ten pounds of Canon gear everywhere I went. The Oly body and four tiny lenses solved that problem for me.
The big issue I have with the Oly bodies is that they don't track moving objects well, especially if they're moving quickly towards you. It's possible to make it work, but it's not easy. If you can live with that drawback, and it's a big one, then I wouldn't hesitate to make the switch.
It's nice to have both systems, of course, because I definitely prefer optical viewfinders and Canon's AF, but since I picked up an OM-10 I haven't used any of my kid's Canon gear, and don't miss it.
The biggest plus is that I always have my Oly with me, because it's so easy to lug around. I could never say the same for my big Canon DSLR's. That is the biggest selling point for me.