The effect you got is definitely motion blur, and definitely not diffraction. For diffraction to be that visible, you'd have needed an oval aperture of perhaps f/128.
F/22 on APS format will cause a slight softening. It's caused by an interference pattern of light waveforms around the edge of the aperture.
To your original question: Yes, buildings can and do sway in the wind. In this case, the lighted tower thing was also swaying, and seemingly to a greater extent than the building you were on. But those street lights look as though they smeared through a couple feet, which would require either a very slight twisting motion, or an exceedingly strong swaying motion. I don't think those trees would still be upright, so I'm going with the former, if building motion it is.
IS run amok could do this easily. So could a slight bump of the tripod halfway through the exposure.
Rick "who has had a bunch of very long exposures go awry for reasons unknown" Denney