You shouldn't do any damage, so long as you let them dry properly before putting them away. The key thing is not to allow any residual dampness to remain for 24 hours, as that could potentially start off a fungus problem.
The thing that puzzles me is why take them in with you, if you don't have time to let them acclimate? Depending on the temperature difference, even without bagging them up, they could take quite some time for the condensation to clear enough for you to take shots.
I have used my gear in such conditions on a number of occasions, never bothered bagging it, but just accept the sitting around for maybe 30 minutes or more, for the condensation in the lens to clear, as part of the process. You may well find it is significantly shorter though, depending on humidity (usually quite humid where I shoot).