Yes indeed water and wind will draw the heat out of the metal sides of the smoker.
Just hold your hand near an old style single pane window on a cold day to feel an example of how much heat is drawn out through the metal sides. Think of the Q as a little old man sitting in a one room cabin. He likes it nice and warm, but any air leaks whrough the windows or the uninsulated walls causes the room to be cold no matter how much he stokes up the stove. But on a warm day, it is nice and comfortable in the room.
You gotta keep the charcoal loaded at a faster rate and open it up to hold the temps inside, or the cooking rate will slow and it will take much longer the finish.
Try to keep the water off the smoker if you can. It consumes much more heat than the wind by itself.