By design of all of the duct work, there will always be some air coming into the cabin even if the climate control is turned off. This air enters via the outside cowel inlet and travels to both the floor and defrost vents. Since the blower isn't operating while in the OFF position, the amount of air will be unnoticable during slow speeds or at a complete stop but can be felt under normal driving conditions. Since the incoming air passes by the air mix door, it will be heated or cooled according to the last temp setting. For instance, if you select 90 on the ECC head then turn the climate control off, the air mix door will remain in the "full hot" position and therefore warm any air that passes by it. If you select 60 and then turn it off, your little piggies will become cold on these chilly mornings as the air mix door will now be in the "full cold" position. The manual climate control works the same. If you leave the dial set to HOT, what little air is coming in will still be heated.
The air mix actuator would be to blame if you select a higher temp and cool air comes out or select a lower temp and the air is still hot. We get many of these topics as the seasons change.