I would start by checking a few items. The first is the pulley on the condenser. I had a similar problem with my air conditioning last year- wouldn't blow at all when the A/C was on, then would work, then wouldn't, etc. After a short while I started having this whining/belt squeeling noise when the a/c was on. I'd turn off the A/C and it would go away. Then it started getting more fluky, sometimes the squeal was when the A/C was off and turning it on would fix it, sometimes while it was on it would squeal and turning it off would fix, and sometimes nothing would fix. Finally, one night I was out with my g/f getting dinner and I smelled smoke. I pulled into a parking lot, popped the hood and noticed a huge cloud of smoke coming from my condenser. Shortly after, the belt snapped from the friction/heat and the car was stranded.
Long story short- first check the pulley on your condenser and make sure it spins freely.
Second, I would pop under your passenger side dash, and pull out your blower motor. I had a bad blower motor that I took out, cleaned out the coils and dumped a bunch of dirt out of, re-installed, made sure the connections were good, and it works like a champ now.