Agree, with the a/c clutch disengaged (climate control shut down, a/c turned off), the clutch is windmilling. An air gap adjustment on the a/c clutch may be neccesary.

I have the exact same problem on my 2000 SLE, A metal swirling noise that stops when any of the climate controls are engaged, i.e window defog or AC/Heater and then returns when the climate controls are disengaged.
I thougt it could be a belt but should I also check this clutch item?

Yea definitely check the AC clutch