The first thing to suspect is a bad pulley or tensioner.
Since your belt is already off, you should check each pulley as mentioned above, for noticeable differences.
It'* very likely a tensioner pulley or possibly the tensioner itself.
Have you noticed any odd noises coming from that side of the engine while running?
Does the pulley your mentioning look like this?
If not, then use this to let us know what you find:
Mike: 1997 SE w/ 183K, aka White Horse
, Gutted Airbox w/K&N, drilled 180* t-stat, tint all around, clear corners w/SilverStars, "PONTIAC" light up tail, PEMs, NGK TR55s