That is what I did with my 96' Naturally Aspirated Park Ave, I gutted the box, only left a small part of it in tact, then put in a rectangular K&N air filter.
Usually a good CAI will pull from the fender area, and should have a heat shield around it, but I agree that the way the current box is that you'd think it would pull in colder air, maybe not quite as much, but colder nonetheless.
Good thing you took it off since the motor did not sound right, the AF ratio was probably out of wack.
2006 Pontiac Grand Prix GT with SC
1996 Buick Park Ave NA