yes the whole panlel needs to come off..
o.k you take off all the screws first even the one behind the lil plactic peice with the lil white led light on it. then remove the light from the bottoms of the door , then remove the two peices of plactic around the door handle that lets you open the door and just pull out the lil clips that are holding the door panel on. they are all around the whole door panel so dont be affraid to brake one or 2 or 3 as you can buy a 24 pack of them at autozone for about $4-$5.
so what kind of speakers are you putting in.?
and welcome to the club.....