I've had this problem as well... if I recall correctly, the actual retaining fixture for the plastic covers isn't very sturdy. So, the best thing to do is make sure that you disengage both clips along the top at the same time, using great caution not to damage the housing. If you do damage it, I've found that black silicone adhesive sealant works well to glue it back together!! Crazy glue doesn't work quite as well for some reason (though I did use it to fix my center console about 3 months ago without a problem yet). The plastic cover fell off one week after using crazy glue; after using the silcone it hasn't fallen off yet (3 months and counting)!