What a beautiful thing. Who'* the jerk that designed this?
The idea is to move the inner duct/box assembly, slide down and access the two nuts through the rear vent opening. I've only been able to do this with the console removed from the car. There are 4 7mm screws that secure the duct. You already found the two behind the access panel near the lighter plug. The other two are on the bottom of the console to the rear of the rear mounting slots. The duct assembly can then be slid downward and forward to access the two nuts.
Here are some pics. Sorry, I don't have any imaging programs on this computer so I couldn't add arrows to point out locations. Note: all of the fasteners have been removed so you will not see ant screw heads... just holes.