Been there... done that... got the t-shirt.
It does take a fair old pull to release at the front. However, it will come free without damage. Do try to use finger tips rather than a levering tool though, otherwise the plastic will be damaged.
I had one hell of a problem when I first tried to get mine off and that was because some cowboy had re-fitted the interior light sub carrier loosely rather than using the proper clips and so I was trying to pull both the console and the sub carrier out at the same time. Even so, it did eventually yield to pressure finger pressure although If I'd known then exactly why it was such a difficult bob, I'd have given it more thought because the deformation of the headlining is the sub carrier being pulled out with the console because it is not clipped properly. With hindsight I'd suggest a sort of contortionist approach, attempting to get the fingers of one hand in the gap to hold the sub carrier in place and pulling down sharply on the console with the other. I do hope that makes sense.
The fact that your headlining is deforming suggests that you may well have the same problem that I initially had.
If you don't mind me asking, please can you say why you are removing the console? The reason I ask is:-
a. because I am a nosy old git
b. The bulb sub carrier assembly is notorious for causing problems because the electrical bus bars are embedded in the plastic which unfortunately, degrades with time, causing lights failure, smell of burning and possible damage to the electronics in the console itself, manifesting as a burned PCB.