I've removed door panels before (with that excellent tool), the problem is the location of the rivet. It's squarely at the front of the door, and only access when the door is wide open. To make matters worse, it's the middle rivet, right behind the door hinge. Even at that, the clearance is only an inch at best, and the outside door panel is in the way, forming a V angle to the frame. I just don't want to scratch the door trying to get it in.
The hinge must have been catching on it and slowly pulling it out. I'll probably just end up taking it to the dealer instead of risking scratching the door. Maybe ask them to put some silicone on it so it doesn't come out again.