I would try moving the piece on the glove box lower. There is some give in that, and you may be able to get enough play to push it into alignment. I did some gentle pushing on mine, as it has the same issue, and it got it closer. This may be another of those, keep doing it kind of things, and over time, it should settle in.
I did not resort to out and out removal and trying to bend the trim piece.