Yes, that was exactly the problem. I was doing something else on the pegboard and lightly touched the controller.
If the angle was just a little bit bigger, it would hold much better. I played around with different angles and a slight adjustment makes a big difference. Certainly not wrong for a controller in this price range.