I had the spare batteries for my calipers taped to the inside of the case. The tape never stuck and they fell out regularly. This was my soluion.

Holds 3 CR 2032 batteries. The batteries friction fit and there is a space to lever them out with a screwdriver.