Making it work in finite but short time should fix that. A large object generally can deform a larger distance. This makes all collisions inelastic, with large ones being different than small ones.

If you can get realistic billiards breaks, you're on the right track.