Newton worked this out in what is now know as the shell theorem. If you have a hollow spherically symmetric body, then any point inside of the body experiences no net gravitational force. In contrast, points outside of the shell experience the same force as if the body were a point mass.

For ideal (spherically symmetric) planets where a point is underground, you can divide the planet into 2 regions. The shell of the planet "above" the point has no net effect, while the shell below has the full effect, resulting in the gravity falling towards 0 as you approach the center.

In practice, planets are not actually spherically symmetric, but are close enough for it to be a good approximation.