You might be interested in The Order of Time by Carlo Ravelli. Time and gravity are certainly linked, but from what I took away from the book (which is a lot to digest, even as non-mathy as it tries to be) is that Time is really heat. Heat moves only from hot to cold, dispersing in some entropic fashion as we move toward the final state of the universe, but in the meantime we can measure that time/heat "flows" at different rates, depending on how near or far you are from large bodies.

I likely need to reread it, though, as some of its ideas are a bit above my weight class when it comes to understanding physics. But, you may enjoy it!