What I ask myself is: is time purely the ordering of things that happen, or do gaps where there is potential for something to happen also count?

Let's assume for simplicity that time is a discrete dimension, which it might be. Then there would be a measure of distance of how many ticks of potential events there are between two actual events, even if nothing happened in between. Or maybe that's not the case and it's more of a directed graph defining the partial ordering of actual events.

Not sure if we could measure that in any case, we always need some kind of actually ticking clock, and it's not like we can isolate a period of time where nothing happens globally, unless its in a simulation. Just like weird things happen at quantum scale, I'm sure weird things happen at small enough time scales where there's really nothing between one event and the next, and there's no good way to determine how far a part they are.