> However, you could also (probably) define the derivative as lim_{h->0} (f(c+h) - f(c-h))/2h, so without needing f(c) to be defined. But that's not standard.

Although this gives the right answer whenever f is differentiable at c, it can wrongly think that a function is differentiable when it isn't, as for the absolute-value function at c = 0.