If you do hit gravel/oil, tracks at least have runoff areas or soft barriers, and no oncoming traffic or cliffs to worry about.

Every track day I've attended required the cars to have been inspected for leaks and loose components. And they were quick to clean up any debris or oil.

Not that tracking cars is the safest hobby, but if someone is gonna drive like that regardless its far safer at a track than on public roads.