Where it goes after it's used is the problem.

Oil is mainly carbon, the carbon isn't destroyed by burning it. It just changes form. But now it's in the atmosphere instead of inside the earth. That's still a problem.