Supply demand. As refineries got better at cracking fuel they made more and more gasoline since that is what the market wanted. However to change the mix they need to make a long term bet on it isn't worth it to build/rebuild their expensive refinery to produce a different mix. A refinery only has a small amount of ability to change what comes out, they can't make more diesel without a major rebuild and that isn't worth it.