Usually, when refineries flare something like that it's because what they are burning is not suitable for use, and making it suitable would cost more than the product would sell for.

Often methane as a by-product of oil production is flared, because the amount is small enough that it's not worth setting up processing plants and supply chains for. Other times, the fluid is heavily contaminated by e.g. sulfur compounds, and would be costly to purify. Still other times the production of the fluid is unreliable or intermittent, and cannot sustain a continuous production process.

Although, flare gas recovery systems exist nowadays to make use of these waste gases, commonly for local power production for the refinery itself.

It's usually a small amount of waste, and handling gas is very different from distillate.

You'd need to either liquify that gas or collect it to a pipeline in order to make it useful. I remember reading that modern refineries make use of the gases instead of flaring them though I'm not sure how.

They flare to quickly burn off excess gases as a safety mechanism rather than anything else. Venting gas into the air would be much worse.

the way it was explained to me is if you see the flares then something is wrong. It may not be catastrophic or anything serious but something isn't going according to plan. Because you're right, why burn it off when you can sell it?