You create formulas to estimate the cost of running a given query plan. Use statistics collected about the tables (e.g. how many rows) to try to be accurate. The topic is "Cost Based Optimization".

Could you A/B at random, use that to collect data and eventually feed that back in to prefer A or B depending on the shape of the query?

Not someone familiar with DB internals, but could you run in parallel, at least for reads?