> Wouldn't it be nice if policy changes were accompanied by an A/B testing plan to evaluate their impact? I have always thought so. I have also seen a major pitfall of A/B testing that real humans can hand-pick and slice data to make it sound as positive or negative as wanted. Nonetheless, the more data the better.

Policies have different effects depending on how likely people judge them to be long-term changes. Construction along a route will cause people to temporarily use alternative forms of transportation, but not e.g. sell their car or buy a long-term bus pass.

Yes, the inability to know counterfactuals will make judging policies more subjective than we might like. The closest we get to A/B testing is when different jurisdictions adopt substantially similar policies at different times. For example, this was done to judge improvements from phasing out leaded-gasoline, since it was done at different times and rates in different areas.