This is an active area of research (look up papers that mention the "sim2real gap"). It is indeed a problem, but there are known strategies to deal with it.