Weight affects speed with minimum sink. That affects the diameter of the circle you fly. Since thermals have more lift towards the center (assuming perfectly circular thermals), you are not able to circle in the strongest lift. So you climb more slowly.

You can glide faster with the same L/D, so that might be worth it if you try to optimize for speed.