This is appealing to authority, assuming the work ought to be perfect just because people are professionals. What if there is no good way to control for these latent factors? What if the latent factors change by the year and their magnitude of effect on the overall result changes? All this happens. No matter what you model with statistics, you are only ever presenting your model and its assumptions based on the data that were used with it. You can never say "now I understand reality" if you actually believe in statistics as they are and not just the popular religion of 'statistics'.
You are asking a reddit style question.
The entire world works around the legitimacy of authority.
Yes - the quality varies, and pollsters don't have a high degree of institutional authority.
But some do.
If you want to learn about 'institutional legitimacy' - go visilt Pew.
Pew is an extremely well regarded institution. They are extremely good at polling. Go look at their methods.
They are a long running institution with serious professionals, big budgets, people who really know waht they are talkinga bout.
It's the difference between getting diagnosis from 'a neighbour' vs a 'doctor' but in this case there is no specific qualification.
Some 'respected pollsters' are basically insiders who have been doing it for a long time, with relationships and credibility - others, more institutional.
Check out Pew, they are probably he closest thing to institutional professional as possible.
Political polling is inherently hard - they will be the first to admit.