I think the fix is to make representative democracy recursive.

I can indicate other citizens that I trust on the basis of their expertise, and they can further indicate others in the same way.

The roots of those trees are my representatives for whatever domain, they have unequal standing proportional to the size of their constituency, and I can reconfigure my pointers towards different representatives at different times, or I can point them all at myself and show up in person for some direct democracy.