I don't like to think of it in terms of freedom. Everyone has the same freedom to use public highways. The difference is responsibility. If you decide to use a bicycle you have a bit more responsibility than if you walk because they can go faster. If you decide to use a motor vehicle then it should come with a huge amount of extra responsibility. Ultimately you have the added responsibility of ensuring the safety and freedoms of those more vulnerable road users. So it's not "less freedom", it's "greater responsibility".
The problem is that there are effective mechanisms for punishing irresponsible drivers (and motor cyclists), but none (at least here in the UK) for punishing irresponsible cyclists.
The other problem is that cyclists often share space with pedestrians. I see cyclists on the pavement all the time (mostly illegally - and without consequences), I never see a car driving on the pavement (some park on the pavement, which is annoying).