I think infrastructure is unlikely to change until it's forced to.
Cars had existed for years before anyone bothered to start paving roads, building gas stations, or designing cities with cars in mind. And I think self-driving cars will also have to exist for years before infrastructure begins to adapt to them.
So again, I want to understand what is the point of giving up the freedom, and yes for some even exhilaration & pleasure of driving a personal vehicle? Just so someone can get from A to B while pecking on their mobile or reading or sleeping?
I think self-driving cars will offer more freedom, not less. The idea isn't necessarily that every self-driving car will only work in self-driving mode. Some will work this way, sure, but you'll still be able to buy a car with a steering wheel and pedals that you can drive yourself.
The main difference will be that you won't have to drive the car yourself if you don't want to. So when you feel like driving, you can. But when you just need to get from A to B and you're not in the mood? Let the car do the work.