When I regularly visited a BBS about my favourite TV series at the time, there were other regulars whose posts I always appreciated - and others whose posts would always rile me up.
When I participated in editing and later moderating a wiki, there were users where I knew their edits were good - and others were I knew that whenever they joined a conversation, it spelled trouble.
When I was active on a now-defunct social media platform, there were people I felt I knew well and liked to chat with - and others that I quickly put on an ignore list.
When I now visit the subreddit of a computer game I currently play, some usernames stand out, both in good and bad ways.
So, in one way you are right, having the "right" people makes or breaks a place. However, what's also true is that the great TV series BBS people I met aren't the same as the great wiki people, those aren't the same as the great social media people, and those aren't the same as the great subreddit people.
Cake is special in that it merges all these otherwise separate places on the web into one big platform without having any apparent borders or "silos", where everyone no longer is a member of this or that "interest group", but just coexists with everyone else. I might be coming to a conversation because of its topic A, while you are coming to the same conversations because of its topic B.
We're currently a somewhat small community, so seeing the same people in most conversations is inevitable. If Cake grows bigger, it would be great though to one day join a conversation to read someone I haven't ever read on Cake before, and vice versa.
Bottom line: People are important, but "always the same people, across all topics" isn't. :)