I think that setting the most sensible defaults for new users is a very important thing to do - and I'd choose the same three combinations that you suggested.
Beyond that, though, I don't think that simply taking away options will help, unless you do it in the context of a more complete overhaul of the feeds&filters idea. The problem here is that different things are important and useful to different groups of users, and all of that might change depending on how much activity there is on the platform as a whole.
To give just one example, I'm currently using "All/New" as a secondary feed to learn about new conversations in topics I'm not yet following. Assuming that Cake grows, and that the frequency of new conversations being started will grow ten or a hundred times, "All/New" will eventually become too noisy for me, at which point I might want to switch to "All/Interesting" - or hope for you to come up with something else altogether. ;)
Repeating the suggestion I already made above, perhaps merging the feeds UI and the filters UI into a single object that can be edited by users would solve all these issues and more. It would allow you to offer sensible defaults (the currently four links, each going to a different feed with a matching filter), while at the same time allowing advanced users to change this to quickly access whatever combinations they consider to be most important.
For example, just like @StephenL I don't really see a need for a "Following" feed if I already have notifications. If I could edit the menu to get rid of that entry and replace it with a different combination of feed&filter (so that I perhaps have quick access to two differently sorted "For You" feeds, or whatever other feed types we might come up with in the future), this would be great.