I can see advantages if there was a method on Cake for privately requesting panelists as well as privately receiving their yes or no responses.
If I understand you correctly, I think panels already work this way?
When you start a panel, you can invite anyone you want to join the panel. If you enter an email address instead of the username of a Cake user, we'll send an email invitation to that person. If they're not already on Cake, they can sign up.
Everyone invited to participate in a panel has the opportunity to either accept or decline the invitation. People who accept become visible to everyone as a panelist. Only the creator of the panel can see who declined.