According to the Washington Post, they've had a Seventh Day Adventist, when only 2 members of Congress were 7dA. In theory, then, you'd think Judaism would have a chance...but yeah, that'll be a wait, I'd say. As for atheist chaplains, apparently there aren't any in the armed forces yet (though I recall they had their first pagan one years ago), and armed forces chaplains are sometimes used as a precedent for this kind of chaplain.
My feelings are mixed. When I think about just having a thoughtful, intelligent person, trained in this specific cross-religion activity of chaplaincy, available to lawmakers for discussions or counseling, that makes some sense to me. If you replaced them with a therapist, the therapist might not be able to discuss ethical questions with lawmakers freely if they thought it was non-therapeutic -- whereas chaplains are free to nettle or discomfit a congressperson. The WaPo article I linked first quotes a Catholic policy adviser saying “Chaplains should not make us comfortable. They should challenge us."
But I am willing to bet VERY FEW if any politicians over the years have trundled to the chaplain to discuss the ethical underpinnings of their votes, or what they should do about X or Y policy. That's the sort of thing TV politicians do, not real ones. So all they're actually doing is opening the proceedings of our legislative branch with prayers. Which does feel wildly inappropriate in terms of the Grand Separation.
I like ritual though, and I think beginning something serious benefits from a moment of ritual. Sometimes the place of prayer in the public life of the republic can be filled by poetry. Only Democrats have had inaugural poems thus far, but I think they're rather lovely. It might be a little whimsical as a way to open Congress, however. Maybe they could have someone dramatically read sections of the Constitution? But then some lector would read the Fifteenth Amendment and get booted for being too political.