You may be seeing coalesced activities. Here's how that currently works:
Instead of adding a new item for every single action that occurs (such as a reaction to your post or a new post in your conversation), Cake coalesces actions of the same type that occur within the same calendar day.
So if two people react to one of your posts, and you mark all as read, and then a third person reacts to that same post within an hour or two, you'll see that original activity marked as unread again because it now represents a third coalesced action that you hadn't previously seen.
To complicate things a little, we also currently have a bug where your own actions (like posting in your own conversation) can cause activities to get marked unread when they shouldn't be.
In addition to fixing that bug, I think there are a few other things we might be able to do to make things more clear when activities are coalesced, like listing the names of more than just one user involved.