Great post. The 'diet' industry is a billion dollar industry mostly based on nonsense. Even for experts in the field it's hard to distinguish fact from fiction and probability from unsupported. Longitudinal studies are the best source but you're right about trying to separate all of the factors involved. We also individually build up a view of what we think is right or makes sense in our world and we fall victim to observation bias in a way that keeps building the strength of our unsupported ideas. I forget the exact numbers but a fairly recent researcher found that something like 75% of science articles findings are later found to be untrue. This isn't because the researchers were lying or purposely manipulating the data but for a whole litany of reasons. I should find the article and add a link. (I found an article that touches on what I was talking about. It's not the article I was thinking of but it helps to get the conversation started.)
I'm sharing this as a person who teaches, studies and loves science. I think science is the best way of knowing that humans have ever created but it has some serious problems that we ignore at our peril.