How We Assess the Evidence

Written by Dr Jatin Joshi, co-founder and Chief Medical and Scientific Officer of Personally, and a researcher at the University of Oxford's Centre for Evidence-Based Medicine. I have a financial stake in this company, and you should weigh what follows with that in mind.

Most people who are sceptical of supplements are sceptical for good reason. I was one of them before I started this company, and most days I still am. So instead of opening with a number (how many papers we've read, how many studies sit in our library), I want to open with the actual question you're asking, which is not "is there evidence" but "how do you tell strong evidence from weak, and how would I know the difference myself?".

That's a fair question, and it's answerable. I spent most of my career on the other side of it, as a surgeon reading trial data to decide whether to change how I operated. The habits from that world are the ones I've brought here. Here is the process, step by step, including the parts of it that are still imperfect.

Most supplement marketing starts from the ingredient: here is a compound, here is a study that mentions it, here is why you should take it. That's backwards, and it's backwards in a way that's easy to miss because it reads as though it's led by science. It isn't. Starting with the compound and working forward to a reason to sell it is starting with the answer.

We try to start from a physiological question instead: does this population, doing this thing, in this state, have a nutrient need that ordinary diet is unlikely to meet on its own? Someone on a GLP-1 medication whose intake has dropped sharply is a different physiological situation from someone eating normally. Someone whose blood work shows a genuine gap is a different situation from someone who is simply anxious about their diet. The ingredient is the answer to that question, not the starting point, and if we can't state the question clearly first, we don't have a case for the ingredient, however interesting the molecule is on its own terms.

Before we look at what a study found, we look at who it was about. A trial in postmenopausal women with an existing deficiency tells you very little about a thirty year old with normal blood levels. A study of a cognitive supplement in people already showing signs of decline doesn't tell you what it does for someone in their thirties with no such history. A great deal of supplement science fails at exactly this step, borrowing a finding from one population and applying it to a different one because the label fits and the study existed. We try not to make that mistake, though I'll be honest that catching it consistently, across every ingredient in a formula that can carry forty or more of them, is a discipline rather than a solved problem. It's the sort of thing that's easy to get right once and quietly slip on the fiftieth time.

This is probably the single most useful distinction in the whole process, and it's one the industry mostly ignores because it's inconvenient. If someone has a genuine, measurable shortfall, say a documented low ferritin or a diagnosed vitamin D deficiency, then correcting it has a clear evidence base and a clear mechanism: the body was short of something, it now has enough, function that depended on it recovers. That's a strong, well-evidenced case.

If someone's levels are already normal, giving them more of the same nutrient is a different question entirely, and the evidence for benefit there is usually much thinner. Sometimes there is none. We try to keep those two cases visibly apart, so that a study about correcting a deficiency doesn't quietly stand in for a claim about boosting normal function. That substitution happens constantly in this industry, usually without anyone involved quite meaning to mislead.

This is probably the single most useful distinction in the whole process, and it's one the industry mostly ignores because it's inconvenient. If someone has a genuine, measurable shortfall, say a documented low ferritin or a diagnosed vitamin D deficiency, then correcting it has a clear evidence base and a clear mechanism: the body was short of something, it now has enough, function that depended on it recovers. That's a strong, well-evidenced case.

If someone's levels are already normal, giving them more of the same nutrient is a different question entirely, and the evidence for benefit there is usually much thinner. Sometimes there is none. We try to keep those two cases visibly apart, so that a study about correcting a deficiency doesn't quietly stand in for a claim about boosting normal function. That substitution happens constantly in this industry, usually without anyone involved quite meaning to mislead.

This is the step the industry skips most often, and it is the one I care about most. For any given ingredient there is almost always a study you can find that says what you want it to say. A single positive trial is the start of a case, not the end of one. So the first move is not to reach for the best study, it is to ask what the whole body of evidence says once you put the flattering and the unflattering results side by side. Where a good systematic review exists, that is where we start, because a review that has gathered the trials and weighed them together is worth more than any one of them read alone. What we are looking for is consistency: do the results point the same way across different studies and different groups of people, or does the effect appear in one trial and vanish in the next.

Within that body, the studies do not all count equally, and size is not the same as quality. A large trial done badly is still done badly. So we look at how each one was run: was it randomised, so the groups were comparable to begin with; was it blinded, so neither the participant nor the assessor was nudging the result; were people analysed in the groups they were assigned to rather than quietly dropped when they didn't respond; and were all the planned outcomes reported, or only the ones that came out well. A small, careful, properly controlled trial can tell you more than a large, loose one.

There is also a thumb on the scale that is easy to miss. Small positive trials are exactly where publication bias does its damage. The studies that find nothing are the least likely to be written up, so a scatter of small positive results can look like a real signal when the negative studies simply never appeared. Industry funding tends to pull the same direction. When the evidence for something is a handful of small, upbeat, industry-funded trials and not much else, that is worth saying plainly rather than counting as a settled case.

When we have done all that, we are left with a genuine spread. Some ingredients, mostly the ones correcting an established deficiency, sit on solid ground: consistent trial data, a clear mechanism, a population that matches the person taking them. Others, particularly a lot of the botanicals marketed heavily for mood, focus or cognition, sit on much thinner ground: a few small trials, mixed results, often run in people who are older or already showing the problem the supplement is meant to help, which is not the same population as most people reading a wellness article. That doesn't make those ingredients worthless. It does mean nobody should be selling them to you as though the case were settled, and we try not to. The honest move is to say which tier a thing sits in, out loud, instead of blurring them together under one reassuring word like "proven".

The evidence doesn't hold still. A botanical that looked promising two years ago might have a larger, better-designed trial land next year that changes the picture in either direction, and a nutrient we thought was well understood sometimes turns out to need a caveat we didn't know about. Most supplement companies build their list once, at launch, and then defend it for years regardless of what the literature does afterwards, because revisiting it is expensive and admitting a change looks like weakness.

So this isn't a list we built once and now defend. It's reviewed and regraded as new evidence comes in, which sometimes means removing something we used to include, or lowering our confidence in it without removing it outright. I'd rather that happened in public, even when it means saying we were more confident than we should have been the first time, than not happen at all.

The honest end point of all this is that some things we'd quite like to include don't make it, because the evidence doesn't clear our own bar, or because we can't defend the dose we'd need to ship at the price and format we're working with. That's a genuine constraint, not a marketing line, and it means the library is smaller than it could be if we were less careful about it.

None of the above is something I invented for this company. It is the standard way clinical evidence is appraised in evidence-based medicine, the discipline I trained in through Oxford's Centre for Evidence-Based Medicine: ask who the population was, keep mechanism and outcome apart, weigh the whole body of evidence rather than the one study that suits you, judge how well each study was run, and be honest about how much confidence the evidence actually supports. I mention it as the source of the method, not as a badge for the label. If the method above doesn't hold up on its own, no affiliation should rescue it. If it does hold up, the affiliation is simply where I learned to do it properly.

You don't need to trust our marketing to use this. The same seven questions work on any product, including ours: what's the actual physiological question, who was the evidence about, is it correcting a real gap or topping up someone who's already fine, is it mechanism or outcome, how strong is the evidence really, has anyone checked it recently, and did anything get left out because it didn't clear the bar. If a brand can't answer those, that's worth knowing before you answer anything else about them.

I'd rather you left this piece able to spot the difference between method and spin generally than simply persuaded about us specifically. The second one, if it happens, should be a side effect of the first.

If this is the sort of thing you want to see us keep doing, that is exactly what the evidence library is. When we regrade an ingredient, or a new trial changes where something sits, we write up the reassessment plainly. You are welcome to have those sent to you as they are published. There is nothing to buy attached to it, it is simply the working shown.

If you would rather read on, two pieces follow naturally from this one. Why doctors are sceptical of supplements applies the same standard from the other direction, and the patent, explained shows what the method looks like once it is built into how a formula is actually made.