← All articles Research

Beyond the Gold Standard

What clinical trials can't tell you, and why that matters for natural health

9 minute read

"There's no evidence for that." Few sentences close down a conversation about natural health so quickly. It sounds like the voice of science itself. Yet when a major review examined more than 1,500 healthcare treatments assessed by Cochrane, the most respected reviewers of medical evidence, only 5.6% had high-quality evidence of benefit.1 That figure covers treatments of every kind, conventional and natural alike. It is no reason to dismiss medicine, and it should make everyone slower to use "no evidence" as a verdict.

A brilliant tool for one kind of question

The randomised controlled trial, or RCT, deserves its reputation. Divide people at random, give one group the treatment and the other a placebo, keep everyone in the dark about who received which, and you have a powerful way of asking a narrow question: does this one thing, in this kind of person, change this measurement, on average, over this length of time?

For a new drug, that is exactly the right question. Trials have exposed harmful treatments and protected millions of people, and they have overturned plenty of confident medical opinion. When researchers reviewed ten years of a leading medical journal, 146 of the 363 studies testing an established practice found it no better than a lesser alternative: four in ten.2 That is science doing its job.

The difficulty comes when a tool built for one kind of question is treated as the only acceptable answer to every question. Even the doctors who shaped evidence-based medicine never meant that. Their founding definition brought together clinical expertise and the best outside evidence, and they said plainly that it was not restricted to randomised trials.3

A trial tells you what happened on average, to people chosen to be alike. You are neither an average nor a selection.

Who gets left out

To get a clean result, a trial selects a tidy group of people. Those with other conditions, other medicines or complicated lives are often turned away. Researchers have a name for the problem this creates, external validity, and a landmark paper in The Lancet asked the awkward question directly: to whom do the results of this trial apply?4

The answer can be startling. When New Zealand researchers checked people with asthma in the community against the entry rules of the 17 major trials behind international asthma guidelines, a median of just 4% would have qualified.5 A wider review of trials in top general medical journals judged fewer than half of all exclusion rules to be strongly justified.6

Now picture the people who come to see a naturopath. Tired, bloated, sleeping badly and anxious, with a thyroid that has been "borderline" for years and two medicines from different clinics. These are precisely the complicated, overlapping stories that trials are designed to screen out.

One thing at a time

An RCT works best when one thing changes and everything else holds still. Natural medicine does the opposite. A naturopath might change someone's breakfast, their evening routine and how they breathe under pressure, support their digestion, and choose a herb for their particular constitution, all at once and all adjusted as they respond. Two people with the same diagnosis may leave with quite different plans.

Forcing that into a trial often means stripping out the very thing that makes it work. Nutrition scientists have made this case in their own journals: nutrients behave differently from drugs. Their effects tend to be gradual, they act together, everyone already has them in their body, and there is no true "placebo group" that eats nothing at all. Evidence-based nutrition, they argue, has to include trials and go beyond them.7

Trials that do respect this complexity can be striking. In the SMILES trial, adults with moderate to severe depression who had seven sessions of personalised dietary support were far more likely to reach remission over 12 weeks than those given social support instead: 32% against 8%.8 In PREDIMED, a Mediterranean diet rich in olive oil or nuts cut major cardiovascular events by about 30%, a result that held after the trial was corrected and republished in 2018.9 Food, studied as food, can do a great deal.

Some questions can't be trialled at all. A wry paper in the BMJ searched for randomised trials of parachutes, found none, and suggested that the fiercest advocates of trials might like to volunteer for the first.10 Nor can anyone randomise thousands of people to a lifetime of one diet. That's why we rely on the consistent pattern across many kinds of study, the same reasoning that established the harm of smoking.

Who pays, and what we never see

Good trials are expensive. The trials that won approval for new medicines in the US in 2015 and 2016 cost a median of about $19 million each, and the largest ran into hundreds of millions.11 A company can recover that through a patent. Nobody can patent nettle, or a walk in the woods, or a bowl of lentils. This is the plainest reason that so many herbal and lifestyle trials are small: the money isn't there. In the US, the national centre for complementary health research receives less than 1% of the national health research budget.12

Money shapes results as well as topics. A Cochrane review of 75 papers concluded that when manufacturers sponsor studies of their own drugs and devices, they get more favourable results and conclusions than other funders, with a bias that the usual quality checks don't pick up.13

Then there is the evidence that disappears. The US regulator holds the records of every registered trial for a drug, which allowed researchers to compare them with what actually reached the journals for 12 antidepressants. In the published literature, 94% of the trials looked positive. In the regulator's own files, 51% were.14 Nearly a third of the trials were never published at all. A doctor reading every journal faithfully would still have seen a rosier picture than the full truth, through no fault of their own.

The double standard

Here is where it matters for natural health. Herbs and whole-person care are often dismissed because large trials are missing, by the same system whose funding makes such trials unlikely. Small studies are waved away as too small, without asking why they are small. Benefits such as calm, clarity and feeling well again are set aside as "subjective", although they are often exactly why someone sought help.

Statisticians have warned about this confusion for decades. As two of the best known put it in the BMJ, absence of evidence is not evidence of absence.15 "Not yet studied well" and "shown not to work" are entirely different claims. The second needs large, well-run trials and major reviews in agreement.

Where herbs have been put through large trials, they can hold their own. A Cochrane review of 29 trials and more than 5,000 people found that the St John's wort extracts tested beat placebo in major depression, worked about as well as standard antidepressants and caused fewer side effects.16 St John's wort also interacts with many medicines, including the contraceptive pill, which is exactly why herbs belong with a practitioner who knows your constitution and your whole picture.

Tradition has its own record too. When the Chinese scientist Tu Youyou searched for a malaria treatment in the 1960s and 70s, she went to the old texts. A fourth-century handbook by Ge Hong described soaking sweet wormwood in cold water rather than boiling it. That detail led her to a low-temperature extraction, and on to artemisinin, one of the most important malaria medicines in the world, and a Nobel Prize in 2015.17 Centuries of careful observation pointed the way, and modern science confirmed it. The two work best together.

Five questions to ask when you hear "there's no evidence"

  • Has it been studied properly, or simply not studied much? Those are very different things.
  • Who would have paid for a large trial, and what would they stand to gain?
  • Were the people in the trials anything like me?
  • Was it tested the way it's really used, or as one isolated ingredient?
  • What else do we know: from long traditional use, mechanisms, population studies, and people's lived experience?

Evidence with a person at the centre

A good practitioner holds several kinds of evidence at once: the published research, including its gaps; the long experience of traditional medicine; the patterns seen in clinic; and, most importantly, what actually happens to you. Researchers call a careful version of that last idea an n-of-1 trial, where a single person is the whole study, and some leading scientists argue it deserves far more serious attention in an age of individualised care.18

In practice, it looks like something very sensible. We note how you are now, change something with good reason behind it, and watch what happens to your energy, your digestion, your sleep and your test results. Then we adjust. Your own body becomes the most relevant evidence of all.

None of this is an argument against science, or against the medicines and surgery that save lives every day. Please never stop or change a prescribed medicine without speaking to your prescriber. It is an argument for more science, of more kinds, asking better questions, and for a little more humility from everyone, natural medicine included. If you would like a plan built around your own story as well as the research, a Rootwise consultation is a good place to begin.

In short

  • Randomised trials are powerful for narrow questions, especially about single drugs, and they have real limits.
  • They often exclude the complex people most likely to seek natural care.
  • Patents and funding decide what gets tested, and unfavourable results can go unpublished.
  • "Not well studied" is very different from "shown not to work".
  • The best care combines research, tradition, clinical experience and your own response.

References

  1. Howick J et al. Most healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic review and meta-analysis. Journal of Clinical Epidemiology, 2022.
  2. Prasad V et al. A decade of reversal: an analysis of 146 contradicted medical practices. Mayo Clinic Proceedings, 2013.
  3. Sackett DL et al. Evidence based medicine: what it is and what it isn't. BMJ, 1996.
  4. Rothwell PM. External validity of randomised controlled trials: "to whom do the results of this trial apply?" The Lancet, 2005.
  5. Travers J et al. External validity of randomised controlled trials in asthma: to whom do the results of the trials apply? Thorax, 2007.
  6. Van Spall HGC et al. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA, 2007.
  7. Blumberg J et al. Evidence-based criteria in the nutritional context. Nutrition Reviews, 2010.
  8. Jacka FN et al. A randomised controlled trial of dietary improvement for adults with major depression (the "SMILES" trial). BMC Medicine, 2017.
  9. Estruch R et al. Primary prevention of cardiovascular disease with a Mediterranean diet supplemented with extra-virgin olive oil or nuts. New England Journal of Medicine, 2018.
  10. Smith GCS, Pell JP. Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ, 2003.
  11. Moore TJ et al. Estimated costs of pivotal trials for novel therapeutic agents approved by the US Food and Drug Administration, 2015-2016. JAMA Internal Medicine, 2018.
  12. National Center for Complementary and Integrative Health. Congressional budget justifications and appropriations history, US National Institutes of Health.
  13. Lundh A et al. Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews, 2017.
  14. Turner EH et al. Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 2008.
  15. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ, 1995.
  16. Linde K et al. St John's wort for major depression. Cochrane Database of Systematic Reviews, 2008.
  17. Tu Y. The discovery of artemisinin (qinghaosu) and gifts from Chinese medicine. Nature Medicine, 2011.
  18. Lillie EO et al. The n-of-1 clinical trial: the ultimate strategy for individualizing medicine? Personalized Medicine, 2011.

This article is for general information and isn't a substitute for personal advice. If you take prescribed medicines, please don't stop or change them without speaking to your prescriber, and check any herbs or supplements with your GP or pharmacist.

Evidence that starts with you

A personal look at your history, your body and what changes when you change, drawing on the best research and the long experience of natural medicine.