Shang in The Lancet, and the Argument About Which Trials Counted

A 2005 meta-analysis compared 110 homeopathy trials against 110 matched conventional trials — and its conclusions rested on a subset whose composition was disputed before the ink was dry.

The Paper and Its Design

Aijing Shang, then at the University of Berne, led a team that published "Are the Clinical Effects of Homoeopathy Placebo Effects?" in The Lancet on 27 August 2005. The study matched 110 placebo-controlled homeopathy trials against 110 placebo-controlled conventional-medicine trials, paired by clinical condition and methodological characteristics. The matching strategy was designed to allow a direct, condition-by-condition comparison rather than pooling homeopathy trials in isolation — a methodological refinement on earlier work, including Klaus Linde et al.'s 1997 meta-analysis, which had examined homeopathy trials without a conventional comparator arm.

The conclusions section, where a review states its own finding rather than a summary of it.Photo: MART PRODUCTION / Pexels

Shang's team applied a funnel-plot analysis to test for publication bias, and then focused on what the paper called "higher quality" trials — those scoring above a pre-specified threshold on the Jadad scale and above a size threshold. The full 110-trial pool showed a small positive effect for homeopathy; when the analysis was restricted to the higher-quality subset, that effect shrank to odds ratios consistent with placebo. The paper's summary statement, appearing in the same issue alongside a Lancet editorial that called for an end to "the debate," was that the evidence was "compatible with the notion that the clinical effects of homoeopathy are placebo effects."

The Dispute Over the Subset

The controversy that followed centred not on the pooled results but on the composition of the high-quality subset. The paper initially published in The Lancet ↗ did not identify which specific trials had been included in the decisive subset — a detail that drew immediate correspondence. Several researchers writing to The Lancet in subsequent weeks argued that a subset of eight trials (the exact number Shang's team ultimately confirmed for homeopathy) was too narrow a base on which to anchor a categorical conclusion, and that the identity of those trials should have been disclosed in the original publication.

When The Lancet published the list of trials in its correspondence section, critics including external commentators examined the selection and argued that small methodological changes to the inclusion criteria would have altered which trials qualified and, consequently, the direction of the result. Edzard Ernst, among others, noted in correspondence that the subset size magnified the influence of any single trial. The Australian National Health and Medical Research Councillater cited the Shang paper as a reference point in its own 2015 systematic review, which applied different inclusion criteria and reached convergent conclusions by a separate route.

What the Paper Did Not Settle

The Shang analysis did not examine which homeopathy traditions or potency levels were represented in its trial pool, and the 110 trials covered a heterogeneous range of conditions and preparations. The paper's funnel plot for homeopathy showed asymmetry consistent with publication bias — meaning smaller, positive studies were over-represented relative to what would be expected if all conducted trials had been published — a pattern the authors factored into their adjusted estimates.

The core methodological objection raised in correspondence was not that the arithmetic was wrong but that the trial-selection rules, applied to a literature of this size, gave the subset-definition step disproportionate leverage over the outcome. Whether eight trials constitute a sufficient high-quality nucleus for a field-wide conclusion remained, after the correspondence closed, an unresolved question in the methods literature. The Shang paper is cited in subsequent systematic reviews both as a primary evidence source and as a case study in the difficulties of applying meta-analytic methods to a clinically heterogeneous and methodologically variable trial base.