How to assess a randomized controlled trial (RCT)
1. Appropriate control arm?
The patients in the control arm should receive the best available care in the setting that the trial is supposed to influence. If a trial should change the standard of care in the U.S. and Europe, the control arm should resemble the standard of care in the U.S. and Europe. If the trial should change the standard of care in a low-income country, the control arm should resemble the standard of care in a low-income country.
A popular "trick" when designing a trial is to run the trial globally and justify a weak control arm by saying that the western standard of care was too expensive for some of the low income countries.
If the therapy in the control arm is up to the treating physician (i.e. "investigator‘s choice"), the choice should not be restricted to substandard options.
2. Does the endpoint matter to patients?
Patients care about living longer and/or better. Therefore, either survival or quality of life should be the primary endpoint of a trial. In some situations, where patients‘ life expectancy is very long, surrogate endpoints like progression-free survival are used. However, in those cases, there has to be good data that the surrogate endpoint has a high degree of correlation with either survival or quality of life.
3. Is the effect clinically relevant?
What we consider statistically significant is rather arbitrary and does not guarantee that the effect is also clinically relevant. If a trial enrolls a lot of patients, it can detect very small differences between its arms. However, a survival difference of two weeks, might not be clinically relevant, especially if it comes at the cost of a lot of side effects. The potential price of the new drug (which is usually high) should be considered as well.
While the trial authors should not be judged for how big the effect is, they should be judged for how they present the effect and its implications.
Further reading
The three questions outlined above are a very basic guide to assessing a trial. There are more things to consider and the game is continuously evolving. For a more comprehensive overview of things to look out for, check out this video from Vinay Prasad‘s lab.