Randomized controlled trials (RCTs), commonly referred to as A/B tests, have long been the gold standard for scientific evidence in medicine and are increasingly used in other fields as diverse as education, government policy, and poverty reduction
| DANVILLE, Pa., June 7, 2019 /PRNewswire/ -- Randomized controlled trials (RCTs), commonly referred to as A/B tests, have long been the gold standard for scientific evidence in medicine and are increasingly used in other fields as diverse as education, government policy, and poverty reduction. A/B tests help to determine whether a new treatment or policy helps or hurts compared to an alternative. A new study by researchers at Geisinger and Microsoft Research published last month in the Proceedings of the National Academy of Sciences sheds light on previous controversies over A/B tests, finding that people often approve of an untested policy (A or B) being given to everybody but object to an A/B test of those same unobjectionable policies designed to determine which one works best. The series of 16 studies with 5,873 participants from three diverse populations spanning nine domains in healthcare and beyond suggests an important obstacle to evidence-based policy and practice. One study involved differing methods of communication aimed at educating providers on best practices for avoiding infections from catheterizations. One group of participants was told that the hospital director decided to print a checklist of safety steps for clinicians on the backs of their ID badges. Another group was told that the director instead decided to print the checklist on posters that will be hung on the wall. A final group of participants was told that the director thought of both these ideas and decided to conduct an experiment in which patients are randomly assigned to a doctor with a badge or a doctor who works in a room with a poster. They were also told that once the hospital director learns which policy is best at reducing infection rates, he or she will implement it for everyone. Among the group who read about the director who implemented the badge policy, very few judged this decision to be inappropriate. So, too, with those who read about the director who implemented the poster policy. But many who read about the director who ran an A/B test to compare the badge and poster ideas thought this decision was inappropriate. According to the study’s lead author Michelle Meyer, PhD, JD, Assistant Professor and Associate Director, Research Ethics, in Geisinger’s Center for Translational Bioethics and Health Care Policy, A/B tests necessarily treat people differently, and potentially unequally—at least temporarily. “But strikingly, we found that people still object to A/B tests even when the alternative was still that different people would randomly get different treatments,” Meyer said. “Because of who they happen to have trained with, or what journal articles they happen to have read, some doctors prescribe one drug for things like high blood pressure—we called it Drug A—and others prescribe Drug B. Patients who are prescribed a blood pressure medication in the ER or in a walk-in clinic are basically being ‘randomly assigned’ to one drug or the other due to chance. Yet many of our participants find that unobjectionable but object to having their treatment formally assigned by an A/B test that would figure out which drug is best so it can be offered to everyone.” Follow-up studies found the same widespread disapproval of A/B tests despite approval of just giving everyone either A or B. Results were consistent in a variety of other domains, including consumer genetic testing products, autonomous vehicle design, and social policy programs. Doctors and participants with advanced degrees were no different than other participants. In examining the reasons participants gave for objecting to the A/B tests, the researchers found that people often assume, incorrectly, that experts already know “what works.” The researchers concluded that more research is needed to better understand why people object to experiments that compare unobjectionable policies and how to address those concerns. About Geisinger CONTACT:
SOURCE Geisinger |
View original content:
