The randomized controlled trial (RCT) is often considered a gold standard in medicine. When done well, RCTs allow for rigorous evaluation of a relationship between an intervention and outcomes. However, there are many areas of medicine where an RCT is not possible or the results not generalizable, and clinicians are left in what some might call a “data-free zone.”
A recent example of an almost completely data-free zone for many clinicians was the COVID-19 pandemic.
“This was a scary time for everybody because I don’t think we are used to being in such a data-free zone as medical professionals,” said Stephanie J. Lee, MD, professor and associate director of the Clinical Research Division at Fred Hutchinson Cancer Center in Seattle and president of the American Society of Hematology (ASH) at the start of the pandemic in 2020.
“We didn’t know how to treat it or how to diagnose it, and we were concerned for our own personal safety on top of that,” Dr. Lee said. “Here was this pandemic and nobody was really an expert.”
ASH Clinical News spoke with Dr. Lee and other clinicians about the idea of data-free zones, the tools clinicians use to navigate these areas, and who the experts guiding treatment are.
Defining “Data-Free”
There are different types of data-free zones that all clinicians are faced with, perhaps the most obvious of which is rare conditions.
“Hematologists tend to see a lot of rare diseases where there simply are not enough patients to power a good, randomized trial,” said Menaka Pai, MD, associate professor in the Division of Hematology & Thromboembolism at McMaster University in Ontario.
Dr. Pai is a consultant for benign hematology, a field in which clinicians might see conditions like catastrophic antiphospholipid syndrome or thrombotic thrombocytopenic purpura (TTP). Her colleagues in malignant hematology are also faced with extremely rare diseases.
The idea of a data-free zone in these areas may be misleading, Dr. Pai continued, as it implies that clinicians are in a “dark cave with no light at all.”
“RCTs are like having an amazing, large flood light, and that is what we would love to always have to guide us,” Dr. Pai said. “Often we don’t have that but instead have a flashlight, or a candle, or just a match.”
Clinicians also face areas with few data when making very common decisions for which nobody has bothered to perform studies on, said David P. Steensma, MD, who until recently served as the Edward P. Evans Chair at Dana-Farber Cancer Institute and Harvard Medical School and is now hematology head at Novartis Institutes for BioMedical Research.
“It is not practical to do studies about everything, like how often to check blood counts, how often to do imaging, or what type of testing to do in every situation,” Dr. Steensma said. “In addition to rare disorders, many patients might also have conditions where, even though the diagnosis is common, like acute leukemia, their specific scenario – for example a patient who also has kidney failure or an unusual inherited syndrome or two malignancies at the same time – is rare.”
Dr. Pai said that social determinants can often dictate how “bright the light is” in certain areas of practice. For example, patients with red cell disorders or sickle cell disease are often marginalized and left voiceless, she said.
“For a lot of systemic reasons these patients may not be the subjects of as many trials as we would want,” Dr. Pai said.
For Lisa K. Hicks, MD, MSc, associate professor at the University of Toronto, data-free zones occur when patients are systematically excluded from RCTs. At her practice, for example, she sees a lot of patients with major comorbid illness, such as hepatitis, HIV, or active psychiatric diagnosis, and they are excluded from RCTs.
“These are more like low-data zones,” she said. “They introduce challenges around extrapolating from existing data.”
Clinician’s Toolbox
When there are not a lot of data available to guide decisions, clinicians have to use the information available to help the patient that is in front of them, according to Eric J. Seifter, MD, associate professor of medicine and oncology at the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins.
“That is exactly what our job is. We have to factor in everything including our own experience, our training, biases in our training, everything,” Dr. Seifter said.
One example of a situation in which clinicians must weigh a lot of factors is in the treatment of low-grade follicular lymphoma.
“Many of these patients can be treated with ‘watch and wait,’” Dr. Seifter said, but even the decision to monitor or treat will vary from physician to physician, and if the patient “shops around enough they will find a doctor who will treat them even if they don’t need treatment.”
In situations in which a clinician decides to treat, there are a potpourri of management choices, including treating with rituximab monotherapy, rituximab plus chemotherapy, or rituximab plus an immunomodulator such as lenalidomide, as well as a choice of treatment schedules or even a choice of maintenance therapy (or not).
“These are all complicated questions with no clinical trials to tell you a superior method,” Dr. Seifter said. Instead, one must use personal experience and training, or call trusted colleagues.
Dr. Hicks said that one of the best aspects of most modern training is the emphasis on the idea that hematology/oncology provides the best care when it is a “team sport.”
“Practicing in isolation is not the right way to practice,” Dr. Hicks said. “That is the training environment I was part of and is something I have introduced in my practice. I have a network of people I turn to for opinions when I have a difficult case, and I also practice with a multidisciplinary tumor board that provides collective intelligence.”
Catherine Bollard, MD, director of the Center for Cancer and Immunology Research at Children’s National Research Institute and professor of pediatrics and immunology at George Washington University in Washington, DC, echoed the importance of shared experience as a tool in the clinician’s toolbox.
“In the field of bone marrow transplantation and especially in pediatric hematology, the likelihood that you are dealing with making decisions not based on RCT data is way more prevalent,” Dr. Bollard said. “In the transplant and/or pediatric community we often come together with consortia or cooperative groups to come to a community consensus as a whole.”
Dr. Bollard frequently turns to colleagues for shared experience or participates in community discussions by email when faced with challenging situations or situations that lack consensus.
One of the other tools that clinicians have for navigating data-free zones is guidelines or recommendations.
Evidence Versus Expert Guidelines
In modern medicine it is rare to find a truly data-free zone. Instead, there are areas with high-quality evidence to guide decisions and areas with lower-quality evidence, Dr. Pai said. In areas with lower-quality evidence, observational data and expert evidence may be used to guide treatment and issue recommendations.
Many of the organizations that release practice guidelines draw a distinction between evidence-based guidelines and consensus-based guidelines. Understanding the differences between these can be very important to the interpretation of the recommendations, said Dr. Seifter, who is also editor-in-chief of the National Cancer Institute’s PDQ® Adult Treatment Editorial Board.
“At PDQ – Physician’s Data Query – we have over 100 statements on different malignancies where we review data and give recommendations,” Dr. Seifter said. “We are very rigorous in terms of what we analyze. We will say, ‘This is the evidence we have. This is what we know, and these are the things we don’t know.’ We don’t comment on what we don’t know. We stop in a data-free zone.”
In contrast, Dr. Seifter said other organizations that put out recommendations, such as the National Comprehensive Cancer Network (NCCN), may include consensus-driven management in their recommendations.1
“PDQ will not endorse particular strategies in a data-free zone, except to say something like, ‘Many doctors do this, but there are no data to support it,’” Dr. Seifter said.
An example of these differences involves the treatment of a rare disorder called Castleman’s disease.
“For many years, the only good therapy we had for this disease was rituximab,” Dr. Seifter said. As research evolved, it was discovered that the reason the lymph nodes were affected in this disease is because of a high level of interleukin-6 (IL-6).
“Scientists developed a drug called siltuximab that blocks IL-6 and causes Castleman’s to go away,” Dr. Seifter explained. “Siltuximab has FDA [U.S. Food and Drug Administration] approval for Castleman’s disease, as long as it is not caused by infection.”
However, clinicians have been using rituximab for Castleman’s since 1998 with good results, and rituximab, which is now available in a generic form, is much cheaper but is not FDA approved.
“Which one should clinicians use? The one that is cost-effective and seems to work but isn’t approved, or siltuximab, which is approved but may cost more?” Dr. Seifter said. “We can’t tell unless we run a clinical trial, and there is no one in this situation that would want to run that trial and come out on the wrong end of the results.”
Dr. Seifter said it is a perfect example of knowing that both drugs work through experience and phase I and II trials, but there will never be a head-to-head RCT. In this case, PDQ provides clinicians data without advice, and NCCN gives options based on data and expert opinion.
Who Is the Expert?
The question of just who the experts are is a difficult one.
“In general, I would say people can be considered experts when they have many years of experience in a given area or are recognized by others for having made multiple relevant research contributions,” Dr. Steensma said. In the case of the NCCN guidelines, he noted, every member institution can have somebody on each of the guideline committees, even if a given institution doesn’t have much strength in a specific disease area, meaning there may be cases where one institution’s representative on a panel may not have a lot of experience or publications and may have less of a “voice.”
“For a long time the expert was the most famous person or the person who spoke the loudest,” Dr. Pai said. “We are increasingly challenging this narrow definition.”
Practice guidelines are also produced by specialty organizations like ASH, whose guidelines “are developed by leading clinical, methodological, and patient experts through a rigorous process to review evidence and write actionable recommendations.”2 ASH has been working to expand its idea of what an expert is, according to Dr. Pai, who is the head of ASH’s Guideline Oversight Subcommittee.
“The idea of expertise comes back to the concept of unique knowledge,” Dr. Pai said. “An expert can be a health professional with lots of clinical experience or expertise. An expert can be a researcher or lab physician who understands pathophysiology, or a patient or patient representative who has lived experience. All those people can be experts, but their expertise has a different focus.”
ASH is also making a concerted effort to widen the net to include more people under the umbrella of “expertise,” such as a mix of junior and senior faculty, diversity in race and ethnicity, and geographic diversity, including the international community.
According to Dr. Pai, ASH is also pushing back on the idea of “expert opinion” by asking people to back up their statements with objective data.
“We tell guideline panelists, ‘We want to hear about your evidence, not just your opinion,’” Dr. Pai said. “We ask panelists, if possible, to provide us with data about why they think a drug works by looking at their practice or telling us about their patients. We want ‘experts’ to summarize their viewpoint in an objective and transparent way.”
This is important because when clinicians are dealing with areas that are data-free or have low-quality evidence, case reports and expert evidence must be taken into account or there will truly be completely data-free zones.
Consensus Considerations
Clinicians must take many things into consideration when reviewing consensus-based guidelines, not the least of which is the level of evidence assigned to a recommendation.
According to Dr. Pai, all ASH recommendations include a couple of important features. First is the direction itself and second is the certainty of evidence.
“The direction is obvious: ‘go left’ or ‘go right,’” Dr. Pai said. “The certainty of evidence tells us how sure we are about the direction.”
Dr. Pai used the analogy of a traffic symbol. A green light means cross: the recommendation is a strong recommendation. A yellow or amber light means cross but be thoughtful or cautious: there is a decision to make and other routes may need to be considered.
“When there is low or very low quality of evidence, that doesn’t mean we should do nothing or we should panic. And it doesn’t mean that the recommendation is useless,” Dr. Pai said. “It just means that we should proceed with caution, since this recommendation may not be right for every patient or for every circumstance.”
There may even be situations when not everyone agrees with the experts, and it is important that these opinions are heard, too, according to Dr. Bollard, who is also editor-in-chief of Blood Advances. For example, in 2021 ASH released a guideline on the use of anticoagulation in patients with COVID-19 being discharged from the hospital. Dr. Bollard said one of the people invited to review the article disagreed with some of the guidance detailed and was invited to write a commentary outlining these differences in opinion.3
“Guidelines are not written in stone,” Dr. Bollard said. “When there are opinions, people can disagree.”
As a different type of example, Dr. Steensma wrote a commentary in 2015 commending an international consortium proposal of uniform response criteria for myelodysplastic/myeloproliferative neoplasms but questioned how panel members were chosen “beyond the coalition of the willing” and expressed the hope that the omission of several active investigators in the field wouldn’t “impede the use and adoption” of what he considered to be reasonable criteria.4
Clinicians are also encouraged to remember that, although financial conflicts of interest or other forms of bias are taken into consideration when forming guidelines panels and communicated in a very transparent way, not all forms of bias can be so straightforward.
“Intellectual bias, for example, is a much tougher one,” Dr. Hicks said. “When someone’s opinion about a topic has become so entrenched that they have difficulty hearing new information, that is harder to address.”
Dr. Steensma said some people might advocate for a certain piece of research to be included in evidence because they or one of their mentees worked on it. Another potential motivation for inclusion in guidelines has to do with reimbursement, he added.
“Sometimes clinicians advocate for inclusion of regimens in guidelines such as the NCCN’s because they want them to be paid for,” Dr. Steensma said. “If a therapy is included in NCCN guidelines as an option with a certain level of evidence, it needs to be covered by Medicare because NCCN guidelines have official compendium status.”
These types of potential conflicts are getting a lot more scrutiny today than they used to, he added.
Building Trust
Consideration of and transparency related to potential conflicts and biases is an important part of establishing recommendations that can be trusted by the profession and the public.
Recent data from Pew Research Center showed that public confidence in medical science and scientists are below the rates seen in January 2019 – after a temporary increase following the start of the COVID-19 pandemic. Although many Americans still have a fair amount of confidence in medical scientists, about one in five has “not too much/no confidence at all.”5
“Destruction of trust in science and institutions is a big thing to deal with after the pandemic,” Dr. Pai said.
Building this trust is relevant to guideline development and data-free zones because medical professionals must get better at communicating the fact that science is fluid and rapidly changing.
“It is important to understand that science is always evolving,” Dr. Hicks said. “The job of anyone involved in a guideline is to ensure that we have the mechanisms in place to share the best science and communicate that we expect it to change and evolve.”
The whole field needs to work on how to communicate the fact that a change in a recommendation or practice is not a failure of medicine, Dr. Hicks said.
Clinicians and patients must also remember that guidelines, especially consensus-driven guidelines, are not a cookbook for medicine.
“Guidelines aren’t one-size-fits-all,” Dr. Pai said. “We should never just follow them blindly. Every patient is unique, and every setting of care is unique. Clinical judgment and patient preferences will always be incredibly important to decision making.”
References
- National Comprehensive Cancer Network. About Clinical Practice Guidelines. Accessed April 27, 2022. https://www.nccn.org/guidelines/guidelines-process/about-nccn-clinical-practice-guidelines.
- American Society of Hematology. ASH Clinical Practice Guidelines. Accessed April 27, 2022. https://www.hematology.org/education/clinicians/guidelines-and-quality-care/clinical-practice-guidelines.
- Spyropoulos AC, Lopes RD. Commissioned commentary on the 2021 American Society of Hematology Guidelines on use of anticoagulation in patients with COVID-19 being discharged from the hospital [published online ahead of print, 2022 Mar 4]. Blood Adv. doi: 10.1182/bloodadvances.2021006871.
- Steensma DP. Speaking a common language in MDS/MPNs. Blood. 2015;125(12):1849-1851.
- Kennedy B, Tyson A, Funk C. Americans’ trust in scientists, other groups declines. Pew Research Center. February 15, 2022. Accessed May 4, 2022. https://www.pewresearch.org/science/2022/02/15/americans-trust-in-scientists-other-groups-declines.