Underrepresented minority (URM) applicants to the American Society of Hematology’s (ASH) Clinical Research Training Institute (CRTI) program received significantly lower reviewer-assigned scores than applicants who were not from URM groups, according to a study published in Blood Advances. The one-year CRTI program began in 2003 and includes didactic learning, mentorship, protocol development, and workshops.
In the study, Sara K. Vesely, PhD, of the University of Oklahoma Health Sciences Center, and colleagues compared scores for applications submitted between 2003 to 2019, focusing on the gender and URM status of applicants and reviewers. The study defined URM status as Blacks or African Americans, Hispanics or Latinos, American Indians or Alaska Natives, and Native Hawaiians and other Pacific Islanders ̶ groups that have historically been underrepresented in medical research.
According to the authors, applicants to the CRTI program between 2003 and 2019 were mostly U.S.- or Canada-based senior fellows or junior faculty within the first three years of their initial faculty appointment who had planned a career in patient-oriented hematology research. The application included information on applicants’ demographic characteristics, career development plan, research proposal, the applicants’ home mentor’s biosketch, the home mentor’s letter of support, and an institutional commitment letter from a division chief or similar official.
ASH members who were clinical researchers reviewed the applications. They submitted an overall score and critique of each application. Scoring of the research proposal was based on approach, feasibility, innovation, and significance. A total of 20 applicants were chosen to participate in the CRTI program each year, but three additional applicants could also be selected to promote diversity.
The scoring system changed over the years, so the researchers transformed the application scores to a 0 to 100 scale, with 100 representing the strongest score. In their assessment, the researchers evaluated whether an interaction exists between the gender and URM statuses and time related to application scores.
In total, the study included 713 applicants with 2,106 reviews. Several years were excluded because the data were not available; researchers divided the applications into three periods that had the same number of years. The number of applicants rose between the time periods of 2003 to 2006 (n=168), 2008 or 2011 to 2013 (n=204), and 2015 to 2019 (n=341). Over time, there were also increases in the diversity of applicants by URM status, race, and ethnicity. In addition, there was significantly more diversity among reviewers by gender, URM status, race, and ethnicity over time.
No significant difference was observed in scores stratified by applicant gender. Compared with non-URM groups, applicants who belonged to a URM group had significantly worse application scores (mean score = 71.4 vs. 67.9; p=0.0355).
The investigators found significant interactions between reviewer gender and time period (p=0.0030) and reviewer URM status and time period (p=0.0424). For the periods of 2003 to 2006 and 2008 or 2011 to 2013, male reviewers’ scores were significantly worse than female reviewers’ scores (p=0.0022 and p=0.0001, respectively), but this difference did not persist in the 2015 to 2019 period (p=0.5548).
Reviewers from URM groups did not give significantly different scores during 2003 to 2019. For the period of 2008 and 2011 to 2013, URM reviewers scored URM applicants higher than non-URM applicants, while non-URM reviewers scored non-URM applicants higher than URM applicants.
The researchers noted that heterogenous scoring mechanisms were applied over time and represent a limitation of the analysis. Additionally, the investigators stated that there was a lack of data on socioeconomic status, which may have strengthened the study. Further, the researchers were unable to review the track records of the mentors, the scientific momentum of the applicants, and the institutional environment, all of which contribute to the application scores.
Despite these limitations, the researchers concluded that there is a need to develop and initiate “efforts to mitigate the impact of applicant URM status on application scores” for the CRTI program.
Any conflicts of interest declared by the authors can be found in the original article.
Reference
Vesely SK, King AA, Vettese E, et al. Influence of participant and reviewer characteristics in application scores for a hematology research training program [published online ahead of print, 2023 Mar 20]. Blood Adv. doi: 10.1182/bloodadvances.2023009792.
