How many stars for physician ratings?
Online physician ratings are not going away any time soon, despite very real and very negative consequences.
ACP Member Monique Tello, MD, didn't used to spend much time thinking about her small sample of positive online reviews. Then, while she was at a conference, a speaker suggested that the audience members search for themselves on commercial physician-rating websites.
Dr. Tello's reviews had proliferated, but not in a good way. “I looked in October 2018 and saw that there were literally hundreds of horrible, one-star [ratings], people saying things like, ‘has her own agenda and doesn't listen to patients,’ ‘causes harm,’ and such,” she said. “And I was horrified.”
As Dr. Tello tried to wrap her head around what was going on, she noticed that the negative reviews began appearing in August 2018. That same month, she had blogged in support of a pediatrician friend who'd been targeted online by antivaccine activists for posting about National Immunization Awareness Month on social media. After that, Dr. Tello became the new target—she just didn't know it.
Realizing the reviews were likely fraudulent, Dr. Tello tried to contact staff at various rating sites but initially got no response. Once her hospital connected her with a lawyer, and after weeks of phone calls and waiting, most of the sites agreed to take the phony reviews down. “They were like, ‘Yeah, you're right. These are all looking very suspicious,’” said Dr. Tello, a primary care internist at Massachusetts General Hospital in Boston.
The damage, however, was done. Not only did some of her existing patients notice, but when a colleague recommended her to a patient for a family member, the patient balked. She had seen the reviews, she said, and Dr. Tello looked like a one-star doctor.
“That was harsh, and I felt really bad that people would take this type of thing seriously,” she said. “And other doctors have gone through very similar things.”
As Dr. Tello's story shows, online physician ratings can have real consequences, regardless of their validity. Commercial websites are open to the public, may not require real names, and may use self-verification rather than independent verification to determine whether users have been a patient of the doctor they are reviewing.
Still, it doesn't look like online reviews are going away any time soon. In fact, more and more health systems have taken to posting ratings themselves, typically publishing ratings of their physicians based on surveys sent to patients after outpatient visits and hospitalizations.
Some physicians, including Dr. Tello, believe that posting reviews from actual patients may counteract the many limitations of commercial rating sites. But there are still potential drawbacks.
A plethora of pitfalls
Patients have been using online physician rating websites for years. A 2012 survey of some 2,100 patients, published in February 2014 by JAMA, found that 59% reported physician rating sites to be somewhat or very important. Of those who searched for online physician ratings in the past year, 35% reported selecting a physician based on good ratings, and 37% had avoided a physician with bad ratings.
While many patients find public physician ratings useful, some doctors are fighting against them. Doctors have sued patients for defamation, and the nonprofit group Physicians Working Together created a petition to remove online doctor ratings, which has garnered more than 40,000 signatures.
In response to the overall lack of reviews and generally poor quality of narratives on commercial rating websites, some heath systems have begun to publicly post individual physicians' Consumer Assessment of Healthcare Providers & Systems Clinician & Group Survey (CG-CAHPS) results and narrative responses, said Tara Lagu, MD, FACP, who has focused on physician reviews in her research. Health systems obtain these from a random sample of patients who were seen by the physician and received mailed or phone surveys. “These differences address the concerns about selection bias and nonverified patients posting reviews,” she said.
Health systems then compile the CG-CAHPS survey data into an individual “star” rating given to each clinician, which is posted on the health system's “find a doctor” website, said Dr. Lagu. Many systems also post narratives that were submitted with quantitative survey responses, and the result is a large number of star ratings and narratives from verified patients, she said.
However, even online ratings posted by health systems don't quite have broad physician support.
For example, patients who search online for Lisa A. Ravindra, MD, FACP, can see that she's a highly rated primary care physician with 4.8 out of 5 stars on her Rush University webpage. Public comments from patients call her an “excellent doctor” who “listens intently” and is “very friendly and knowledgeable.”
Despite the glowing feedback, Dr. Ravindra, an assistant professor of internal medicine at Rush University Medical Center in Chicago, said she and many of her colleagues find the ratings to be arbitrary and biased. “In fact, there is significant variability in my own scores from quarter to quarter,” she said. “I find it unlikely that my personality, communication skills, expertise, and bedside manner vary in this way.”
In a 2015 survey of about 830 physicians at a large accountable care organization, only 23% and 21% supported making numerical ratings and narrative comments, respectively, publicly available online. In addition, 78% reported that doing so would have a somewhat or very negative effect on their job stress, according to results published in the June 2017 Journal of General Internal Medicine.
However, ratings posted by health systems can drown out those posted on commercial sites with a larger data set and random sample, said Dr. Lagu. Although the limitations of patients' reviews and narratives are well known, both positive and negative responses to survey questions may be able to help health systems improve care, she said.
“I think, at the very least, public reporting of experience scores and comments can give clinicians some of the why behind the patient experience scores they receive, which offers an opportunity to identify specific improvement targets,” said Dr. Lagu, an associate professor of medicine at the University of Massachusetts Medical School–Baystate.
But no matter whether health systems or commercial websites are posting them, physician ratings are vulnerable to bias, particularly with regard to race and gender. These variables make interpreting these data more difficult, experts said.
In a February 2019 perspective in the New England Journal of Medicine, Kenneth G. Poole Jr., MD, FACP, the medical director of patient experience at the Mayo Clinic in Arizona, described what he and his colleagues found after analyzing patient experience data for three black community-based primary care clinicians (including himself). Over three years, when seeing black patients, the clinicians had an overall top-box score of 93.8%. When seeing white patients, they had a top-box score of 78.1%.
“How can I be a leader in this space, when I cannot consistently get top-of-class scores? Feelings of deflation and lack of appreciation arise, because it seems that no matter what techniques I use, how much empathy I exhibit, and how much extra time I spend with patients, I cannot substantially improve my patient-experience scores,” Dr. Poole wrote.
Women in medicine face similar challenges. Their online reviews were consistently less favorable than those of their male counterparts, even after adjustment for specialty, according to a study published in November 2019 by Proceedings of Machine Learning Research. Patients also used a different set of words to describe female versus male clinicians: “Sweet,” “considerate,” and “caring” were significantly more likely to be in reviews of women, whereas “professional,” “knowledgeable,” and “thorough” were more likely to be found in reviews of men.
Reviews of female clinicians seem to relate more to interpersonal manner, whereas reviews of male clinicians seem to relate more to technical competence, said Urmimala Sarkar, MD, MPH, a coauthor of the study and professor of medicine at the University of California, San Francisco. The results echo a June 2018 perspective in the Journal of General Internal Medicine, which hypothesized that patients have higher expectations of female physicians in terms of empathy and time, she noted.
“These findings do raise a concern that the increasing focus on physician ratings will exacerbate challenges for women seeking to advance to leadership in medicine,” Dr. Sarkar said. “I hope our study demonstrates that physician ratings should be interpreted in the light of prevalent implicit and overt gender bias.”
Future opportunities
Many patients seem to be on board with health systems posting physician reviews. Of about 500 patients who responded to the Journal of General Internal Medicine survey, slightly more than 50% supported posting both numerical ratings and narrative comments. About 53% of physicians and 39% of patients reported visiting a health care rating website at least once.
With a ratings culture in the U.S. that reviews everything from restaurants to rideshares, health systems are trying to provide a feature that patients are looking for, said Dr. Ravindra. “Unfortunately, patients are misguided in relying on online reviews, but that does not prevent hospitals and websites from responding to what the market demands,” she said.
Dr. Lagu agreed that there is a marketing component to health systems getting involved in publicizing physician ratings. But it's also about transparency, she said, adding that she serves on a committee that reviews every patient comment that is questionable at her institution. “We have very clear criteria about how to approach it. We include all comments, positive or negative, unless there is something that is offensive,” said Dr. Lagu.
The committee has declined to publish comments that contain racial slurs, as well as comments that mention an aspect of the clinician that is inappropriate because it's not about the quality of care, such as sexual orientation or accent, she said. “But if there are negative comments that say something about the quality of care, we really try to include them,” said Dr. Lagu, “because they can help physicians learn and help patients choose physicians who share their values.”
Compared to reviews on commercial websites, reviews posted by health systems are a more trustworthy way to be transparent about the kind of care a physician provides, said Dr. Tello. While social media experts advised her that the best way to combat negative reviews is to post your own content and generate lots of positive reviews, she was unwilling, as some had suggested, to enlist friends and family to write nice comments about her online.
“I thought, ‘I want to do this the right way,’ so I didn't do that,” said Dr. Tello, who now has a modest selection of generally positive online reviews to her name. “But I would be thrilled if Mass General wanted to post its own internal physician reviews based on patient surveys.”
While ratings on health systems' websites might seem more useful because they are verified, they also have a lack of variation, potentially making it difficult to differentiate between clinicians, according to a study published in March 2019 by the Journal of Medical Internet Research. Of 42 health systems that published star ratings and 33 systems that published narratives, 64% stated that they excluded narratives that were deemed inappropriate or offensive. Of 140 clinicians, the median rating was 4.8 out of 5 stars, and no clinician had a score less than 4.2 stars.
Thus, it's no surprise that negative reviews on commercial websites do not reflect negative ratings on industry-vetted patient satisfaction surveys, as one study found. Researchers compared 98 physicians with negative online reviews with 82 physicians in similar departments/divisions without negative reviews and found no differences between groups in average patient satisfaction survey scores, according to results published in the April 2018 Mayo Clinic Proceedings.
For health systems, ratings can be useful in cases where multiple patients have given a physician consistently low scores, said Dr. Ravindra. “This could be a red flag that allows the health system to intervene,” she said. “However, hospital quality committees and state medical boards have traditionally done this even before.”
Sometimes, physician reviews aren't about the physicians themselves. Whether they discuss the availability of parking or the friendliness of front desk staff, even verified reviews may describe more about the patient experience than the clinical encounter. With this in mind, health systems should focus on narrative data to target improvement efforts, said Dr. Lagu.
“We just need as health systems to take these narrative data (because this stuff isn't in the surveys) and figure out what's really making patients' experiences in any given health system difficult. We hypothesize that fixing those things could have a positive impact on the rest of the ratings,” she said.
In a paper published in March 2019 by the Milbank Quarterly, Dr. Lagu and colleagues used a validated elicitation protocol to systematically collect narratives from 348 patients. They found that the narrative content was distributed evenly among three categories: aspects of care currently measured by patient experience surveys, those related to measured domains but not captured by survey questions, and those that are omitted from surveys entirely. Three-quarters of the narratives had some actionable content.
This is yet another example of ways that patient stories identify opportunities for improvement, Dr. Lagu said. One challenge is that analyzing large quantities of narrative data is very difficult, and health systems would need to invest time and money in areas like artificial intelligence, but she believes that some innovative health systems will take the lead.
“My vision for this is that we're eventually going to use this protocol to get patients' stories,” she said. “And those stories will help drive the way we try to improve care at the outpatient and then, eventually, at the inpatient level too.”
