Hospitals, doctors and the health care system as a whole have become ever more focused on measuring the quality of the care patients receive. And with good reason: as the system leans ever more towards tying reimbursements to quality, everyone recognizes that you can’t improve quality if you’re not measuring it.
Of the many ways one can look at quality in an inpatient setting, patient experience has earned a lot of attention. Hospitals, payors, survey vendors and government agencies are spending millions to develop, deploy and analyze tools like the adult and child Hospital Consumer Assessment of Healthcare Providers and System (HCAHPS) surveys, which give voice to patients and their concerns about the care they receive.
Are there other ways to hear what patients are saying? Jared Hawkins, MMSc, PhD, of Boston Children’s Hospital’s Computational Health Informatics Program (CHIP), Boston Children’s chief innovation officer, John Brownstein, PhD, and their colleagues wanted to see whether they could harness the power of social media—specifically, Twitter—to supplement survey-based methods. Their data, published in the journal BMJ Quality & Safety, give encouraging hints, but it’s too early to retire those patient surveys just yet.
A social view of patient experience
Could tweets be a reasonable form of quality measurement? When you consider it, we often turn to Twitter, Yelp, Angie’s List and other networks to provide instant feedback on and offer our thoughts about pretty much any company, contractor or store we do business with. Why would healthcare be any different?
That “instant” aspect is part of what captured the team’s attention.
“The main problems with measuring patient experience by survey are the small numbers of people who respond to surveys and the lag time,” says Hawkins, who works under Brownstein in his Computational Epidemiology Group. “It can take up to two years before survey data are released to the public.
“Given that social media data are close to real time,” he continued, “we wanted to see if we could capture this discussion and if the content is useful.”
Sifting the tweets from the chaff
Hawkins, Brownstein and their collaborators collected more than 400,000 public tweets directed at the Twitter handles of nearly 2,400 hospitals in the U.S. between 2012 and 2013. Using machine learning, natural language processing and manual curation, they tagged 34,735 patient experience tweets directed at 1,726 hospital-owned Twitter accounts, determined the sentiment of those tweets (positive, neutral, negative) and binned the tweets into topical categories (e.g., Time, Communication, Pain).
“We were able to capture what people were happy or mad about, in an unsolicited way,” Hawkins explains. “No one else is looking at patient experience this way because surveys ask very targeted questions. Unsurprisingly, you get back very targeted, narrow answers.”
The holy grail in this work would be the ability to correlate tweets and the sentiments expressed in tweets to outcomes metrics that relate to care quality. Hints of that ability are there. When Hawkins and Brownstein compared their data with outcomes data from Medicare’s Hospital Compare website, they found a weak negative correlation between tweet sentiment and hospitals’ 30-day readmission rates. “Hospitals that people thought highly of had lower readmission rates,” Hawkins explains.
Surprisingly, though, they did not see a relationship between tweet sentiment and HCAHPS experience data.
“This is a brand new way of using Twitter data,” Hawkins notes. “It may be that we have to be cautious about using tweet sentiment to understand quality.”
Just the beginning
Probably the study’s largest weakness, Hawkins notes, is the number of tweets out there for analysis.
“There’s not a tremendous volume,” he says, noting that of the 2,400 hospital Twitter accounts they looked at, only about 300 received more than 50 inbound tweets in a year. “If you look at a lot of hospitals, there’s not a lot going on, maybe three or five tweets that are about patient experience.
“But remember, this was 2012,” he adds. “If we were to look today and beyond, we would expect the volume to be much higher. And we’re already looking to see whether we can integrate data from other social networks like Yelp and Reddit.
At the moment, the low volume of data could make it hard to identify trends over time, notes Sara Toomey, MD, MPhil, MPH, MSc, managing director of the Center of Excellence for Pediatric Quality Measurement at Boston Children’s, who was not involved in Hawkins and Brownstein’s study. The Twitter users who produced the tweet pool analyzed may not be entirely representative of each hospital’s patients.
But the data are suggestive and highlight Twitter’s possible utility as a way to supplement HCAHPS and other surveys.
“We’re always looking for ways to listen to the voices of patients and families and use what they say to inform the quality of the care we provide,” explains Toomey, one of the co-leaders in the effort to develop the child version of HCAHPS. “There’s been a steady decline in survey response rates, so people involved in quality measurement are actively seeking ways of augmenting the standard approach with other collection methods.”
“It’s exciting to think about how a hospital might feed not only comments and scores from surveys but also comments from other sources like tweets into our efforts to improve quality,” she adds.
Hawkins agrees, adding that in his view the study provides proof of principle for incorporating Twitter into the box of tools hospitals use to measure their interactions with patients. “The discussion about patient experience is going on out there on social media,” he says. “We need to be looking at it, and in a systematic way.”