Update 90: Assess the Quality of COVID-19 Info with a Validated Tool
本文由‘中国推动’学者、江南大学药学院石漱新同学编辑整理。
MedCram.com. Welcome to another MedCram COVID-19 update. And officially, we’re over 10 million confirmed infected cases worldwide and over half a million global deaths. In the United States, daily new cases are increasing. While daily deaths continue to flatten out and decrease so though this may be because of a number of factors including the fact that it takes about two to three weeks to catch up in terms of the natural history of the disease.
And for this talk, I wanted to do something a little bit different. We here at MedCram have gone over the therapeutics, the diagnostic aspects, the pathophysiology of coronavirus and the day-to-day discoveries about COVID-19. But what I wanted to talk about today is something a little bit different and it’s something I think you’ll be interested in. You’re using media on the internet to get information about COVID-19. And I want to show you a study that looked at that in terms
of validity and accuracy. So, the question is how do we best tell if something that we’re watching or information that we’re being given is giving us accurate information in a way that we can process and use. And believe it or not, there’s actually a tool that you can use to make that type of judgment.
And that tool is something called DISCERN. Now, DISCERN was developed back in the late 90s, around 1996/1997 out of Oxford University, UK. And it’s a validated tool that’s been in use for some time. DISCERN is a brief questionnaire which provides users with a valid and reliable way of assessing the quality of written information on a treatment choices for a health problem. DISCERN can also be used by authors and publishers of information on treatment choices as a guide
to the standard which users are entitled to expect. So, if we go to the background to DISCERN, you’ll find out why this is an important thing to use. It can, for instance, look at the clinical effectiveness and evidence-based treatment. “Health professionals are increasingly provided treatment based on evidence of clinical effectiveness. The evidence consists of rigorous and up-to-date scientific research that has shown the treatment to be largely beneficial.” And this has to do with as they talked about here “consumer choice”.
This is not too unlike what you’re doing right now looking at YouTube to get information about a medical problem that affects us all, in this case, it’s COVID-19. And see if this sounds familiar. “The practice of shared decision-making is enabling health professionals and consumers to work together to find the best treatment choice for an individual consumer. A vital part of increased consumer involvement in decisions about treatment is access to good quality information.” And that’s what you’re all looking for, “
quality information”. And they go on to talk about good quality information being about treatment choices will be accurate based on the most and best up-to-date scientific evidence and it will help you consider all aspects of treatment choice, including the outcomes of a treatment choice and any areas of uncertainty.
Now, DISCERN was developed over a long period of time. And let’s talk a little bit more about what it is and who can use it. So, “there’s currently a lot of written consumer health information on treatment choices
available from a variety of sources”, and of course, this was all written down before COVID-19 came in. But you can see that this is exactly the case for COVID-19. And here’s the keywords, “including the internet”. How many of you that are now watching this video have looked over the internet to try to find sources of good information about what is going on in COVID-19. And what’s happened is you’ve run into a lot of things that are politicized or that are one-sided or you don’t know if it’s true or not. It says here, “many of the publications available provide in
accurate or confusing advice, and it may be hard to know which information to use and which to discard. DISCERN is an instrument or a tool which has been designed to help users of consumer health information judge the quality of written information about treatment choices.” And of course, this is not just written information but also information that’s available on the internet, on video platforms.
So, who can use DISCERN? It’s suitable for anyone who uses a produces information about treatment choices, and its users
are diverse and include: an aide for individual consumers; a screening tool for health information providers; a checklist for authors and producers of written consumer health information and a training tool for health professionals to improve communication and shared decision-making skills. Now, this is an extensive process of development and evaluation. It was evaluated, it was published, it was peer reviewed and it is validated. They say here, “the rigorous process used to develop DISCERN has enabled us
to identify a general set of guidelines for the content of written information on treatment choices, which can be consistently understood and applied by a wide range of users. Consequently, DISCERN is the first standardized index of quality of consumer health information.” So, that’s really important to understand because what a group of scientists decided to do was to apply this DISCERN instrument to videos on YouTube about COVID-19.
And here’s the article titled “YouTube as a source of patient information for COVID-19: A content-quality and audience engagement analysis”. And what Szmuda et al from the Neurosurgery Department at the Medical University of Gdansk, Poland, decided to do was to apply this tool that was developed at Oxford to see what type of videos are being produced on YouTube in the analysis and to see which ones are helpful, and to break
down, perhaps, who the authors are, and see which one’s might be better than others. And more importantly, whether or not the information on YouTube is actually adequate.
But before we get into that, let’s look at the DISCERN tool itself. So, here is the DISCERN instrument. Of course, we’ll put a link to this in the description below. You can also download it as a PDF document. I will tell you that it is copyrighted by Oxford University, but here it is on their website at DISCERN
.org.uk. And you can see that basically it’s a survey, it’s a survey of 15 questions, the 16th being an overall survey. And here’s the first question, “are the aims clear?” No to yes, and it’s a scale of 1 to 5 points. So, 15 times 5 will be a total point of 75. And you can see that if you score one on all of these which would be very poor. You’ll have a minimum score they’re of 15. 2,“does it achieve its aims?”
3,”is it relevant?” 4,”is it clear what sources of information were used to compile the publication other than the author or the producer?” Number 5,”is it clear when the information used or reported in the publication was produced?” 6,”is it balanced and unbiased?” 7,”does it provide details of additional sources of support and information?” 8,”does it refer to areas of uncertainty? 9,”does it describe how each treatment works?” 10,”does it describe the benefits of each treatment?”
11,”does it describe the risks of each treatment?” 12,”does it describe what would happen if no treatment is used?” 13,”does it describe how the treatment choices affect overall quality of life? 14,”is it clear that there may be more than one possible treatment choice?” And then 15, “does it provide support for shared decision-making?” And then of course the 16th question is “based on the answers to all of the above questions, rate the overall quality of the publication as a source of information about treatment choices.”
So again, what Szmuda et al decided to do was to look around the internet and specifically he did this just after March 11th, 2020, and they went through and they searched YouTube by typing in some health terms, such as “2019 nCoV,” and “SARS CoV-2,” and “COVID-19 virus,” “coronavirus treatment,” “coronavirus explained,” “what is the coronavirus” and “coronavirus information”.
Of course, they made sure to do this in incognito mode so that it was an unbiased rendering. They did this on the day after the WHO had classified COVID-19 as a pandemic. And they graded it within two days. So, it was being graded based on the information that was at hand. Now in this case, only the first 30 videos were chosen since 90% of YouTube users do not look past that number. Furthermore, they classify the author of these videos as were they
a physician, a hospital and educational channel, a health organization or news shows. They also looked at a number of other things to see whether or not they have qualitative video elements, for instance, if they said that coronavirus was a zoonotic virus, common symptoms of the disease, which populations were at risk of severe illness, a discussion of causes, the viral incubation period, the methods of diagnosis if a vaccine is available, etc, etc.
They also looked like other statistics, for instance, the views duration and seconds, the number of comments, number of likes, number of dislikes, etc. etc. Now, to make sure that there was Independence and precision in this endeavor, all videos were evaluated independently by two final year medical students with more than 5 years of experience using the DISCERN instrument. They did this using this 16-part questionnaire. Like we just discussed to assess the quality of health information,
which has been used in several YouTube quality analysis studies and they give references to that.
They use the first 15 questions of the DISCERN instrument to summarize whether or not the video was very poor, 16 to 26, poor, fair, good and excellent, 63 to a maximum score of 75. And again, here you can see the different questions that were used. Of course, this last question is just basically a summarization of the previous
fifteen.
So, let’s see what the results were. Out of the total 210 videos, 137 videos met our inclusion criteria and were evaluated. Figure 1 illustrates, the qualitative video content of the videos. And you can see here in Figure 1 that the majority we’re talking about prevention. There was also a large amount that were talking about symptoms, spread of the virus, a doctor was speaking, SARS mention, zoonosis nature, all the way down to radio graphical findings at
the minority. In terms of the authors of the videos, figure 2 showed that most of the videos were uploaded by news channels at 48.2% and educational channels at 38.7%. The rest were uploaded by health organizations at 5.1%, hospitals at 5.1% and the minority were physicians at 2.9%.
And you can see that here in this pie chart. Okay. Now, let’s look at the video quality evaluation. This was actually quite disturbing, “the mean DISCERN score for the first 15 questions between the two raters was 31.33 plus or minus 9,92, indicating that the overall quality of the videos that they looked at was poor. Interestingly, when they compared the first rater and the second rater, they had a DISCERN score of 31.35
and 31.29 respectively. This shows that there was a lot of inter rater reliability. So, first take-home point is that the DISCERN instrument worked and what it found was that most of the videos on YouTube, at least on March 11 and 12 were very poor when it came to DISCERN instrument. In other words, it didn’t give you a lot of good information. Figure 3 illustrates that videos uploaded by physicians, health organizations and educational
channels had a higher average DISCERN score, normally above 30. Then the news shows and the hospitals which were below 30. However, videos uploaded by physicians had an overall fair quality, a score of 40.3. And as we said here, the intraclass correlation coefficient for the absolute agreement was 0.96 for DISCERN between the two raters. This is regarded as an excellent reliability.
And again, here are the ratings for the different uploaders: physicians, health organizations, education channels, news channels, hospital channels. And remember, this score could go as high as 75, but you can see here that many of them were in the very low category. Again, they mention here, “in our study, news channels comprise the majority of the videos uploaded on COVID-19. However, their quality was the worst (mean DISCERN score of
27.9).”
And here you can see how in general the questions went. Usually, question 1 was the best answered, followed by question 2, and from there, things went downhill very quickly.
So, what are the authors have to say about this? They said that we found that overall, the quality and reliability of YouTube videos on SARS-CoV-2 and COVID-19 is poor, which indicates that the information about this disease on YouTube does not provide a comprehensive review.
They felt that their analysis was valid because their analysis was conducted just a few days after COVID-19 was declared a world pandemic. They said as Figure 1 illustrates, more than half of the videos failed to provide rudimentary information, such as symptoms, spread of the infection and prevention. They say today the information on YouTube is neither regulated nor reviewed. Thus, anyone with an internet connection may upload health information videos, which may be assessed by millions worldwide. In the case of
COVID-19 pandemic, misinformation spread on this platform has the potential to cause panic and worse health outcomes for patients. They even go as far as to say that they recommend that the public avoid news channels if they wish to obtain holistic knowledge about SARS-Cov-2 virus and COVID-19. They say that their results show that the videos uploaded by news channels had a relatively low average DISCERN score of 27.9. Videos uploaded by physicians,
however, had the highest DISCERN scores with a mean of 40.3 points, which is fair quality and which is better than any other result by a specific category of uploader. Thus, watching videos uploaded by physicians is better than gaining a basic knowledge compared especially to news channels.
They even go on to say that in their analysis, the news channels had a poor DISCERN score. These videos not only failed to mention the treatment and management strategies, but often fail to provide almost any medical information
that we took into our analysis, although their video title claim to. We assert that the COVID-19 crisis is not only a viral crisis but also a health information crisis. Recent studies have shown that wide dissemination of misinformation concerning the outbreak has created panic among the public. Larson et al has dubbed the biggest risk of COVID-19 “viral misinformation.” Thus, not only is there an urgent need to scale a public health
measures but also want to address the pandemic of social media panic and misinformation.
Under limitations of the study, they say here that medical student raters may be considered unreliable raters compared to physicians. However, the creators of DISCERN state that their instrument was not dependent on specialist knowledge of a health condition or treatment. Moreover, the intraclass correlation coefficient between the two raters was excellent. Thus, we assert that the medical student raters who each had five
years of experience using the DISCERN instrument while in medical school, provided adequate results for this study. In terms of future directions, they say since SARS-CoV-2 and COVID-19 are such relevant topics in the emerging pandemic, we encourage researchers to conduct a follow-up analysis in the future. We also encourage video content creators to use audience engagement suggestions and the DISCERN instrument to assess their videos for educational content so that they may provide more
robust information about these diseases going forward.
Finally, let’s look at their conclusion. “Healthcare consumers obtaining health information on COVID-19 on YouTube are obtaining an incomplete comprehension. This may lead to misunderstandings concerning treatment, prevention and the general characteristics of the virus. We recommend that people verify the facts that they learn on YouTube with more reliable sources of information, like peer-reviewed research papers, healthcare professionals or online updates
by The WHO or the centers for disease prevention and control. We recommend physicians refer patients to the list of highest quality videos that we provided if a patient wishes to use YouTube as a source of their medical information. So, the authors of this study were so kind as to list the top five videos that they found in their study.
And here they are listed. So, getting a top scorer at 66 is Coronavirus Epidemic Update 34: US Cases Surge, Chloroquine & Zinc Treatment Combo, Italy Lockdown, uploaded by MedCram-Medical Lectures Explain Clearly. So, you can imagine we at MedCram.com were pleasantly surprised to see a peer-reviewed article using a validated instrument showed that our videos were of high quality.
We also noticed that we didn’t get a perfect score and that’s something that will work on. But I also wanted to say that we can’t take total credit. It’s because of you, our audience that we strive to put forth good quality material. And many of the ideas and coverage that we do is a result of your comments and we will link to all of these videos, here the top five in our description below. I think if we all look at that DISCERN instrument a little more carefully, we have
an opportunity to improve the medical content on YouTube.
Thanks for joining us.
Add comment