Media Effects Research Lab - Research Archive
Deliberative quality of comments on social news websites: A content analysis
Student Researcher(s)
Ye Tian (Ph.D Candidate);
Yuqi Zhao (Masters Candidate);
Faculty Supervisor
INTRODUCTION
This paper aims to employ quantitative metrics of public deliberation theory and evaluate the deliberative merit of discussions on social news websites. A content analysis was conducted to evaluate the deliberative quality from seven aspects, and discussions are sampled from two topics and from two types of websites with different specialty. A 2 by 2 factorial ANOVA shows that the overall deliberation quality differs significantly between the two topics and between the two types of websites. Regression analysis suggests a strong correlation between the degree of participation and other metrics of deliberative quality.
RESEARCH QUESTIONS
RQ1:For discussion threads on social news websites, what is the relationship between topic of story, specialty of website and deliberative quality?
RQ2:For discussion threads on social news websites, controlling for topic of story and specialty of website, what is the relationship (1) between participation and the usage of respectful language, (2) between participation and the quality of argumentation, and (3) between participation and degree of interaction?
METHOD
In this study, we randomly selected discussion threads from both technology and public affairs topics across both general social news websites and specialized social news websites. Both authors were involved in the coding process (high inter-coder reliability). For each thread of comments, we calculated quantitative data to evaluate deliberative quality from the following aspects: claim-making, sourcing, interaction, respect and participation. The first set of analyses uses a 2× 2 factorial design (topic of story: technology and public affairs × specialty of website: general and specialized) to examine their effects on the deliberative quality of comments. This set of analyses employ factorial ANOVA to evaluate all the dependent variables on topic of news story, specialty of websites, and the interaction between the two. The second set of analyses examines the relation between degree of participation and other metrics of deliberative quality. With the two categorical variables (topic of news story and specialty of website) as control variables, each metric is regressed on the participation variable (average number of posts per user in a thread).
RESULTS
The complete results of the first set of analyses are shown in Table 2. The degree of participation on general websites is significantly different from that on specialized websites. Significant difference exists between the average length of message on general websites and that on special websites, and between technology topic and public affairs topic. In terms of the percentage of claim-making posts and posts providing sources to support claims, specialized websites show a consistently higher quality than general websites. The topic of public affairs has a significantly higher percentage of claim-making posts than technology. For the percentage of posts showing agreement, significant difference exists between the topic of technology and public affairs. However, for all of the three metrics (percentage of posts showing agreement, disagreement, or using impolite languages), the interactions between them are significant. Specifically, for the topic of technology, specialized websites have significantly higher percentage of posts showing agreement and disagreement than general websites, and also have significantly higher percentage of posts that use impolite language. For the topic of public affairs, however, all of the three metrics show an opposite pattern between specialized websites and general websites. Table 3 shows the detailed results of the second set of analyses. Among the four metrics, percentage of claim-making posts is significantly negatively related to the average number of posts made by a user, while percentage of posts showing agreement, disagreement and using impolite language are significantly positively related to the average number of posts per user.
CONCLUSION
Overall, significant differences exist between general and specialized website users on specialized websites exhibit a more deliberative behavior. The reason varies on the case-to-case basis. For the topic of technology, due to the average high professional profile of Hacker News users, lay people tend to be reluctant to post links or comments. Also, based on the data analysis in Moore’s study (Moore, 2012), we would argue that their prudence of voicing their opinions is the reason for their higher-quality conversation: Hacker News users do not comment a lot, but when they do, they give insightful thoughts. For the topic of public affairs, the strict moderating mechanism on specialized websites is the main cause for the high quality of comments. However, the rigorous moderating system also lowers the degree of interaction in discussions on the specialized public affairs websites. Topic of news article is also a predictor of the deliberative quality of discussion in some way. Notably, discussions on the topic of public affairs have a higher quality of claim-making and sourcing than those in technology. We found that many posts under the technology topic were questions and answers, which provide details and explanations of the technology involved in the news story rather than giving the commenters’ judgments on them; therefore these posts were not counted as claim-making posts in our analysis. Controlling for topic and website specialty, the second set of analyses showed a significant correlation between the degree of participation and other metrics. The results confirm our intuitive understanding that when less people are involved and each user makes more posts, the degree of interaction is significantly improved; simultaneously, the impolite language is also more frequently used. Contrary to the widely accepted perception that more interaction (turn-taking) facilitates better communication, our results indicate that interaction is negatively related to the percentage of claim-making posts.
For more details regarding the study contact
Dr. S. Shyam Sundar by e-mail at sss12@psu.edu or by telephone at (814) 865-2173