Patterns and Determinants of Civility in Online Discussions

Final Report to the National Institute for Civil Discourse

 

Kate Kenski, Kevin Coe, and Steve Rains

University of Arizona

October 13, 2012

 

 

Overview

 

Civility is a principle intimately tied to the American experiment—an ideal that speaks to the “the fundamental tone and practice of democracy” (Herbst, 2010, p. 3). Indeed, a commitment to civil discourse—the free and respectful exchange of different ideas—has been viewed as central to the effective practice of democracy from the ancient Athenian forums to the founding of America to our nation’s most recent debates about various political issues. This is not to say the commitment to civil discourse is always realized. More than 8 in 10 adult Americans view the “lack of civil or respectful discourse in our political system” as a “somewhat serious” or “very serious” problem (Public Religion Research Institute, 2010). American political debate has always had its share of incivility, and the current era is no different.      

What is different now, however, is that the 21st century’s vast, interactive media environment (e.g., blogs, video-sharing websites, social network websites) has created broader opportunities for public debate and engagement, be it civil or uncivil. One example is the discussion forums available on newspaper websites. Many national (e.g., The New York Times) and local papers (e.g., Arizona Daily Star) offer a digital space for the general news-reading public to post “comments” in response to a given article. These discussion forums offer a means for the lay public to engage in civil discourse about contemporary political issues as well as an opportunity for scholars to better understand the nature of civility in political discussion. Although some research has studied disagreement in online discussion groups by experimental means (e.g, Price, Nir, & Cappella, 2006; Stromer-Galley & Muhlberger, 2009), little work has examined how non-experimentally contrived online discussion spaces act as public forums for civil discourse about contemporary political issues.

The project we undertook with funding from the National Institute for Civil Discourse (NICD) was among the first to develop and validate a content analysis scheme that can track incivility in naturally occurring public discussions. Via a case study of three weeks’ worth of comments to articles appearing in the online version of the Arizona Daily Star, our study provides insight into the use of incivility in such forums and offers a foundation upon which future scholarship might build.

 

Development of the Coding Scheme

The NICD grant funded six University of Arizona undergraduate honors students to serve as coders during the Fall 2011 semester, with each student working an average of seven hours per week. Over the course of the first two months of the semester, the primary investigators developed and trained the student coders on a coding scheme that could reliably capture incivility in online discussions. Drawing on prior research that had tracked incivility or related concepts in other contexts (Jamieson, 1997, 2011; Stromer-Galley, 2007), we devised 7 categories of incivility. Table 1 presents these categories, with definitions, examples, and chance-corrected reliability scores.

 

Table 1: Operationalization of Incivility

Category

Definition and Examples

Krippendorff’s α

Name calling              

 

Attack on a person (e.g., idiot, you’re dumb, Obamamama)

.6674

Aspersion

Attack on an idea (e.g., that policy is asinine, what a stupid idea)

.6120

Lying              

Implying disingenuousness (e.g., liar, dishonest, not trustworthy)

.7330

Vulgarity

Use of vulgar terms (e.g., crap, hell, bitching)

.9065

Pejorative for speech  

 

Disparaging the manner in which someone communicates (e.g., bellyache, blather, crying)

.7388

Hyperbole

Massive overstatement (e.g., makes pulling teeth with pliers look easy, like superman on cocaine)

.6829

Noncooperation

Discussion of a situation in terms of a stalemate (e.g., immovable, polarizing, rigid, gridlocked)

.8600

 

Notably, two other categories (sarcasm, and a catchall category of “other incivility”) were coded as well. However, the coding team was unable to capture these reliably, so they were omitted from the analysis. This indicates the difficulty of fully capturing incivility in a coding scheme (because incivility is often in the eye of the beholder), and suggests that our findings are a somewhat conservative measure of incivility in political discussions.

 

This coding scheme was applied to every comment posted within 24 hours to every Daily Star article published between October 17 and November 6, 2011. The total number of comments analyzed, after removing duplicate posts, was 6,457. Notably, the Arizona Daily Star has a policy prohibiting abusive comments and removes comments deemed so. Removed comments amounted to only 1.7% of the total comments, however, indicating that the level of incivility observed in our analysis is beneath the site’s threshold for removal. The removed comments were signaled on the site with a placeholder; the content itself was not visible. Removing these placeholders left us with 6,347 comments in the final analysis.

 

Results

Our preliminary results suggest that incivility is a regular part of online discussions. Table 2 reports the percentage of comments that included some form of incivility. Of all the comments posted to the Arizona Daily Star over the three week period of analysis, more than 1 in 5 included at least one form of incivility. This pattern was driven in large part by the most common form of incivility, name calling, which is an ad hominem attack on another individual. Political figures were common targets of these attacks, though commenters attacked one another as well.

 

Table 2: Presence of Incivility in Comments

Category

% of Comments

Total Incivility

21.7%

Name calling

14.3%

Aspersion

2.6%

Lying                    

1.7%

Vulgarity

3.0%

Pejorative for speech

2.0%

Hyperbole

1.2%

Noncooperation

0.3%

Another way to understand just how commonly incivility appeared in these discussions is to consider the extent to which an entire discussion could be free of incivility (that is, the possibility that some article would have an entire comment thread without any uncivil comments present). It turns out it was more likely to have a discussion with incivility present than not. Of all the articles that had at least some comments posted, roughly 52 percent had at least one uncivil comment. So, entirely civil discussions are present on the Arizona Daily Star website, but the majority of discussions have at least some incivility.

 

Our coding team also recorded the number of “thumbs up” and “thumbs down” each comment received. These features on the site are a simple way for commenters to show approval or disapproval of fellow commenters’ views. The presences of these features allowed for a simple test of whether incivility is well received when it turns up in comments. Our findings suggest that incivility is neither uniquely popular nor unpopular among commenters. Indeed, civil comments received a net positive score in the thumbs metric in 68.6 percent of the cases, nearly identical to the 68.1 percent mark received by uncivil comments. What this would appear to indicate is that incivility is such a familiar part of these online discussions that commenters do not use its presence or absence to determine their evaluation of a comment. Incivility is simply there. This explanation is further supported by the fact that only one time in the 6,347 comments analyzed did a commenter explicitly mention the uncivil tenor of the comment thread. It seems that incivility is both common and accepted in online discussions at the Arizona Daily Star.

 

 

 

 

Table 3: Identity Information Included
in Comments

Category of
Identity Information

% of Comments

Gender

<0.01%

Ethnicity

<0.01%

Race

<0.01%

Religion

0%

Age

0.2%

Class

0.1%

Education

0.1%

Sexual orientation

<0.01%

Health status

0%

Organizational membership

<0.01%

Employment status

0.3%

Geography

0.3%

Party identification

0.3%

Political ideology

0.2%

 

A secondary goal of the project was to explore features of the discussion context that might be associated with incivility. Anonymity is one factor that has been identified in previous research as important to consider. The Arizona Daily Star uses a standardized format in which commenters are identified by their first name and the first initial of the last name (based on the name they entered to create an account with the newspaper and post comments). Because comments were all identified consistently by name, we used the identity information communicated in their comments based on 14 types of identity characteristics to assess anonymity (Marx, 1999). The degree to which commenters communicated one or more types of identity information makes them appear more or less anonymous. The frequency of each type of information included among comments in sample is reported in Table 3. Relatively few comments included information about the commenter’s identity. These results suggest that commenter at the Arizona Daily Star are relatively anonymous. Unfortunately, this paucity of identity information made it impossible to conduct statistical tests to examine the relationship between anonymity and incivility.

 

 

Implications and Future Research

 

Empirical inquiry into civil discourse has shown that, although the effects of mediated incivility are not uniformly negative, it does appear to weaken political trust (Mutz & Reeves, 2005). Given this, the regular presence of incivility in the discussions on the Arizona Daily Star website is normatively concerning. It also seems unnecessary. The vast majority of the incivility came in the form of name calling, a form of attack that has nothing to do with the substance of meaningful political disagreement.

 

Going forward, we have two objectives. First, we plan to investigate the characteristics of the newspaper articles to which these comment are attached to see if we can identify factors that contribute to higher incidence of incivility. These insights will potentially have practical applications for journalists who create online news content and for citizens who hope to use online forums to participate in meaningful political discussions. Second, we hope to secure additional funding to expand our analysis to other newspaper around the U.S., allowing us to draw more general conclusions about the tenor of online discussions and also explore the possibility of regional differences in incivility.

 

 

 

 

 

 

References

Herbst, S. (2010). Rude democracy: Civility and incivility in American politics. Philadelphia, PA: Temple University Press.

Jamieson, K. H.  (1997, March 1).  Civility in the House of Representatives.  APPC Report #10.  The Annenberg Public Policy Center at the University of Pennsylvania.  Retrieved August, 12, 2012, from http://www.annenbergpublicpolicycenter.org/NewsDetails.aspx?myId=195

Jamieson, K. H.  (2011, September 27).  CIVILITY IN CONGRESS (1935-2011) as reflected in the taking down process.  APPC Report No. 2011-1. The Annenberg Public Policy Center at the University of Pennsylvania.  Retrieved August, 12, 2012, from http://www.annenbergpublicpolicycenter.org/Downloads/Civility/Civility_9...

Marx, G. T. (1999). What’s in a name? Some reflections on the sociology of anonymity. The Information Society, 15, 99-112.

Price, V. (2006). Citizens deliberating online: Theory and some evidence. In T. Davies & B. S. Noveck (Eds.), Online deliberations: Design, research, and practice (pp. 37-58). Chicago, IL: University of Chicago Press.

Public Religion Research Institute (2010, November 5-8). PRRI/RNS religion news survey. Retrieved from www.publicreligion.org.

Stromer-Galley, J. (2007). Measuring deliberation’s content: A coding scheme. Journal of Public Deliberation, 3(1). Available: http://services.bepress.com/jpd/vol3/iss1/art12.

Stromer-Galley, J., & Muhlberger, P. (2009). Agreement and disagreement in group deliberation: Effects on deliberation satisfaction, future engagement, and decision legitimacy.  Political Communication, 26(2), 173-192.