Content Credibility Indicators Vocabulary (CCIV)

This document aims to increase interoperability among systems cooperating to make the web more trustworthy. The approach taken here is to exchange "indicators" which potentially show which items of web content (initially news articles) are worthy of greater trust.

Article Structure

Is Original

Data Type: Multiple Choice
Definition: (A) Most likely original (B) Appears to be a copy of one or more articles, with some portions different or remixed (C) Extensive quoting from another source, with some original content (D) A wholesale duplicate of another article
Frame as a Question: Has the text of this article appeared in exactly the same words or very similar words in another publication?
Location: Article Context
Potential Markup Effort: Medium
Priority Rationale: Like doing a reverse image search, knowing that the article is original does not by itself guarantee credibility. However, knowing if it's *not* original can be a flag that the site is either duplicating content or running content from a newswire service. We might want to set this as scalar, as there might be slight modifications to original content.
Proposed Priority for WebConf Paper: Very High
Rationale for Inclusion: Sometimes article texts get repurposed in new publications. This can be due to licensing agreements from a wire service or the article can simply be stolen without crediting the original article. Sometimes the article is copied wholesale while other times some words are changed or a new article is created that copies from multiple articles. Finally, some articles will quote extensively from one or more other articles with only a small amount of original reporting or writing.
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Attribution of Non-Original Content

Data Type: Multiple Choice
Definition: (A) Attribution was not given (B) Attribution was given but was inaccurate (C) Attribution was given and was accurate (D) Unclear which is the original
Frame as a Question: If the content of the article is not original, was attribution given and if so, was the attribution accurate?
Location: Article Context
Potential Impact: Medium
Potential Markup Effort: Medium
Proposed Priority for WebConf Paper: Medium
Rationale for Inclusion: Originality or non-originality is not by itself a sufficient condition for measuring credibility. Understanding the attribution of non-original content is going to be key.
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Article/Site Metadata

Ads.txt Exists

Data Type: Boolean
Definition: ads.txt (Authorized Digital Sellers) is an Interactive Advertising Bureau initiative. It specifies a text file that companies can host on their web servers, listing the other companies authorized to sell their products or services. This is designed to allow online buyers to check the validity of the sellers from whom they buy, for the purposes of internet fraud prevention. (from https://en.wikipedia.org/wiki/Ads.txt)
Dynamic; Change over Time?: true
Frame as a Question: Does an ads.txt file exist on the domain?
Good Toy Example: https://www.huffingtonpost.com/ads.txt
Known Weakness: Standards still under development
Machine Generation from Article: true
Notes: Status: DRAFTING
Potential Markup Effort: Very Low
Problematic Toy Example: https://www.reddit.com/r/adops/comments/703k4s/is_there_a_legitimate_reason_to_accept_an_adstxt/
Rationale for Inclusion: Evolving standard. some initial brainstorming on other potential features: ```- the number of sellers in given file (could consider SSPs and seller accounts separately) - the balance of DIRECT vs. RESELLER flags - which SSPs are included (some are more quality/stringent than others) - weighting SSPs included by the number of accounts per SSP.``` I’m not sure that any of those would be strong misinfo signals especially since it seems like publishers are still figuring out how to handle ads.txt, but could be things to explore.
Release Target: June 2018
Source Team: Credibility Coalition Study: June 2018
Supporting Evidence: https://wiki.appnexus.com/display/industry/AppNexus+Support+for+Ads.txt
Tags: Revenue Model

Logic/Reasoning

Straw Man Argument

Data Type: Likert
Definition: Presentation of a counterargument as a weaker, more foolish version of the real counteargument
Frame as a Question: Does the author present the counterargument as a weaker, more foolish version of the real counterargument (use a Straw Man Argument)? If so, highlight the relevant section(s).
Location: Article Content
Potential Markup Effort: Very High
Problematic Toy Example: "Proponents of nuclear energy have argued that it is safer than it used to be. But we don’t want energy that is barely safer than Chernobyl."
Proposed Priority for WebConf Paper: High
Release Target: 1
Source Team: Public Editor

Slippery Slope Argument

Data Type: Likert
Definition: Argument that one small change will lead to a major change
Frame as a Question: Does the author say that one small change will lead to a major change (use a slippery slope argument)? Highlight the relevant section(s).
Location: Article Content
Potential Markup Effort: Very High
Problematic Toy Example: "If we allow human cloning, we will soon be overrun by armies of human clones."
Proposed Priority for WebConf Paper: High
Release Target: 1
Source Team: Public Editor

Naturalistic Fallacy

Data Type: Likert
Definition: Suggestion that something is good because it is natural or bad because it is not natural
Frame as a Question: Does the author suggest that something is good because it is natural, or bad because it is not natural (the naturalistic fallacy)?
Location: Article Content
Potential Markup Effort: Medium
Problematic Toy Example: "Vaccines aren't natural; they're full of all kinds of lab-manufactured chemicals. That stuff can't possibly be good for you."
Proposed Priority for WebConf Paper: High
Release Target: 1
Source Team: Public Editor

False Dilemma

Data Type: Likert
Definition: Presentation of a complicated choice as if it is binary
Frame as a Question: Does the author present a complicated choice as if it were binary (construct a false dilemma)? If so, highlight the relevant section(s).
Location: Article Content
Potential Markup Effort: Very High
Problematic Toy Example: "America; love it or leave it! If you're not happy with U.S. policies, you should just move somewhere else."
Proposed Priority for WebConf Paper: Very High
Release Target: 1
Source Team: Public Editor

Causal Claim Types

Data Type: Multiple Choice
Definition: General Causal Claim Singular Causal Claim No Causal Claim
Frame as a Question: Is a general or singular causal claim made? Highlight the section(s) that supports your answer.
Location: Article Content
Potential Impact: High
Potential Markup Effort: Very High
Release Target: 1
Source Team: Public Editor

Calibrating Confidence - Level of Confidence

Article Example URL: http://www.theblaze.com/news/2017/09/02/scientist-shuts-down-climate-change-alarmists-with-new-report-about-hurricane-harvey/
Data Type: Likert
Definition: Expressed confidence in a claim
Frame as a Question: Do they acknowledge uncertainty or the possibility that things might be otherwise? If so, highlight the relevant section(s).
Location: Article Content
Potential Markup Effort: Very High
Problematic Toy Example: "According to Mass, the idea that human-caused climate change had any effect on Harvey is more than far-fetched — it’s downright not true."
Proposed Priority for WebConf Paper: Very High
Release Target: 1
Source Team: Public Editor

Calibrating Confidence - Justification

Article Example URL: http://www.theblaze.com/news/2017/09/02/scientist-shuts-down-climate-change-alarmists-with-new-report-about-hurricane-harvey/
Data Type: Likert
Definition: Expressed confidence in a claim, with justification
Frame as a Question: To what extent does their confidence in their claims seem justified?
Location: Article Content
Potential Markup Effort: Very High
Problematic Toy Example: "According to Mass, the idea that human-caused climate change had any effect on Harvey is more than far-fetched — it’s downright not true."
Proposed Priority for WebConf Paper: Very High
Release Target: 1
Source Team: Public Editor

Appeal to Fear Fallacy

Data Type: Likert
Definition: Exaggeration of the dangers of a situation and use of scare tactics in an attempt to persuade
Frame as a Question: Does the author exaggerate the dangers of a situation and use scare tactics to persuade (the appeal to fear fallacy)?
Location: Article Content
Potential Markup Effort: Very High
Release Target: 1
Source Team: Public Editor

Outbound References

Source Types

Data Type: Multiple Choice
Definition: None Experts Studies Organizations Other
Frame as a Question: Which of the following types of sources are cited in the article? Check all that apply. If Other, please highlight.
Location: Article Context
Potential Impact: Medium
Potential Markup Effort: Medium
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Contains Link to Scientific Journals

Data Type: Boolean
Definition: A simple link to the a scientific journal article that backs up the assertion made. This may also be paired with a URL to the specific article.
Frame as a Question: Is a link provided in the article to where the original content came from?
Location: Article Context
Potential Impact: Medium
Potential Markup Effort: Low
Proposed Priority for WebConf Paper: Medium
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Accuracy of representation of source article

Article Example URL: https://climatefeedback.org/evaluation/breitbart-misrepresents-research-58-scientific-papers-falsely-claim-disprove-human-caused-global-warming-james-delingpole/
Data Type: Likert, with Multiple Choice
Definition: This article properly characterizes the methods and conclusions of the cited or quoted source. In addition to a Likert measure, two other options are possible: (A) Unable to find source, (B) Source is behind a paywall
Frame as a Question: Does this article properly characterize the methods and conclusions of the original source?
Location: Article Context
Potential Impact: High
Potential Markup Effort: Very High
Priority Rationale: Ability to accurately reflect the source articles in outbound references requires a good deal of knowledge of the field and of the scientific method in general and may be difficult to replicate.
Problematic Toy Example: "It’s sad that the blogger did not understand what this study is about..."
Proposed Priority for WebConf Paper: Very High
Rationale for Inclusion: This ideally would be done per source article
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Academic Journal Impact Factor

Data Type: Integer
Definition: From Wikipedia: The impact factor (IF) or journal impact factor (JIF) of an academic journal is a measure reflecting the yearly average number of citations to recent articles published in that journal. It is frequently used as a proxy for the relative importance of a journal within its field; journals with higher impact factors are often deemed to be more important than those with lower ones. https://en.wikipedia.org/wiki/Impact_factor
Frame as a Question: What is the impact factor of the journal or conference cited?
Location: Article Context
Potential Impact: Medium
Potential Markup Effort: Medium
Proposed Priority for WebConf Paper: Medium
Release Target: 1

Revenue Model

Spam or Clickbait Advertisements

Data Type: Likert
Definition: The page of the article has spammy or clickbaity advertisements. This is limited to a subjective assessment at this time.
Frame as a Question: How strongly do you agree or disagree that the page of the article has spammy or clickbaity advertisements?
Location: Article Context
Potential Impact: Medium
Potential Markup Effort: Medium
Priority Rationale: Advertisements that the reader feels, subjectively, are spammy or clickbaity could indicate a site that prioritizes clicks and shares over useful content and therefore represents financially-motivated misinformation.
Proposed Priority for WebConf Paper: High
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Number of Advertisements

Data Type: Integer
Definition: There are multiple types of ads to look for. (1) Display ads. These are boxes that are clearly advertisements, typically in the form of a graphic image or, in the case of Google Adwords, a box with text. (2) Content recommendation engines, specifically, Taboola, Outbrain, Tivo, RevContent. A box of content recommendations on a page counts as one. (3) Sponsored content. This is content recommended on the site with a clear label: “Sponsored.” (4) Call for social sharing. (5) Call to subscribe to a mailing list
Frame as a Question: How many ads appears on the article page?
Location: Article Context
Notes: Credibility Coalition Study note: intention to split post Summer 2018
Potential Impact: Medium
Potential Markup Effort: Low
Priority Rationale: The number of ads can be an indicator of a site that is focused on financially-motivated misinformation. A simple tally of ads can reveal a lot, if there’s an unusual number of ads on the page.
Proposed Priority for WebConf Paper: High
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Aggressive Social Shares

Definition: The page of the article has aggressive social shares, which may include calls to share the article within the text. This is limited to a subjective assessment at this time.
Dynamic; Change over Time?: true
Frame as a Question: How strongly do you agree or disagree that the page of the article has aggressive social shares?
Good Toy Example: A call to share in the content of the article text itself.
Notes: One of two replacing https://credweb.org/cciv/#aggressive-advertisements-or-social-shares
Rationale for Inclusion: Replaces https://credweb.org/cciv/#aggressive-advertisements-or-social-shares which is double barreled.
Release Target: June 2018
Source Team: Credibility Coalition Study: Summer 2018
Supporting Evidence: Credibility Coalition Study: Web Conference 2018 examples

Aggressive Advertisements or Social Shares

Data Type: Likert
Definition: The page of the article has aggressive advertisements. This is limited to a subjective assessment at this time.
Frame as a Question: How strongly do you agree or disagree that the page of the article has aggressive advertisements?
Location: Article Context
Notes: Deprecated for Credibility Coalition Study: Summer 2018
Potential Impact: Medium
Potential Markup Effort: Medium
Priority Rationale: Advertisements that the reader feels, subjectively, are aggressive could indicate a site that prioritizes clicks and shares over useful content and therefore represents financially-motivated misinformation.
Proposed Priority for WebConf Paper: High
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Aggressive Advertisements

Definition: The page of the article has aggressive advertisements. This is limited to a subjective assessment at this time.
Dynamic; Change over Time?: true
Frame as a Question: How strongly do you agree or disagree that the page of the article has aggressive advertisements, including calls to join a mailing list?
Good Toy Example: An ad that follows you around as you scroll.
Notes: One of two replacing https://credweb.org/cciv/#aggressive-advertisements-or-social-shares
Rationale for Inclusion: Replaces https://credweb.org/cciv/#aggressive-advertisements-or-social-shares which is double barreled.
Release Target: June 2018
Source Team: Credibility Coalition Study: Summer 2018
Supporting Evidence: Credibility Coalition Study: Web Conference 2018 examples

Rhetoric

Title Representativeness Types

Data Type: Multiple Choice
Definition: Types of title representativeness (A) Title is on a different topic than the body (B) Title emphasizes different information than the body (C) Title carries little information about the body (D) Title takes a different stance than the body (E) Title overstates claims or conclusions in the body (F) Title understates claims or conclusions in the body
Frame as a Question: How is the title unrepresentative of the content of the article? (Select all that apply).
Location: Article Content
Potential Impact: Very High
Potential Markup Effort: Medium
Proposed Priority for WebConf Paper: Very High
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Title Representativeness

Data Type: Likert
Definition: A measure of how representative the content of the title is with the content of the body copy.
Frame as a Question: Does the title of the article accurately reflect the content of the article?
Location: Article Content
Potential Impact: Very High
Potential Markup Effort: Medium
Proposed Priority for WebConf Paper: Very High
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Proportion (Exaggeration - Minimization Spectrum)

Article Example URL: Examples taken from https://en.wikipedia.org/wiki/Exaggeration, https://en.wikipedia.org/wiki/Minimisation_(psychology)
Data Type: Scale
Definition: The extent to which language in the text is proportional to the situation, or exaggerates or minimizes events. Exaggeration as defined in Webster 1913, "...the act of doing or representing in an excessive manner;..." and Minimization as defined in Oxford Living Dictionary accessed June 2018, "1.1 Represent or estimate at less than the true value or importance"
Frame as a Question: Is the description an extreme exaggeration, an extreme minimization, or proportional to the event or situation described?
Good Toy Example: "[Certain people group] are either all good or all bad," or "It's just a flesh wound" (_Monty Python and the Holy Grail_ Black Knight's response to his having his left arm severed)
Location: Article Content
Notes: DRAFT. Replacing part of https://credweb.org/cciv/#exaggeration for Credibility Coalition study
Rationale for Inclusion: Refining 'overly emotional language' indicator into multiple
Release Target: June 2018
Source Team: Credibility Coalition Study: Summer 2018

Exaggerated Claims

Data Type: Likert
Definition: Claims are exaggerated, as indicated by the tone
Frame as a Question: 1.21 mc Does the author exaggerate any claims? If so, highlight the relevant section(s).
Location: Article Content
Potential Impact: High
Potential Markup Effort: High
Release Target: 1

Emotionally Charged Tone

Data Type: Likert
Definition: Article has an emotionally charged tone
Frame as a Question: Does the article have an emotionally charged tone? (i.e, outrage, snark, celebration, horror, etc.). If so, highlight the relevant section(s).
Location: Article Content
Potential Markup Effort: High
Proposed Priority for WebConf Paper: High
Release Target: 1
Source Team: Public Editor

Emotional Valence

Data Type: Scale
Frame as a Question: Is the language extremely negative, extremely positive, or somewhere in the middle?
Good Toy Example: "VADER is VERY SMART, handsome, and FUNNY!!!"
Location: Article Content
Machine Generation from Article: true
Rationale for Inclusion: Refining 'overly emotional language' indicator into multiple
Release Target: June 2018
Source Team: Credibility Coalition
Supporting Evidence: VADER (Valence Aware Dictionary and sEntiment Reasoner) Natural Language Processing library, available at https://github.com/cjhutto/vaderSentiment

Clickbait Headline

Data Type: Likert
Definition: A measure of how much the title of the article conforms to a predetermined set of clickbait genres.
Frame as a Question: Is the headline clickbaity?
Known Weakness: Major news outlets are starting to borrow clickbait conventions to attract readership
Location: Article Content
Potential Impact: Very High
Potential Markup Effort: Medium
Priority Rationale: High clikbaity-ness in a headline can indicate that the article is engineered for a surface-level virality. This may be particularly instructive when seen in the context of reader behavior.
Proposed Priority for WebConf Paper: Very High
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Clickbait Genres

Data Type: Multiple Choice
Definition: A typology of clickbait headlines: Listicle (“6 Tips on …”) Cliffhanger to a story (“You Won’t Believe What Happens Next”, “Man Divorces His Wife After Overhearing This Conversation”) Provoking emotions, such as shock or surprise (“...Shocking Result”, “...Leave You in Tears”) Hidden secret or trick (“Fitness Companies Hate Him...”, “Experts are Dying to Know Their Secret”) Challenges to the ego (“Only People with IQ Above 160 Can Solve This”) Defying convention (“Think Orange Juice is Good for you? Think Again!”, “Here are 5 Foods You Never Thought Would Kill You”) Inducing fear (“Is Your Boyfriend Cheating on You?”) Other
Frame as a Question: What clickbait techniques does this headline employ (select all that apply)?
Location: Article Content
Potential Impact: Very High
Proposed Priority for WebConf Paper: Very High
Release Target: 1
Source Team: Credibility Coalition Study: Web Conference 2018

Intro