This document specifies various types of information, called credibility signals, which are considered potentially useful in assessing credibility of online information.

This document is automatically assembled from a crowd-source Google doc and various data sources. It may contain completely bogus content. You may prefer the most recent stable release


Comments are welcome and are especially useful if they offer specific improvements which can be incorporated into future versions. Please comment either by raising a github issue or making inline comments on the google doc (easily reached using the pencil 🖉 link in the right margin). If neither of those options work for you, please email your comments to public-credibility-comments@w3.org (archive, subscribe).

Introduction

src

Purpose

src

This document is intended to support an ecosystem of interoperable credibility tools.  These software tools, which may be components of familiar existing systems, will gather, process, and use relevant data to help people more accurately decide what information they can trust online and protect themselves from being misled. We expect that an open data-sharing architecture will facilitate efficient research and development, as well as an overall system which is more visibly trustworthy.

The document has three primary audiences:

  1. Software developers and computer science researchers wanting to build systems which work with credibility data.  For them, the document aims to be a precise technical specification, stating what they need for their software to interoperate with any other software which conforms to this specification.
  2. People who work in journalism and want to review and contribute to this technology sphere, to help make sure it is beneficial and practical.
  3. Non-computer-science researchers, interested in helping develop and improve the science behind this work.

In general, we intend for this document to be:

Credibility Data

src

The document builds on concepts and terminology explained in Technological Approaches to Improving Credibility Assessment on the Web.  Our basic model is that an entity (human and/or machine) is attempting to make a credibility assessment — to predict whether something will mislead them or others — by carefully examining many different observable features of that thing and things connected with it, as well as information provided by various related or trusted sources.

To simplify and unify this complex situation, with its many different roles, we model the situation as a set of observers, each using imperfect instruments to learn about the situation and then recording their observations using simple declarative statements agreed upon in advance. Because those statements are inputs to a credibility assessment process, we call them credibility signals.  (The term credibility indicators is sometimes also used.)

This document, then, is a guide to these signals.  It states what each observer might say and exactly how to say it, along with other relevant information to help people choose among the possible signals and understand what it means when they are used.

Because this is a new and constantly-changing field, we do not simply state which signals should be used.  Instead, we list possible signals that one might reasonably consider using, along with information we expect to be helpful in making the decision.

Example

src

[explain]

Assessing credibility of https://news.example/article-1

   Looking at title

      I consider it to be clickbait

      It's clickbait because it's a cliffhanger

   Looking at article

      It cites scientific research

   Looking at provider

      Established in 1974

      Owned domain since 2006

Factors in Selecting Signals

src

When building systems which use credibility signals and trying to decide which signals to use, there are different factors to weigh.  This section is aspirational; we hope this document will in time provide guidance on all these factors.  

Measurement Challenges

src

There are factors about how difficult it is to get an accurate value[a][b] for the signal[c][d]:

  1. Do people independently observing it get approximately the same value?  
  2. Do observations vary with the culture, location, language, age, beliefs, etc, of the people doing the observation?
  3. Would the same people make the same observation in future months or years?
  4. How much time and effort does it take people to make the observation?
  5. Do people need to be trained to make this specific observation?
  6. What kind of general training do people need (eg a journalism degree) to do it?
  7. How do machines compare to humans in making this observation, in terms of cost, quality, types of errors, and susceptibility to being tricked.

Many of these factors can be measured using inter-rater reliability (IRR) techniques.  When studies have made such measurements, our intent is to include that data in this document.

Here is a table of the data we have.  Excerpts are listed with the relevant signals.

Value in Credibility Assessment

src

Another important set of factors relates to how useful the measurement is in assessing credibility, assuming the observation itself is accurate.

  1. Does the signal have a strong correlation to content accuracy, itself determined by consensus among experts[e][f]?
  2. Is it particularly indicative of credibility when used in combination with other signals?  (For example, as part of computing the value of a latent variable.)
  3. Is it conceptually easy for people to understand?
  4. Do professionals in the field think it's likely to be a useful signal?
  5. How dependent are these characteristics on the culture or time period being considered?
  6. How dependent are these characteristics on the subject matter of the information being assessed for credibility?

Feedback Risks (“Gameability”)

src

One should also consider how the overall ecosystem of content producers and consumers might be changed by credibility tools adopting the signal. Once attackers see it’s being used, a signal that works well today might stop working, or even be used to make things worse. See Feedback Risks.

  1. Is it disproportionately useful for attackers (eg viral call to action) ?  If so, making this a negative credibility signal should generally be beneficial
  2. Is it disproportionately expensive for attackers (eg journalistic language) ?  If so, making this a positive credibility signal should generally be beneficial.
  3. Who might get impacted by “friendly fire”?  Even if adopting a signal might — on average — harm attackers more than everyone else, certain individuals or communities who have done nothing wrong might be penalized.  Tradeoffs must be carefully made, ideally in a consensus process with the impacted people.

Interoperability

src

The value of sharing signal data depends on how that signal is used by other systems.

  1. Are others producing data using this signal?
  2. Are there useful data sets available?
  3. Are others consuming data, paying attention to reported observations of this signal?
  4. Are there tools which work with it, eg running statistics?
  5. Is the definition clear and unambiguous, so people using it mean the same thing?
  6. Are there clear examples?
  7. Is there an open history of commentary, with questions and answers, and issues being addressed by various implementers?
  8. Is documentation available in multiple languages?
  9. If the definition is under development, how can one participate?
  10. If the definition could possibly change, who might change it, and under what circumstances?
  11. Are there any intellectual property considerations? See W3C Patent Policy.
  12. Is there a test suite / validation system for helping confirm that an implementation is working properly?
  13. Are there implementation reports, confirming that tools are functioning properly, according to the testing system? (For an example, see ActivityPub).

Publishing Credibility Data

src

TBD, basically follow schema.org technique using json-ld.

Consuming Credibility Data

src

TBD, point to some tools and the relevant specs.  Basically JSON-LD.

Organization of this document

src

Section 1 (“Introduction”) provides instructions for how to use and help maintain this document, along with general background information.

The rest of this document, after the introduction, is a list of signals and information about them, as discussed in the introduction.  The signals are organized into related groups, in hierarchical sections.  At the lower levels of the hierarchy are the signals themselves, while the higher levels provide grouping of the signals, to help people understand them.

One important level of the hierarchy identifies the subject type of the signal.  This is the conceptual entity being examined, considered, or inspected, when one makes the observation being recorded in the signal data.  This could be imagined in different ways: when you are observing a claim made in the 3rd paragraph of an article published in some newspaper, are you observing the claim, the paragraph, the article, the newspaper, or even the author of the article?  In general, we aim for the smallest granularity that makes sense, which in this case would probably be the claim.

At times, it may not be obvious to which subject type a signal belongs, or it could sensibly belong with several different ones.  In this case, it might be moved to a different section in the document as people come to understand it better.  When it’s not clear, there should be links from the places a signal could reasonably be to the place it actually is.

This may require discussion, and might remain open for debate.  When a signal or group of signals makes sense in two places, consider linking it from the places it isn’t, to help people find it.

In many cases, a signal could be seen as a set of similar signals which are not strictly identical. This can be handled by adding additional signal headings with the finer distinction, when necessary. In this case, template statements might appear under more than one signal.

Note that sections may be moved and renumbered.  Do not rely on section numbers remaining the same.  For linking to a part of the document, consider using the gdocs h.xxxxxx fragment ids, provided by the Table of Contents; those should remain stable.  Also, whenever changing a heading, especially a signal heading, if someone might be referring to it by name, please move the old text into a paragraph starting “Also called:”.  

Template Statements

src

The most important thing about a signal definition is to be clear what observation the signal data is recording. If the signal heading is “Article length”, does that mean length in words or bytes or characters or some other metric? Does it include the title? For each signal, we want an easy way to communicate its definition that is short but clear, while being as detailed as necessary.

The technique we use here is to express the semantics of the signal using plain and simple sentences in natural language which convey the same knowledge as the signal data. If you imagine people using credibility software exchanging these statements (perhaps in text messages or on Twitter), you should get the right semantics. You can assume metadata, like who sent it and when it was sent is available, so the statements can include terms like “I” and “now”.

For machine-to-machine data Interoperability, these template sentences and the signal heading are turned into a data schema, after which the JSON-LD/schema.org/sematic web/linked data technology stack can be used.

The statements we use are templates because they abstract over a variety of similar sentences which differ in specific limited ways.  For example, these statements:

  1. I have examined the article at https://example.com/alice and find it highly credible
  2. I have examined the article at https://example.com/brian and find it highly credible
  3. I have examined the article at https://example.com/casey and find it highly credible

are all the same, except in the URL.   We convey this using a template statement, which has a variable portion in square brackets, like:

Tech note

If we (automatically or manually) map this template to a property with the pname :iHaveExaminedHighlyCredible, then the sentence number 2 above would be encoded in turtle as

  • { <https://example.com/brian> :iHaveExaminedHighlyCredible true }.

Alternatively, we could make it a class, but boolean valued properties may be better, so that all signals remain as properties..

The bracketed template expression “[subject]” is required in every template, to indicate what entity is being observed.  Additional bracket expressions can be used when there are other elements of the statement to make variable.  In particular, [string] (for text in quotes) and [number].

(For now, try to just use those three.  Software and documentation is being developed to allow more features. If you find this too restrictive, go ahead and write something else inside the square brackets and we'll deal with it later, but include a question mark so it's clear you knew you were making it up.)

An example needing multiple variables:

  1. https://example.com/alice took 4.75 seconds to load, just now.
  2. https://example.com/brian took 5.9 seconds to load, just now.

could be matched by:

Instructions for editing this document

src

As an experiment, this document is currently set so everyone can edit it, like Wikipedia. It is the Google docs version that is editable. We suggest you change the “Editing Mode” to “Suggesting” (using the pencil icon in the upper-right) until you are quite familiar with this document. You may also comment using the usual Google Docs commenting features.

If you make or suggest any edits to this document, you are agreeing to the W3C Community Contributor License Agreement which has significant copyright and patent implications.

The subsections below give some advice for how to make edits which are helpful.

Expand discussion

src

Each section should begin with a short introduction written with a neutral point of view, reflecting consensus about why the signal might be useful and what the risks might be. To enable consensus among a broad community, the intent is for this text to be developed iteratively, with each contributor adding their perspective while respecting what is already present.

Questions and minor concerns should generally be added as annotations using the “Add a Comment” function, without editing the document. If they become issues requiring back-and-forth discussion, they should be turned into github issues and linked from the most relevant place in this document with a paragraph starting “Issue:”

These discussion sections are intended to be nonnormative. That is, they do not say how software using the signal is required to behave for interoperability. The normative content of this specification is the template statements and the mapping of the statements to RDF.

Add new template statements

src

If you are confident you understand what a signal is intended to measure, and think you can provide a template statement which expresses it more clearly and simply, with little ambiguity, please add a new row to the bottom of the “Proposed template statements” table and add your entry.  Please also put the next higher number in the Key field for reference, and your name in the By field. This “by” field is optional; it is intended to help simplify discussion, telling people who to talk to, and to give some credit. Listing the name of a large group in this field is not particularly useful.  

After adding an entry, for a short time (perhaps a few hours, guided by any comments on it) it’s okay to edit it if you change your mind. After that, please leave it, and just add a new row for the new version. You can put new versions in the middle of the table and use keys like 1a.

Add new signals

src

Once you’re familiar with the structure of this document and all the signals in your area of interest, you may add new signal sections (with a title starting “Signal:” or even new group sections.  (For heading numbering, you can use the “Table of contents” add-on from LumApps to number the headers. Or just leave the numbering for someone else using the add-on.)

When you add a new signal, please copy this table to the new section, and then fill in at least one row to clarify what the signal data conveys.

Key

Proposed Template Statement

By

Contributors

src

Folks who add content to this document are encouraged to add themselves in this section, potentially with some affiliation & credential information.  This also allows the “By” column to stay short, as people can use short forms of names (eg only first or last name, if unique in this doc).

Sources

src

This document is assembled from multiple data sources. They provide both the overall structure of this document and the details about each signal, include definitions, example data, and implementation status.These sources are fetched started with a source list, which appears as the first entry below. In general, text in this document links back to its source with a link-out icon.

The sources used for this current view were:

If you want to privately experiment with bookmarkable alternative views generated using a different source list, try Custom View of Credibility Signals.

Subject type: Claim

src

This section is for signals about claims.

A claim is “an assertion that is open to disagreement; equivalently, a meaningful declarative sentence which is logically either true or false (to some degree); equivalently, a proposition in propositional logic.” [credweb report]

Claims can be stated (with various decree of clarity) in some content or implied by the content (even non-textual content, like a photograph).

Claims are usually the smallest practical granularity. Credibility data about claims is largely focussed on what other sources have said about that claim, as in fact checking, but could also involve relationships between claims and textual analysis of claim text.

Claim Review

src

The “ClaimReview” model developed at s[g][h][i]chema.org grows out of the tradition of independent, external fact-checking, as in PolitiFact.  With this model, a fact-checker reviews a claim, typically made by a public figure, and then publishes a review of that claim, a “claim review”. Within schema.org, this parallels other reviews, like restaurant reviews.

[ Can we fit claimreview neatly into this observer/signal model?  It’s a bit of a stretch.  TBD. ]

Signal: Fact-check status of claim 

src

From Section 7.7.1. Signal: Article has a central claim, claims in articles according to Credibility Coalition WebConf2018 and more recent studies includes the following values for fact-check results at the time of the study: false, true, unclear, mixed; not finding a fact-check is equivalent to an empty statement.

Interoperability with ClaimReview: This signal seems to relate to https://schema.org/reviewRating and bestValue of https://schema.org/ClaimReview, with bestValue in this case equal to VERIFIED; further discussion is needed with members of schema.org to confirm.

Signal: Fact-check status of claim — VERIFIED

src

Ref Definition (Template) Tags
0bd70468 An IFCN signatory did a fact-check and verified claim [Claim].
cbd33df5 The fact-check result by [Venue] of claim [Claim] is that it is TRUE.

Signal: Fact-check status of claim — REFUTED

src

Ref Definition (Template) Tags
16c7ee48 An IFCN signatory did a fact-check and refuted claim [Claim].
97e6d1a0 The fact-check result by [Venue] of claim [Claim] is that it is FALSE.

Signal: Fact-check status of claim — UNCLEAR

src

Ref Definition (Template) Tags
c9434942 The fact-check result by [Venue] of [Claim] is UNCLEAR.

Signal: Fact-check status of claim — MIXED [j]

src

Ref Definition (Template) Tags
9a0da6b7 The fact-check result by [Venue] of [Claim] is that the claim contains elements that are TRUE and FALSE.

Signal: Claim - Risk of Harm

src

Ref Definition (Template) Tags
d0d44331 [Claim] is a claim that asserts a risk of harm.

Developed for CredCo Political Indicators Study 2018-19. Can be used in connection with 7.2.3. Signal: Generalization/Characterization of Group.

Signal: Claim - Coded Meaning

src

Ref Definition (Template) Tags
9f86e8b2 [ClaimA] is a claim that equals another claim, [ClaimB].[k]

Developed for CredCo Political Indicators Study 2018-19. Original example question: “Are there claims that contain phrases, words, or coded language that have taken on a special loaded meaning, in the understanding of the speaker and audience?”, with an example of "go to work," used as code for killing during the Rwandan genocide.

Can be used in connection with 7.2.3. Signal: Generalization/Characterization of Group.

Fact-checking Organization[l][m]

src

Signals below [2.2.1. Signal: Fact-checking Organization commitments — member of the IFCN, 2.2.2. Signal: Fact-checking Organization commitments — accuracy and professionalism, and 2.2.3. Signal: Fact-checking Organization commitments — unknown ] were developed in combination with those under  7.8. Claims in Articles, and originally expressed as a question:

If the publication is from a fact-checking organization, what are its commitments to accuracy and other standards?

A) IFCN Signatory

B) Not IFCN signatory but organization/institution with similar standards and commitments

C) Unknown, not discernable

Signal: Fact-checking Organization commitments — member of the IFCN

src

Ref Definition (Template) Tags
177559e3 [Organization] which published fact-check [Webpage] is a member of the International Fact-Checking Network at Poynter (IFCN). on [date].[o][p][q][r]

Signal: Fact-checking Organization commitments — accuracy and professionalism

src

Ref Definition (Template) Tags
ec6b8ebf [Organization] has expressed commitments to accuracy and other fact-checking professional standards similar to IFCN organizations.
35a66989 [Organization] has expressed commitments to accuracy and other fact-checking professional standards.

Signal: Fact-checking Organization commitments — unknown

src

Ref Definition (Template) Tags
c0632f51 [Organization]’s commitments to accuracy and other professional standards are unknown.

Explicitly Unverified Claims

src

From CredCo Political Indicators Study 2018-2019: in some cases, articles may reference claims or pieces of information that do not contain citations or references. In some cases, within an article, an author can make explicit reference to a claim that has not been verified, using language that specifies that the claim has not been validated or proven to be true. This includes language in an article explicitly referencing that a claim has not yet been verified to date - but the claim is being mentioned in the article nonetheless.

This is used in connection with 7.7.2. Signal: Article has a claim.

Key

Proposed Template Statement

By

1

[Claim] is explicitly unverified, containing language such as “charges have not been proven true.” [s]

CredCo

Subject type: Text

src

Includes: phrase, sentence, paragraph, document, document fragment

A text, in this sense, is a sequence of words, with the usual punctuation, and sometimes embedded multimedia content or meaningful layout, like tables.  That is, it’s a document or portion of a document. As examples, a phrase, sentence, paragraph, document section, book chapter, book, and complete book series would typically each count as a text.  

Signals here concern properties of the text, itself, separate from how it might be published (eg on a Web Page, on a billboard, spoken at a rally) or where it might be published (in some Venue).  The text should be considered immutable: a text (in this sense) doesn't change.  If you take a text and change it, you are making a new text, which needs to be reexamined, to see which observations (and thus which signal data) applies to this other, new text.

Issue: (tech) How to represent texts in RDF?  Options include annotation URL with secure hash, annotation object URL with secure hash, data: URI, etc.

Formality

src

Texts adopt a tone to appeal to their audience and/or attempt to convey how the text should be used. For instance, an academic study is written in formal, verbose and grammatically correct language, while a listicle is short, informal and often humorous. The academic study uses these characteristics to convey authority, while the listicle is intentionally unauthoritative.  

Signal: Formal tone

src

Ref Definition (Template) Tags
d721df20 Text of [subject article] has a formal tone.

Signal: Correct Spelling

src

Ref Definition (Template) Tags
0255f049 Text of [subject article] has a formal tone, as measured by correct spelling.

Signal: Correct Grammar

src

Ref Definition (Template) Tags
7bd59393 Text of [subject article] has a formal tone, as measured by correct grammar.

Signal: Informal tone

src

Incorrect or colloquial grammar, slang, and humor are some indications of informal tone.

Ref Definition (Template) Tags
381634fc Text of [subject article] has an informal tone.

Signal: Slang

src

Ref Definition (Template) Tags
a5304038 Text of [subject article] has an informal tone, as measured by slang.

Example sentence: "In this moment we all learned that Johnny Depp isn't a teen and has no clue what "Bae" means." (Source)

Signal: Informal grammar

src

Ref Definition (Template) Tags
d73fb6ec Text of [subject article] has an informal tone, as measured by incorrect, casual or colloquial grammar.

Example sentence: "If you're a Friends fan, you probably know that Ross and Rachel's relationship was...kind of a disaster 95% of the time." (Source)

References or citations

src

Signal: Uses standardized references or citations

src

These standards are required and enforced by professions that demand accuracy, and are typically found in highly researched, and therefore more authoritative, texts. Examples: Legal, academic, or scientific citations, e.g., MLA, APA.

Ref Definition (Template) Tags
c3b7d174 Text of [subject article] uses standardized references or citations.

Example sentence: "Changes in body temperature have long been used as an indicator of injury, inflammation or infection in veterinary medicine (George et al., 2014), however, the use of

temperature devices such as rectal thermometers and thermal microchips can be both invasive

and time consuming (Johnson et al., 2011)." (Source)

Signal: Uses formal but not standardized references or citations

src

Examples: Journalism, nonfiction or explanatory material

Some texts use references extensively, even if they are not written according to a rigid structure. These texts tend to be authoritative but not as authoritative as the texts using the rigidly structured citations. The content of the references is also extremely influential.

Ref Definition (Template) Tags
1e9f797d Text of [subject article] uses references or citations that are not recorded according to professional standards.

Example sentence: "Families that receive benefits are now over $2,600 worse off every year, according to an analysis by the Child Poverty Action Group, an advocacy group." (Source)

Signal: Few to zero references or citations

src

A text with no references to other materials is original content, which often means it is opinion, personal experience, or even fiction. These tend to be less authoritative than texts with references.

Ref Definition (Template) Tags
baa89d32 Text of [subject article] has few or no references or citations.

One exception is a first-hand account, which can become a primary document for later research. These personal accounts, however, should be vetted and cross-referenced with other sources to evaluate its accuracy.

Example sentence: "The shrine is the work of SUNY Purchase sophomore Phillip Hosang, who, like a lot of students at the school, had long heard rumors about a secret room in a men's bathroom somewhere in the visual arts building." (Source)

Pronouns

src

Signal: Many or multiple instances of the pronouns "I" or "you"

src

Texts that use the pronouns "I" or "you" are typically opinion, correspondence or personal account. These texts are usually not trying to be authoritative or explanatory, however, they sometimes form a primary document that is used in secondary research.

Ref Definition (Template) Tags
43280bee Text of [subject article] has many instances of the words "I" or "you."

Example sentence: "After paying close attention to many of your campaigns, I believe you are united by a desire to get things done to help a lot of people who’ve been left behind." (Source)

Signal: Few or no instances of "I" or "you"

src

Texts that do not use first or second person are less likely to be opinion content. However, this is no indication of credibility.

Ref Definition (Template) Tags
dcbf79f6 Text of [subject article] has few or no instances of the words "I" or "you."

"President Trump said he would not overrule his acting attorney general, Matthew G. Whitaker, if he decides to curtail the special counsel probe being led by Robert S. Mueller III into Russian interference in the 2016 election campaign." (Source)

Signal: Vocabulary or reading level

src

Texts with a wide and varied vocabulary, which may include jargon or uncommon words, is an indicator of formal tone.

Incivility and impoliteness

src

Signal: Incivility

src

Ref Definition (Template) Tags
fb2e81c3 Text of [subject article] contains stereotypes, such as calling a person a “faggot,” “terrorist,” or “backward” (e.g. “Muslims are terrorist sympathizers”)
ca34c8ce Text of [subject article] contains threats to people’s individual rights, such as freedom of speech or personal freedom (e.g. “You foolish Republicans better shut up”)
929fa567 Text of [subject article] contains verbalized threat to democracy, such as a proposal to overthrow democratic government by force or undemocratic way (e.g. “Obama is a Muslim Agent with Brotherhood Ties. American people must take him down.”)

Source: Oz, M., Zheng, P., Chen, G. M., & Park, R. H. (2018). Twitter versus Facebook: Comparing incivility, impoliteness, and deliberative attributes. New Media & Society, 20(9), 3400–3419. http://doi.org/10.1177/1461444817749516 

Signal: Impoliteness

src

Ref Definition (Template) Tags
700e99be Text of [subject article] contain insults or name-calling (e.g. “stupid” or “moron”)
5491ed97 Text of [subject article] contains profanity (e.g. “hell” and “damn”)
ec40d14f Text of [subject article] contains words in all capital letters (e.g. “Who flew the planes into the towers on 9/11? ILLEGAL IMMIGRANTS!”)

Source: Oz, M., Zheng, P., Chen, G. M., & Park, R. H. (2018). Twitter versus Facebook: Comparing incivility, impoliteness, and deliberative attributes. New Media & Society, 20(9), 3400–3419. http://doi.org/10.1177/1461444817749516 

Subject type: Image

src

Includes: Picture, Photograph, Drawing, Illustration

Implied association or tone

src

When pictures of people are used, there are often choices about which image to use, and how to manipulate it, to make the person look better/worse or associate them with some positive or negative concept.  Some people have pointed out how media gets to choose, when someone is arrested, whether to use flattering photos provided by supporters or a mug shot provided by the police.

Signal: Flattering image

src

Signal: Unflattering image

src

Originality of Photo Used in an Article

src

These signals are designed with the assumption that the image is used in the broader context of a journalistic article.

Originality Types

src

Signal: Most Likely Original

src

Ref Definition (Template) Tags
58b71d6c Image is mostly likely original.

Signal: Appears to be a Copy, with Some Modifications

src

Ref Definition (Template) Tags
a8239499 Image appears to be a copy of one or more articles, with some portions different or remixed

Draft typology of modifications

  • Cropping
  • Changing the lighting
  • Adding contrast
  • Change the colors
  • Color saturation
  • Merging images
  • Object or image has been removed or obscured

Signal: Quotes Extensively From Another Source

src

Ref Definition (Template) Tags
91491699 Text of [quotes extensively from another source, with some original content

(Section with no title?)

src

Attribution of Non-Original Image

src
  • Contains a hashtag

Subject type: Audio

src

Also called: Audio Clip, Sound Clip, Audio Recording

Audio type

src

Editor note: This should probably be abstracted to all different types of contents.

Signal: Audio type is news

src

Ref Definition (Template) Tags
18da5ac5 [Audio] appears to be news.

Signal: Audio type is opinion

src
Ref Definition (Template) Tags
e1f7bea2 [Audio] appears to be an opinion piece

Signal: Audio type is advertising or marketing.

src
Ref Definition (Template) Tags
25efaf85 [Audio] appears to be advertising or marketing.

Signal: Audio roles - host

src
Ref Definition (Template) Tags
bdde72c3 [Audio] has an in-studio host.

Signal: Audio roles - reporter

src
Ref Definition (Template) Tags
941fef55 [Audio] has a reporter.

Signal: Audio roles - members of the public

src
Ref Definition (Template) Tags
2afde61a [Audio] has interviews with members of the public.

Signal: Audio roles - experts and/or officials

src
Ref Definition (Template) Tags
4696bbb1 [Audio] has interviews with expert and/or official sources

Signal: Studio conversation

src
Ref Definition (Template) Tags
6b6d067f [Audio] has conversation between host and interviewee who is not a reporter.

Signal: Call-ins

src
Ref Definition (Template) Tags
28c8729a [Audio] has call-ins from members of the public.

Signal: Studio

src

Ref Definition (Template) Tags
8fe1900f [Audio] sounds like it was at least partially recorded in a studio.

Signal: Outside

src

Ref Definition (Template) Tags
ea839d7b [Audio] sounds like it was at least partially recorded outdoors.

Signal: Station/company identification

src
Ref Definition (Template) Tags
2e3c943a Station or company that produced the [audio] is identified.

Signal: Host/reporter identification

src
Ref Definition (Template) Tags
2d3a576f Host of [audio] identifies themselves.
52e6532c Reporter of [audio] identifies themselves

Signal: Quoted individuals are identified.

src

Ref Definition (Template) Tags
4b5d600a Individuals quoted in [audio] are identified by affiliation, if being quoted in a professional capacity.
dc425827 Individuals quoted in [audio] are identified by name.

Signal: Attribution

src

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.

Rhetoric

src

Signal: Proportional rhetoric

src

Editor: These should go to some category that includes both text and audio and video.   Linguistic content.

Ref Definition (Template) Tags
fe3b53d1 The rhetoric used in [audio] is proportional to the event or situation described.

Signal: Extreme Exaggerating Rhetoric

src

Ref Definition (Template) Tags
853a3706 The rhetoric used in [audio] is an extreme exaggeration of the event or situation described.

Signal: Extreme Minimizing Rhetoric

src

(Section with no title?)

src

Key

Proposed Template Statement

By

1

The rhetoric used in [audio] is an extreme minimization of the event or situation described.

Tamar Wilner, adapting Credibility Coalition

Emotional valence

src

Signal: Extremely negative valence

src

Ref Definition (Template) Tags
c11d9c4c The language of the reporter or main speaker in the [audio] is extremely negative.

Signal: Extremely positive valence

src

Ref Definition (Template) Tags
af0af713 The language of the reporter or main speaker in the [audio] is extremely positive.

(Section with no title?)

src

Signal: Neutral valence

src

Ref Definition (Template) Tags
88d6b211 The language of the reporter or main speaker in the [audio]  is neutral.

Sound quality

src

Signal: Clear speech

src

Ref Definition (Template) Tags
854f4591 There is audio distortion making speech in the [audio] difficult to understand.

Music

src

Signal: Emotional music

src

Ref Definition (Template) Tags
72ecf4dd The [audio] contains music that appears designed to manipulate listener emotions.

Signal: Loud music

src

Ref Definition (Template) Tags
1e4e78a0 The [audio] contains music loud enough to make speech difficult to hear.

Subject type: Video

src

Also called: Video Clip, Video Recording, Movie

Signal: Video type is news

src

Ref Definition (Template) Tags
18da5ac5 [Video] appears to be news.

Signal: Video type is opinion

src
Ref Definition (Template) Tags
e1f7bea2 [Video] appears to be an opinion piece

Signal: Video type is advertising or marketing.

src
Ref Definition (Template) Tags
25efaf85 [Video] appears to be advertising or marketing.

Signal: Station/company identification

src
Ref Definition (Template) Tags
2e3c943a Station or company that produced the [audio] is identified.
2e3c943a Station or company that produced the [video] is identified.

Ref Definition (Template) Tags
2e3c943a Station or company that produced the [audio] is identified.
2e3c943a Station or company that produced the [video] is identified.

Signal: Host/reporter identification

src
Ref Definition (Template) Tags
2d3a576f Host of [audio] identifies themselves.
2d3a576f Host of [video] identifies themselves.
52e6532c Reporter of [audio] identifies themselves
52e6532c Reporter of [video] identifies themselves

Ref Definition (Template) Tags
2d3a576f Host of [audio] identifies themselves.
2d3a576f Host of [video] identifies themselves.
52e6532c Reporter of [audio] identifies themselves
52e6532c Reporter of [video] identifies themselves

Signal: Quoted individuals are identified.

src

Ref Definition (Template) Tags
4b5d600a Individuals quoted in [audio] are identified by affiliation, if being quoted in a professional capacity.
dc425827 Individuals quoted in [audio] are identified by name.
4b5d600a Individuals quoted in [video] are identified by affiliation, if being quoted in a professional capacity.
dc425827 Individuals quoted in [video] are identified by name.

Ref Definition (Template) Tags
4b5d600a Individuals quoted in [audio] are identified by affiliation, if being quoted in a professional capacity.
dc425827 Individuals quoted in [audio] are identified by name.
4b5d600a Individuals quoted in [video] are identified by affiliation, if being quoted in a professional capacity.
dc425827 Individuals quoted in [video] are identified by name.

Signal: Attribution

src

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
31755cbb [Video] does not include attribution.

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
31755cbb [Video] does not include attribution.

Rhetoric

src

Signal: Proportional rhetoric

src

Editor: These should go to some category that includes both text and audio and video.   Linguistic content.

Ref Definition (Template) Tags
fe3b53d1 The rhetoric used in [audio] is proportional to the event or situation described.
fe3b53d1 The rhetoric used in [video] is proportional to the event or situation described.

Ref Definition (Template) Tags
fe3b53d1 The rhetoric used in [audio] is proportional to the event or situation described.
fe3b53d1 The rhetoric used in [video] is proportional to the event or situation described.

Signal: Extreme Exaggerating Rhetoric

src

Ref Definition (Template) Tags
853a3706 The rhetoric used in [audio] is an extreme exaggeration of the event or situation described.

(Section with no title?)

src

Key

Proposed Template Statement

By

1

The rhetoric used in [video] is an extreme exaggeration of the event or situation described.

Tamar Wilner, adapting Credibility Coalition

Signal: Extreme Minimizing Rhetoric

src

(Section with no title?)

src

Key

Proposed Template Statement

By

1

The rhetoric used in [video] is an extreme minimization of the event or situation described.

Tamar Wilner, adapting Credibility Coalition

Emotional valence

src

Signal: Extremely negative valence

src

Ref Definition (Template) Tags
c11d9c4c The language of the reporter or main speaker in the [audio] is extremely negative.
c11d9c4c The language of the reporter or main speaker in the [video] is extremely negative.

Ref Definition (Template) Tags
c11d9c4c The language of the reporter or main speaker in the [audio] is extremely negative.
c11d9c4c The language of the reporter or main speaker in the [video] is extremely negative.

Signal: Extremely positive valence

src

Ref Definition (Template) Tags
af0af713 The language of the reporter or main speaker in the [audio] is extremely positive.
af0af713 The language of the reporter or main speaker in the [video] is extremely positive.

Ref Definition (Template) Tags
af0af713 The language of the reporter or main speaker in the [audio] is extremely positive.
af0af713 The language of the reporter or main speaker in the [video] is extremely positive.

(Section with no title?)

src

Signal: Neutral valence

src

Ref Definition (Template) Tags
88d6b211 The language of the reporter or main speaker in the [audio]  is neutral.
88d6b211 The language of the reporter or main speaker in the [video] is neutral.

Ref Definition (Template) Tags
88d6b211 The language of the reporter or main speaker in the [audio]  is neutral.
88d6b211 The language of the reporter or main speaker in the [video] is neutral.

On-screen text

src

Relates to all on-screen text including chyrons, attributions…?

Signals for identification of quoted individuals; rhetoric; valence; what else?

CHYRON DISAGREEMENT WITH AUDIO

Images

src

Moving images

src

Video or film.

Signals for valence, what else?

Data graphics

src

Signal: Attribution

src

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
384bbffc [Video] data graphic does not include attribution.
31755cbb [Video] does not include attribution.

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
384bbffc [Video] data graphic does not include attribution.
31755cbb [Video] does not include attribution.

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
384bbffc [Video] data graphic does not include attribution.
31755cbb [Video] does not include attribution.

(Section with no title?)

src

Signal: Graph Y-axis does not start at zero.

src

Issue: Should graphic display of data get their own subject-category?  Charts?   Cf Tufte

Ref Definition (Template) Tags
5c73f2de [Video] data graphic y-axis starts at a number other than zero..

Still photography

src

Signal: Attribution

src

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
384bbffc [Video] data graphic does not include attribution.
31755cbb [Video] does not include attribution.
364044b4 [Video] still photography does not include attribution.

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
384bbffc [Video] data graphic does not include attribution.
31755cbb [Video] does not include attribution.
364044b4 [Video] still photography does not include attribution.

Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
384bbffc [Video] data graphic does not include attribution.
31755cbb [Video] does not include attribution.
364044b4 [Video] still photography does not include attribution.
Ref Definition (Template) Tags
bf3c76e5 [Audio] does not include attribution for the claims made.
384bbffc [Video] data graphic does not include attribution.
31755cbb [Video] does not include attribution.
364044b4 [Video] still photography does not include attribution.

Other graphics

src

Signals for rhetoric, valence, what else?

Music

src

Signals for valence and drama/exaggeration?

Subject type: Article

src

Includes: News Story, News Article, Scientific Paper, Blog Post

An article is a collection of information intended to convey some information, usually factual, usually created by one or more identifier people, and usually released at a specific point in time in some venue. It consists of elements like a body, a title, a publication date, and an author list.   Unlike Texts, where any change makes it a different Text, an Article may be revised over time and still be considered the same Article (albeit a different version).  Usually only minor changes are socially appropriate, however. Consumers of credibility data may need to be cautious of which version an observation applies to.

If an article appears on a web page, or in a portion of a web page, we can use its URL to identify the article.

Differentiation between Article and Text. Consider whether the signal data would be the same if the text were moved to a different article, perhaps published in a different venue, with a different title, at a different time, and with other text before or after it in some article. If the observation would be the same, then the signal is a property of the text, not the article. In that case it be in 3. Subject type: Text not here.  

Originality

src

Originality Types

src

Signal: Most Likely Original

src

Ref Definition (Template) Tags
58b71d6c Image is mostly likely original.
e595c4b2 Text of [subject article] is mostly likely original.

Ref Definition (Template) Tags
58b71d6c Image is mostly likely original.
e595c4b2 Text of [subject article] is mostly likely original.

Signal: Appears to be a Copy, with Some Different Portions

src

Ref Definition (Template) Tags
6acabe52 Text of [subject article] appears to be a copy of one or more articles, with some portions different or remixed

Signal: Quotes Extensively From Another Source

src

Ref Definition (Template) Tags
8001f76c Text of [subject article] quotes extensively from another source, with some original content
91491699 Text of [quotes extensively from another source, with some original content

Ref Definition (Template) Tags
8001f76c Text of [subject article] quotes extensively from another source, with some original content
91491699 Text of [quotes extensively from another source, with some original content

Signal: Wholesale Duplicate

src

Ref Definition (Template) Tags
69857b93 Text of [subject article] is a wholesale duplicate of another article

Attribution of Non-Original Content

src

These signals assume that the content has already been flagged as not original.

Signal: Attribution Given and Accurate[u]

src
Ref Definition (Template) Tags
9ce28cf4 [subject article] includes accurate attribution, pointing to the original.

Signal: Attribution Given and Inaccurate

src
Ref Definition (Template) Tags
334e1ec7 [subject article] includes inaccurate attribution.

Signal: Attribution Not Given

src
Ref Definition (Template) Tags
31755cbb [subject article] does not include attribution.

Signal: Unclear Which is Original

src
Ref Definition (Template) Tags
b1794319 [subject article] is a copy, but it is unclear which is the original.

Language and Rhetoric

src

To-do: Move Rhetoric to a different bucket, not Article.

Rhetorical Proportionality

src

Signal: Proportional Rhetoric

src

Ref Definition (Template) Tags
fe3b53d1 The rhetoric used in [Text] is proportional to the event or situation described.

Signal: Extreme Exaggerating Rhetoric

src

Ref Definition (Template) Tags
853a3706 The rhetoric used in [Text] is an extreme exaggeration of the event or situation described.
853a3706 The rhetoric used in [audio] is an extreme exaggeration of the event or situation described.

Ref Definition (Template) Tags
853a3706 The rhetoric used in [Text] is an extreme exaggeration of the event or situation described.
853a3706 The rhetoric used in [audio] is an extreme exaggeration of the event or situation described.

Signal: Extreme Minimizing Rhetoric

src
Ref Definition (Template) Tags
853a3706 The rhetoric used in [Text] is an extreme exaggeration of the event or situation described.

Signal: Emotional Valence

src

Could be measured by VADER (Valence Aware Dictionary and sEntiment Reasoner) Natural Language Processing library

Ref Definition (Template) Tags
d3f90e88 Is the language extremely negative, extremely positive, or somewhere in the middle? *** idea(cciv)

Signal: Extremely Negative Valence

src

Ref Definition (Template) Tags
7a192974 The language in [Text] is extremely negative.

Signal: Extremely Positive Valence

src

Ref Definition (Template) Tags
c07b1b5d The language in [Text] is extremely positive.

Signal: Neutral Valence

src

Ref Definition (Template) Tags
d8c8293c The language in [Text] is neutral.

Signal: Polarizing Language

src

Ref Definition (Template) Tags
d8e3739c [Text] uses language such as “pro” and “anti,” signaling a division into two sharply contrasting groups or sets of opinions or beliefs.

Developed for CredCo Political Indicators Study 2018-19. Taken from the Oxford Living Dictionary’s definition of polarization as the “division into two sharply contrasting groups or sets of opinions or beliefs.” Can be used in combination with 7.8. Claims in Articles.

Signal: Generalization/Characterization of Group

src

Ref Definition (Template) Tags
1a6fe948 [Text] in  [Content-Object] [v][w][x]characterizes a group or groups of people along lines that explicitly differentiate them from others.

Developed for CredCo Political Indicators Study 2018-19.This can apply to situations in which the author is associated with the defined group or defining an external group. Can be used in combination with 7.8. Claims in Articles and other “Content-Objects.”

Signal: Dehumanization

src

Ref Definition (Template) Tags
7fcf4da7 [Text] equates a human individual or group(s) as insects, bacteria, despised animals, cancer — less than human beings.

Developed for CredCo Political Indicators Study 2018-19.  See https://dangerousspeech.org/about-dangerous-speech/.

Signal: Exhortation

src

This signal is meant to capture exhortations, or “an address or communication emphatically urging someone to do something.”

Ref Definition (Template) Tags
9a77b9cf [Text] is an address that exhorts, or urges someone to do something.

Signal: Call to Violence

src

This signal is meant to capture a call to violence.  Perhaps also expressed as part of ‘Dangerous Speech’: “ any form of expression (speech, text, or images) that can increase the risk that its audience will condone or participate in violence against members of another group” (see https://dangerousspeech.org/about-dangerous-speech/).

Ref Definition (Template) Tags
b051c9b2 [Text] contains language that can be understood as a call to violence or seems harmful.

Developed for CredCo Political Indicators Study 2018-19.

Signal: Call to Action (Political)

src

This signal is meant to capture a textual call to action, not to be confused with a marketing call to action https://en.wikipedia.org/wiki/Call_to_action_(marketing). Sometimes, these calls to action are also associated with requests for enacting/executing an action as an expression of one’s loyalty, identity, or affiliation.

Ref Definition (Template) Tags
70dd369e [Text] contains language that can be understood as a political call to action, which requests readers to follow-through with a particular task, or tells readers what to do such as: signing online petitions, joining a mailing list, giving donations, voting, protesting, boycotting.

Developed for CredCo Political Indicators Study 2018-19.

Logic/Reasoning

src

Types of Bias

src

Signal: Confirmation Bias

src

Ref Definition (Template) Tags
596f4162 *** idea(cciv)
5b19acfa Text of [article title] contains examples of …..

Outbound References

src

Source Types

src

Signal: No Source Type Cited

src
Ref Definition (Template) Tags
bf2e6507 There is no source cited in [subject article].

Signal: Domain Expert Cited

src
Ref Definition (Template) Tags
af7ab75e There is an expert cited in [subject article].

Signal: Study Cited

src
Ref Definition (Template) Tags
bfe3b411 There is a study cited in [subject article].

Signal: Organization Cited

src
Ref Definition (Template) Tags
56ecd8ab There is an organization cited in [subject article].

Signal: Other Type of Source Cited

src
Ref Definition (Template) Tags
ccb6072c There is another type of sourced cited in [subject article].

Signal: Anonymous Sources Cited

src
Ref Definition (Template) Tags
5ab55dfc One or more anonymous sources are cited in [subject article].

Signal: Single Anonymous Sources Materially Cited

src
Ref Definition (Template) Tags
db18b042 A single anonymous source is materially cited in [subject article].

Would the interpretation of the article be substantively different without the single anonymous source.

Signal: Anonymous Sources Materially Cited

src
Ref Definition (Template) Tags
27096b57 One or more anonymous sources are materially cited in [subject article].

Signal: Multiple Anonymous Sources Materially Cited

src
Ref Definition (Template) Tags
70fff801 More than one anonymous sources are materially cited in [subject article].

Signal: Multiple Anonymous Sources are Cited in Corroboration for Information

src
Ref Definition (Template) Tags
d0c94f6a More than one anonymous sources are corroboratively cited in providing information for [subject article].

Is more than one source cited for information?

Signal: Motivation of Anonymous Source Wanted Anonymity is Given

src
Ref Definition (Template) Tags
69a1b877 The motivation of the anonymous sources to be anonymous is given in [subject article].

Signal: Documents are Cited in the Article

src
Ref Definition (Template) Tags
457405d2 Documents are cited in [subject article].

Signal: Documents Cited in the Article Are Made Available in Publication

src
Ref Definition (Template) Tags
5da755f8 Documents cited in [subject article] are also made available in publication.

Signal: Contains Link to Scientific Journals

src

Ref Definition (Template) Tags
fd96f7bf The text includes a link to original content idea(cciv) st(cciv)

Notes (not normative):

  • cciv: A simple link to the a scientific journal article that backs up the assertion made. This may also be paired with a URL to the specific article.
2a85d78f There is a link provided in [subject article] to where the original content came from.

Signal: Accuracy of representation of source article

src

Also called: Representative Citations

Ref Definition (Template) Tags
268b0193 The text properly characterizes the methods and conclusions of its sources idea(cciv) st(cciv)

Notes (not normative):

  • cciv: This article properly characterizes the methods and conclusions of the cited or quoted source. In addition to a Likert measure, two other options are possible: (A) Unable to find source, (B) Source is behind a paywall
890a77ac This article properly characterizes the methods and conclusions of the cited or quoted source (Source 1).
4bce9ba4 This article properly characterizes the methods and conclusions of the cited or quoted source (Source 2).
e60e0259 This article properly characterizes the methods and conclusions of the cited or quoted source (Source 3).
f2e110d3 [subject article] properly characterizes the methods and conclusions of the original source.

Signal: Academic Journal Impact Factor

src

Ref Definition (Template) Tags
18db0556 The impact factor of the journal or conference cited is [number].
059e6c34 What is the impact factor of the journal or conference cited? *** From Wikipedia: The impact factor (IF) or journal impact factor (JIF) of an academic journal is a measure reflecting the yearly average number of citations to recent articles published in that journal. It is frequently used as a proxy for the relative importance of a journal within its field; journals with higher impact factors are often deemed to be more important than those with lower ones. https://en.wikipedia.org/wiki/Impact_factor idea(cciv)

Signal: Academic Journal Impact Factor Cannot Be Found

src

Ref Definition (Template) Tags
8c312a70 The impact factor of the journal or conference cited cannot be found.

Article/Site Metadata

src

Signal: Subhed/Dek

src

Ref Definition (Template) Tags
a675ee6e A “dek” is a subhed in journalism that appears below the headline of an article, usually in a smaller font (but in a larger font than the main body of the article). It typically summarizes the article or highlights a main point from the article.

Claims in Articles

src

Although there is a separate section for Claims [2. Subject type: Claim], this section deals with the case when the analysis of one or more claims within an article is made to signify something about the article itself. [Probably could use an introductory paragraph on different levels/objects once those are clarified, since this translation is taking place for a number of projects, consider articles to domains/publishers.]

In the following signals, an assumption is made on the existence of a central claim of the article that is recognizable.

Signal: Article has a central claim

src

Ref Definition (Template) Tags
c871d2dc The central claim in [Article] is [Claim].[y]
001c97cb There is a central claim in [Article].

The first version of this signal was used in Credibility Coalition’s WebConf2018 study in which it was expressed as a question with multiple choice answers as follows:

Has the central claim in this article been fact-checked by an IFCN Verified Signatory?

A) Most likely not fact-checked by an IFCN Verified Signatory

B) Most likely not fact-checked by an approved source

C) Fact-checked and determined false

D) Fact-checked and determined true

E) Fact-checked with unclear results

F) Fact-checked with mixed results

It was initially deprecated due to the recognition of a number of valuable fact-checking efforts that are not IFCN Signatories, but then has remained with a change to its options as follows:

Does the article rely on a claim that has been fact-checked by a member of the International Fact Checking Network (IFCN)? If so, has it been debunked?

A) The article was fact-checked and determined false

B) The article was fact-checked and determined true

C) The article was fact-checked with unclear results

D) The article was fact-checked with mixed results

E) The article was most likely not fact-checked by an IFCN member

To express these questions as signals, combine with the signals related to fact-checking organization, see section 2.2. Fact-checking Organization and 2.1. Claim Review above.  [z][aa][ab][ac]

Signal: Article has a claim

src

Ref Definition (Template) Tags
b7f2d683 [Claim] is a claim in [Article].

Subject type: Title

src

Also called: Headline

A Title is an immutable association of an Article and some short Text which is typically presented first. When a Title text changes, that's a different Title. An article may have many different titles at different points in time and in different contexts, although social practice is usually against this.

The same title text may used for many different articles, but those are considered different Titles here.  For example, many of the signals about a Title with the text, “Local man dies”, will depend on which Article it's associated with.

Ref Definition (Template) Tags
edeccdeb [The article] has a title or headline. idea-GL(cciv)

Quality

src

These are general quality signals, not containing many details about why something might be high or low quality.

Clickbait

src

A measure of how much the title of the article conforms to a predetermined set of clickbait genres. See also specific signals below that might be considered kinds of clickbait, like “Listicle”.

Signal: Listicle

src

Ref Definition (Template) Tags
277ffe1c Title is a listicle (“6 Tips on …”)

Signal: Cliffhanger

src

Ref Definition (Template) Tags
35d6f6dd Title is a cliffhanger (“You Won’t Believe What Happens Next”, “Man Divorces His Wife After Overhearing This Conversation”)

Signal: Provoking emotions

src

Ref Definition (Template) Tags
40508eff Title provokes emotions (“...Shocking Result”, “...Leave You in Tears”)

Signal: Curiosity Gap (Hidden Secret or Trick)

src

Ref Definition (Template) Tags
ceff00a1 Title creates a curiosity gap through a hidden secret or trick (“Fitness Companies Hate Him...”, “Experts are Dying to Know Their Secret”, “You’ll never guess…”)

Signal: Challenge to the Ego

src

Ref Definition (Template) Tags
d3f1ce30 Title contains challenges to the ego (“Only People with IQ Above 160 Can Solve This”)

Signal: Defying Convention

src

Ref Definition (Template) Tags
257314fa Title defies convention (“Think Orange Juice is Good for you? Think Again!”, “Here are 5 Foods You Never Thought Would Kill You”)

Signal: Inducing Fear

src

Ref Definition (Template) Tags
9ba34544 Title induces fear (“Is Your Boyfriend Cheating on You?”)

Misleading about content

src

Signal: Title Representativeness

src

Ref Definition (Template) Tags
1c858afa Does the title of the article accurately reflect the content of the article? *** A measure of how representative the content of the title is with the content of the body copy. idea(cciv)
63bdbf87 Title is representative of the content of the article.

See also specific signals below that encapsulate what might be considered unrepresentative.

Signal: Differs from body topic

src

Ref Definition (Template) Tags
62c31a40 Title differs from the primary content of the article body.

Signal: Emphasizes different information than the body topic

src

Ref Definition (Template) Tags
5a2ef41f Title emphasizes different information from the primary content of the article body.

Signal: Carries little information about the body

src

Ref Definition (Template) Tags
142ae588 Title carries little information about the primary content of the article body.

Signal: Takes a different stance than the body

src

Ref Definition (Template) Tags
4ebbb75b Title takes a different stance from the primary content of the article body.

Signal: Overstates claims or conclusions in the body

src

Ref Definition (Template) Tags
4c0e31e7 Title over states claims or conclusions from the primary content of the article body.

6 Signal: Understates claims or conclusions in the body

src

Ref Definition (Template) Tags
44bff56c Title understates claims from the primary content of the article body.

Misleading about the world

src

Non-misleading consumer manipulation

src

(Section with no title?)

src

Subject type: Web Page

src

Layout

src

Issue: Should this be a Heading1 like Title?  Probably no, because the statements naturally get phrased with the subject of the statements being a web page. Most people wouldn't conceptualize the page layout as its own entity.

Signal: Framed with navigation

src

Also called: topnav, sidenav, framenav

Ref Definition (Template) Tags
7b21549b [subject] has a prominent top or side menu structure or buttons or links, taking user to other parts of site
b0b323b3 [subject] has obvious navigation elements at one or more edges of the content, providing a way to reach other content on the same website

(Consensus discussion including benefits and risks goes here)

(External data from studies and implementation reports gets inserted here, matched by heading text, “also called” text, and the template text.)

Typefaces

src

Metadata in transmission headers

src

Metadata in page head

src

Metadata inline in body

src

Subject type: Website

src

Markup

src

Signal: HTML Standard Version

src

Ref Definition (Template) Tags
8666b6f0 As of [date], the domain has implemented HTML [version number].
bf72c3b4 [website] uses features from HTML which did not appear until version [number] like(CSVDemo)

Notes (not normative):

  • CSVDemo: Which feature? Still pretty vague

Advertisements

src

Signal: Ads.txt Exists

src

Ref Definition (Template) Tags
74758d68 Does an ads.txt file exist on the domain? *** ads.txt (Authorized Digital Sellers) is an Interactive Advertising Bureau initiative. It specifies a text file that companies can host on their web servers, listing the other companies authorized to sell their products or services. This is designed to allow online buyers to check the validity of the sellers from whom they buy, for the purposes of internet fraud prevention. (from https://en.wikipedia.org/wiki/Ads.txt) idea(cciv)
363d7780 The domain contains an ads.txt file.

Signal: Spam or Clickbait Advertisements

src

Ref Definition (Template) Tags
45f891af How strongly do you agree or disagree that the page of the article has spammy or clickbaity advertisements? *** The page of the article has spammy or clickbaity advertisements. This is limited to a subjective assessment at this time. idea(cciv)
37896453 The page of the article has spammy or clickbaity advertisements. This is limited to a subjective assessment at this time.

Signal: Number of Advertisements

src

Ref Definition (Template) Tags
507e52c1 How many ads appears on the article page? *** There are multiple types of ads to look for. (1) Display ads. These are boxes that are clearly advertisements, typically in the form of a graphic image or, in the case of Google Adwords, a box with text. (2) Content recommendation engines, specifically, Taboola, Outbrain, Tivo, RevContent. A box of content recommendations on a page counts as one. (3) Sponsored content. This is content recommended on the site with a clear label: “Sponsored.” (4) Call for social sharing. (5) Call to subscribe to a mailing list idea(cciv)
4168b252 The number of ads that appear on [subject article] is [number]. This includes display ads, content recommendation engines, sponsored content and call for social sharing

Signal: Aggressive Advertisements

src

Ref Definition (Template) Tags
94cd8f08 How strongly do you agree or disagree that the page of the article has aggressive advertisements, including calls to join a mailing list? *** The page of the article has aggressive advertisements. This is limited to a subjective assessment at this time. idea(cciv)
efcfda72 The page of the article has aggressive advertisements. This is limited to a subjective assessment at this time.

Signal: There is false advertising

src

Ref Definition (Template) Tags
bc643d60 The domain contains advertisements for false products.

Identification

src

Signal: ‘About Us’ page exists

src
Ref Definition (Template) Tags
91f47d8c The domain has a clear 'About Us' page with details and description about itself.

Signal: Contact information exists

src
Ref Definition (Template) Tags
39e08bab The domain has provided means to contact staff.[ae]

Subject type: Aggregation

src

Includes: news feed, content portal, site using syndicated content

 

An aggregation is a collection of content from other providers.  As such, attribution and related trust issues require special consideration.

Subject type: Venue

src

A Venue is a branded content channel, which might be separable from the provider of that channel.  For instance a particular newspaper’s “Lifestyle” and “Sports” sections would typically be considered distinct Venues with distinct reputations. In this case, they would be sub-Venues of the newspaper itself.

The distinction between Venue and Provider is not always clear in people’s mind; it is essentially the distinction between a brand and the brand’s owner.  We try to make the distinction in order to be able to understand the impacts on reputation when, for example, one company sells a content brand to another company.

Subject type: Provider

src

Subject type: Creator

src

Also called: author, writer, reporter, byline

A Creator (in this context) is a Person is a person who creates content, such as by writing articles. All the signals which apply to a Person also apply to a Creator.  Signals are listed here if they only really make sense for content creators.  Even if every Person was a Creator, this grouping could still be convenient.

Subject type: Person

src

Good Faith

src

Self-Assessment

src

Use of “ethos”

Domain Expertise

src

Affiliation

src

History

src

Other

Subject type: Organization

src

Subject type: Testing

src

Using this section to help migrate over signals from other spaces.

To Be Categorized

src

Signal: Attribution of Non-Original Content

Ref Definition (Template) Tags
038b0a36 If the content of the article is not original, was attribution given and if so, was the attribution accurate? *** (A) Attribution was not given (B) Attribution was given but was inaccurate (C) Attribution was given and was accurate (D) Unclear which is the original idea(cciv)

Signal: Is Original

Ref Definition (Template) Tags
9721ce9a Has the text of this article appeared in exactly the same words or very similar words in another publication? *** (A) Most likely original (B) Appears to be a copy of one or more articles, with some portions different or remixed (C) Extensive quoting from another source, with some original content (D) A wholesale duplicate of another article idea(cciv)

Signal: Publication (site)

Ref Definition (Template) Tags
07de125b *** Parent of the Article. idea(cciv)

Signal: Dateline (Date)

Ref Definition (Template) Tags
656673f8 When does the article claim it was published? *** From An: "i think technically this states both the date and location of the article’s publication. in which case it needs two forms of operationalization idea(cciv)

Signal: Author

Ref Definition (Template) Tags
d397cb76 The article has a byline identifying an author or authors. idea-GL(cciv)

Notes (not normative):

  • cciv: Articles 1:M authorship. Note that not all articles have bylines, even in traditional news sources. Bylines also don't always start with 'by' (https://medium.com/@rchang/advice-for-new-and-junior-data-scientists-2ab02396cf5b)

Signal: Length

Ref Definition (Template) Tags
5b04d907 [The article] contains [#] words. idea-GL(cciv)

Notes (not normative):

  • cciv: Need to count the words in the article.

Signal: Language

Ref Definition (Template) Tags
09798548 What language(s) does the publication publish in? *** idea(cciv)
14087209 [The article] language is [language]. idea-GL(cciv)

Notes (not normative):

  • cciv: Needs a tool that identifies the language.

Signal: Correction/Redaction

Ref Definition (Template) Tags
1d49cd64 The article contains a stated correction or redaction. idea-GL(cciv)

Signal: Article Awards

Ref Definition (Template) Tags
c56851d7 The article identifies awards it or the website has received. idea-GL(cciv)

Notes (not normative):

  • cciv: *** Awards are also assigned to specific Articles (but rarely)

Signal: Article Locator

Ref Definition (Template) Tags
38d79b22 *** URL, DOI, idea(cciv)

Signal: Genre

Ref Definition (Template) Tags
ce682f01 What is the state genre, if available? *** Opinion, Feature, Biography -- but it will not always be labeled and i am not sure this will be defined consistently idea(cciv)

Signal: Dateline (Location)

Ref Definition (Template) Tags
7459cd99 Where does the article claim it was published? *** idea(cciv)

Signal: Is Translation

Ref Definition (Template) Tags
5874c92d Is the article a translation? *** idea(cciv)

Signal: Source Language (if translation)

Ref Definition (Template) Tags
28eb8320 If the article is a translation, what is the source language? *** idea(cciv)

Signal: Factual assertions

Ref Definition (Template) Tags
1562b9d1 Does the article contain factual, verifiable assertions or is entirely opinion based? *** Complements the Genre (is the content actually opinion piece? or verifiable information?) idea(cciv)

Signal: Subject Area

Ref Definition (Template) Tags
0ffe6939 What is the article's genre? *** Indicates a subject of an item: Sports, Entertainment idea(cciv)

Signal: Shows versions and changes

Ref Definition (Template) Tags
269464a3 Does this article show revisions/diffs? (Most places do not) *** Shows versions, changes, diffs of an article. idea(cciv)

Signal: Subhead/Dek

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Publication Domain Registration Date

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Publication Domain Registration Location

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Article Rights

Ref Definition (Template) Tags
c400e7d6 What are the rights for this article? *** Explicit Copyright, Creative Commons, Unstated, etc. idea(cciv)

Signal: Photo/Video Geotags

Ref Definition (Template) Tags
1a6a039b What are the geolocations for the photos and videos on the article? *** idea(cciv)

Signal: Occupation

Ref Definition (Template) Tags
89df0693 What is the occupation of the author? *** What is the occupation of the author. Are they a full-time journalist or something else idea(cciv)

Signal: Education Credentials

Ref Definition (Template) Tags
ce8ba291 What is the educational background of the author? *** What is the educational background of the author. Do they have a degree in journalism? Do they have any post-grad education? idea(cciv)

Signal: Track record

Ref Definition (Template) Tags
0bbd02f7 Has the author already published articles containing misleading or credible information? *** idea(cciv)

Signal: Number of publications

Ref Definition (Template) Tags
5203fc59 How many publications does this author have and in which venues? *** How many publications does this author have and in which venues. idea(cciv)

Signal: Public Accessibility

Ref Definition (Template) Tags
e6d13230 How accessible/responsive is this author to the public? *** How accessible is this author to the public. Do they have a website? Does this author have a publicly available email address? Are they on Twitter or Facebook? Do they often respond to readers? idea(cciv)

Signal: Followers/Listeners

Ref Definition (Template) Tags
34b38463 What degree of attention does this author command from other individuals? *** How many people follow this author (on social media). How many other journalists follow this author? What speaking engagements does this author command? idea(cciv)

Signal: Has Author Bio

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Claim contains logical fallacy

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Claim contains false assertion

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Claim contains false and misleading assertion

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Claim contains misleading assertion

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Claim contains bad data

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Links from other news sites

Ref Definition (Template) Tags
6c269cbf What are the 5 most popular (Alexa rating) news sites linking to this article? *** What other news sites are linking to this URL? Does it include credible sites or not? idea(cciv)

Signal: Facebook shares

Ref Definition (Template) Tags
aca0b2b3 How many facebook shares? *** Number of times shared on facebook. Important to estimate the reach of the news article. idea(cciv)

Signal: Facebook engagement

Ref Definition (Template) Tags
73dbfb79 How many people engaged on facebook? *** Number of facebook engagement. Important to estimate the reach of articles. idea(cciv)

Signal: Ratio of comments to likes <10%

Ref Definition (Template) Tags
21c9abae Does it have many more likes than comments? *** Ratio of comments to likes on Facebook. Indicative of bot-promoted content. idea(cciv)

Signal: Links from social media

Ref Definition (Template) Tags
915c563f What are the 5 most liked (Twitter, Facebook), upvoted (Reddit) social accounts that shared this link *** What social media accounts have shared the URL? Do they include credible accounts? idea(cciv)

Signal: Facebook comment quality

Ref Definition (Template) Tags
0113f86a All or nearly all the Facebook comments are mono-sentence. idea(cciv) j8l(cciv)

Notes (not normative):

  • cciv: *** Number and content of facebook comments. Important to estimate the reach and sentiment of articles.

Signal: Facebook comment quantity

Ref Definition (Template) Tags
1fa2c4b8 There are 10 or fewer Facebook comments idea(cciv) j8l(cciv) new addition(cciv)

Signal: Facebook comment sentiment

Ref Definition (Template) Tags
4104d376 What is the sentiment of the facebook comments? *** Sentiment analysis on Facebook comments. Important to gauge the attitudes towards the news, and types of reactions they invoke. idea(cciv)

Signal: Number of links in Wikipedia

Ref Definition (Template) Tags
c9af8b4d Multiple Wikipedia articles point to the article idea(cciv) st(cciv)

Notes (not normative):

  • cciv: Number of inbound links from Wikipedia main namespace.

Signal: Represents scientific literature

Ref Definition (Template) Tags
e7776710 The content fairly represents scientific literature on the issue at hand idea(cciv) st(cciv)

Signal: Represents scientific process

Ref Definition (Template) Tags
f9fc7eb3 The content rigorously represents the scientific process idea(cciv) st(cciv)

Signal: Presents multiple perspectives

Ref Definition (Template) Tags
7f4d0899 The content fairly represent multiple perspectives on an issue idea(cciv) st(cciv)

Signal: Cites wire services

Ref Definition (Template) Tags
b4fafa09 The article clearly states a known wire service as a source idea(cciv) st(cciv)

Signal: Dependency on anonymous sources

Ref Definition (Template) Tags
2c269b10 THe premise of this article is based on anonymous sources. idea(cciv) ng(cciv)

Notes (not normative):

  • cciv: This can be made more nuanced depending on how many sources, and how critical it is to the main premise of the article.

Signal: Gives motivation of anonymous sources for revealing information

Ref Definition (Template) Tags
1bc9cc79 This article explains the motivation for all of the anonymous sources revealing information. idea(cciv) og(cciv)

Signal: Shares any documents cited in piece

Ref Definition (Template) Tags
3ea072fe This article publishes (or links to?) any important documents cited within it. idea(cciv) ng(cciv)

Notes (not normative):

  • cciv: Does this refer to inlinks within the article? or if the creators of the article only cites articles they themselves publish

Signal: Primary subjects of article have opportunity to respond

Ref Definition (Template) Tags
50dbc854 The main subjects of the article have a chance to respond (to the main points of the artice? in a direct or indirect quote after being interviewed?) idea(cciv) ng(cciv)

Notes (not normative):

  • cciv: Not sure if the responses metric refers to responses within the article, or responses via commenting

Signal: Premise of article is disputed by primary subjects

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Corroboration?

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Straw Man Argument

Ref Definition (Template) Tags
7e90dbc5 The [text] presents a counterargument as a weaker, more foolish version of the real counterargument (uses a Straw Man Argument). idea(cciv) sh(cciv)

Signal: False Dilemma

Ref Definition (Template) Tags
e0958f51 The [text] presents a complicated choice as if it were binary (constructs a false dilemma). idea(cciv) og(cciv)

Signal: Slippery Slope Argument

Ref Definition (Template) Tags
aec147ca The text says that one small change will lead to a major change idea(cciv) st(cciv)

Notes (not normative):

  • cciv: Slippery slope argument.

Signal: Naturalistic Fallacy

Ref Definition (Template) Tags
534e79cd The [text] suggests that something is good because it is natural, or bad because it is not natural (the naturalistic fallacy). idea(cciv) og(cciv)

Signal: Calibrating Confidence - Justification

Ref Definition (Template) Tags
98495085 The author's confidence in their claims is well justified idea(cciv) og(cciv)

Signal: Appeal to Fear Fallacy

Ref Definition (Template) Tags
a5e90858 The [text] exaggerates the dangers of a situation and uses scare tactics to persuade (uses the appeal to fear fallacy). DK(cciv) idea(cciv)

Signal: Calibrating Confidence - Level of Confidence

Ref Definition (Template) Tags
c2385f39 The [text] acknowledges uncertainty or the possibility that things might be otherwise (expresses level of confidence in a claim). DK(cciv) idea(cciv)

Signal: Causal Claim Types

Ref Definition (Template) Tags
527f60d7 Is a general or singular causal claim made? Highlight the section(s) that supports your answer. *** General Causal Claim Singular Causal Claim No Causal Claim idea(cciv)

Signal: Draws sound conclusions from available evidence

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Begging the Question

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Mistaking Noise for Signal

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Orders of Understanding

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Number of Enthymemes (Arguments with Missing Premises)

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Numbers of Argument Components

Ref Definition (Template) Tags
6d3d2db5 The argumentation and logic in the article is complex idea(cciv) st(cciv)

Signal: Number of Claims

Ref Definition (Template) Tags
ffc5d663 How far could this article go wrong? *** idea(cciv)

Signal: Number of Arguments Against

Ref Definition (Template) Tags
5f5ead5b the text is biased against one set of premises idea(cciv) st(cciv)

Signal: Number of Supporting Premises

Ref Definition (Template) Tags
8c6efeeb The article backs up arguments with clear hypotheses idea(cciv) st(cciv)

Signal: Average Number of Premises Per Claim

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Number of Arguments For

Ref Definition (Template) Tags
a1edbdbe the text is biased towards one set of premises idea(cciv) st(cciv)

Signal: Number of Attacking Premises

Ref Definition (Template) Tags
f2ee8607 The text strongly contradicts itself idea(cciv) st(cciv)

Signal: Use of conspiratorial thinking

Ref Definition (Template) Tags
a857394f The article suggests a conspiracy idea(cciv) st(cciv)

Signal: Identifiable Victim Effect

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Just World Fallacy

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Supporting Claim Types

Ref Definition (Template) Tags
0728c3c4 What evidence is given for the primary claim? Select all that apply. *** Types of evidence for a claim: Correlation Cause precedes effect The correlation appears across multiple independent contexts A plausible mechanism is proposed An experimental study was conducted (natural experiments OK) Experts are cited Other kind of evidence No evidence given idea(cciv)

Signal: Source Types

Ref Definition (Template) Tags
5fa3c065 Which of the following types of sources are cited in the article? Check all that apply. If Other, please highlight. *** None Experts Studies Organizations Other idea(cciv)

Signal: Quotes reputable scientists

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Databases

Ref Definition (Template) Tags
1e255af1 *** Databases as a 'Publication' owned by a Publisher idea(cciv)

Signal: Agencies for Authority

Ref Definition (Template) Tags
db742e34 *** I think that Agencies could also be considered Publishers. But Publisher captures I think the relationship to the Article idea(cciv)

Signal: Quotes outside experts

Ref Definition (Template) Tags
c6a45cc1 Does the article quote experts who are not part of the study but are part of the field? *** idea(cciv)

Signal: Contains original quotes

Ref Definition (Template) Tags
aec59fc7 The article contains original quotes that appear to be sourced directly by the reporter. DK(cciv) idea(cciv)

Signal: Contains Image Macros

Ref Definition (Template) Tags
3648b6fc The article contains content with image macros. DK(cciv) idea(cciv)

Signal: Number of links

Ref Definition (Template) Tags
4f7edf95 How many URLs does the article link out to? *** idea(cciv)

Signal: Number of quoted sources

Ref Definition (Template) Tags
7113e6b1 How many sources does the article quote? *** idea(cciv)

Signal: Contain Original Images

Ref Definition (Template) Tags
8f88a1f5 The [content] contains original images. DK(cciv) idea(cciv)

Signal: Contains Attributed images

Ref Definition (Template) Tags
ed994261 The [content] contains images attributed to a photographer or other source. DK(cciv) idea(cciv)

Signal: Contains Video Embeds

Ref Definition (Template) Tags
686014f0 Does the article embed content from video sites? *** List of embeddable video sites (YouTube, Vimeo, etc.) idea(cciv)

Signal: Trust Project Metadata

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Clear Editorial Policy

Ref Definition (Template) Tags
c30585d3 The website includes a clear editorial policy idea(cciv) st(cciv)

Signal: Masthead (Nameplate)

Ref Definition (Template) Tags
b385555d The publication includes a masthead or nameplate idea(cciv) st(cciv)

Signal: Awards

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Publication End Date

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Publication Start Date

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Publication Type

Ref Definition (Template) Tags
de8b889e *** One of the following: Media Outlet, Governmental Agency, Non-profit, Private Corporation idea(cciv)

Signal: Publication Identifier

Ref Definition (Template) Tags
3ef5b5e0 *** Publishers may have 1:M Publications idea(cciv)

Signal: Publisher (Publication Owner)

Ref Definition (Template) Tags
920ca1c1 What is the name of the person or organization that published the publication under review? *** Publishers: Individuals or group entities -- see Publisher Table for more related attributes idea(cciv)

Signal: Publication Name

Ref Definition (Template) Tags
4fc9bd5f What is the name of the magazine, newspaper, journal, book that the article appeared within? *** Publications are parents of articles. Publishers may have 1:M Publications. Publications newsline magazines, series, newspapers, etc. idea(cciv)

Signal: Publication CMS

Ref Definition (Template) Tags
a0a8fb0a What is the CMS that the publication uses? *** idea(cciv)

Signal: Niche Topic

Ref Definition (Template) Tags
c24fc749 Is the publication focused on a niche topic? If so, what is the topic? *** idea(cciv)

Signal: Publication Domain

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Masthead (Imprint)

Ref Definition (Template) Tags
f6387d51 Does the publication have a clear and logical masthead/imprint? *** idea(cciv)

Signal: Links to Relevant Articles

Ref Definition (Template) Tags
402ca152 Does the publication regularly link to other relevant articles on its own site? *** idea(cciv)

Signal: Site Analytics

Ref Definition (Template) Tags
a32bb352 Does the publication use an analytics platform? Which one? *** idea(cciv)

Signal: Has a Wikipedia Entry

Ref Definition (Template) Tags
86edc186 Does the publication have an entry in Wikipedia? *** idea(cciv)

Signal: Working Phone Number

Ref Definition (Template) Tags
d91ed958 Is there a phone number and does calling the phone number lead someone to a representative of the publication? *** idea(cciv)

Signal: Is Part of Press Corps

Ref Definition (Template) Tags
43056b49 Is the publication part of the press corps? *** idea(cciv)

Signal: Average time spent on page

Ref Definition (Template) Tags
d8b18fd6 Average user spends [number] seconds on [webpage]. idea(cciv)

Notes (not normative):

  • cciv: (needs to go under Website) Dwell Time: Time a user spends on a page? Objectively captured by web logs. Sum of times / number of users. Note, this cannot identify the time spent actually reading the article.

Signal: Common Referrers

Ref Definition (Template) Tags
d158ef87 What is driving traffic to this article? *** From where did readers come from to read this article. Social media/unknown/news org page/search result page. idea(cciv)

Signal: Volume of readership over time

Ref Definition (Template) Tags
3ca839fe What is the nature of the volume of readership over time? *** What is the nature of the volume of readership over time. Is it spiky? How many spikes are there? Is there a long tail? idea(cciv)

Signal: Emotional comment shared in response

Ref Definition (Template) Tags
62f91575 Did the article provoke emotional -- positive or negative -- comments and response (in the discussion space around its content)? *** The direction and weight of reader interaction with this article. idea(cciv)

Signal: Aggressive Advertisements or Social Shares

Ref Definition (Template) Tags
08f317b3 How strongly do you agree or disagree that the page of the article has aggressive advertisements? *** The page of the article has aggressive advertisements. This is limited to a subjective assessment at this time. idea(cciv)

Signal: Presence of Donors

Ref Definition (Template) Tags
2a7d1df2 The site donors are clearly stated and reputable idea(cciv) st(cciv)

Signal: Presence of Paywall or Subscription

Ref Definition (Template) Tags
3453d15b The site contains a paywall or subscription idea(cciv) st(cciv)

Notes (not normative):

Signal: Presence of Sponsors

Ref Definition (Template) Tags
6b2f930b The article sponsors are clearly stated and reputable idea(cciv) st(cciv)

Signal: Presence of Freemium Content

Ref Definition (Template) Tags
c90707a8 The site contains selectively free content outside of its paywall idea(cciv) st(cciv)

Signal: Top Call to Action for Donations

Ref Definition (Template) Tags
ea4cfc89 The site or article has a topline call to action for donations idea(cciv) st(cciv)

Signal: Aggressive Social Shares

Ref Definition (Template) Tags
f4b0f415 How strongly do you agree or disagree that the page of the article has aggressive social shares? *** The page of the article has aggressive social shares, which may include calls to share the article within the text. This is limited to a subjective assessment at this time. idea(cciv)

Signal: Emotionally Charged Tone

Ref Definition (Template) Tags
15fff94d Does the article have an emotionally charged tone? (i.e, outrage, snark, celebration, horror, etc.). If so, highlight the relevant section(s). *** Article has an emotionally charged tone idea(cciv)

Signal: Clickbait Headline

Ref Definition (Template) Tags
1aa19828 Is the headline clickbaity? *** A measure of how much the title of the article conforms to a predetermined set of clickbait genres. idea(cciv)

Signal: Title Representativeness Types

Ref Definition (Template) Tags
c11c62b6 How is the title unrepresentative of the content of the article? (Select all that apply). *** Types of title representativeness (A) Title is on a different topic than the body (B) Title emphasizes different information than the body (C) Title carries little information about the body (D) Title takes a different stance than the body (E) Title overstates claims or conclusions in the body (F) Title understates claims or conclusions in the body idea(cciv)

Signal: Clickbait Genres

Ref Definition (Template) Tags
c43d0138 What clickbait techniques does this headline employ (select all that apply)? *** A typology of clickbait headlines: Listicle (“6 Tips on …”) Cliffhanger to a story (“You Won’t Believe What Happens Next”, “Man Divorces His Wife After Overhearing This Conversation”) Provoking emotions, such as shock or surprise (“...Shocking Result”, “...Leave You in Tears”) Hidden secret or trick (“Fitness Companies Hate Him...”, “Experts are Dying to Know Their Secret”) Challenges to the ego (“Only People with IQ Above 160 Can Solve This”) Defying convention (“Think Orange Juice is Good for you? Think Again!”, “Here are 5 Foods You Never Thought Would Kill You”) Inducing fear (“Is Your Boyfriend Cheating on You?”) Other idea(cciv)

Signal: Exaggerated Claims

Ref Definition (Template) Tags
43d71ea6 1.21 mc Does the author exaggerate any claims? If so, highlight the relevant section(s). *** Claims are exaggerated, as indicated by the tone idea(cciv)

Signal: Reading Level

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Politicizing Tone

Ref Definition (Template) Tags
8297892a Does the content of the article politicize an issue unrelated to politics? *** idea(cciv)

Signal: Contains Profanity

Ref Definition (Template) Tags
8b9ac8e1 Does the article contain profanity? *** idea(cciv)

Signal: Grammatical Rules

Ref Definition (Template) Tags
9f83797e Does the article follow rules of US or UK English grammar? *** idea(cciv)

Signal: Spelling Errors

Ref Definition (Template) Tags
3bd39eaf How many spelling errors does the article have? *** idea(cciv)

Signal: Number of Exclamation Points

Ref Definition (Template) Tags
74b754a2 How many exclamation marks appear in the article? *** idea(cciv)

Signal: Contains Hyperbolic Language

Ref Definition (Template) Tags
d0513609 Does the article contain hyperbolic language? *** idea(cciv)

Signal: Hyperpartisanship / Political bias

Ref Definition (Template) Tags
23018043 *** Extreme political bias, e.g. unconstructive political discourse idea(cciv)

Signal: Astroturfing

Ref Definition (Template) Tags
17f1269a *** the practice of the masking sponsors of a message idea(cciv)

Signal: Overly Emotional Language

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Exaggeration

Ref Definition (Template) Tags
596f4162 *** idea(cciv)

Signal: Hate speech

Ref Definition (Template) Tags
0a580d75 *** Usage of abusive or toxic language, e.g. racism, sexism, etc. idea(cciv)

Signal: Apophasis

Ref Definition (Template) Tags
50bb076a *** A rhetorical device wherein the writer brings up a subject by either denying it, or denying it should be brought up idea(cciv)

Signal: Cognitive Distortion

Ref Definition (Template) Tags
88d4a6cf *** (the representation of something in an excessive manner) idea(cciv)

Signal: Proportion (Exaggeration - Minimization Spectrum)

Ref Definition (Template) Tags
e3876cec Is the description an extreme exaggeration, an extreme minimization, or proportional to the event or situation described? *** The extent to which language in the text is proportional to the situation, or exaggerates or minimizes events. Exaggeration as defined in Webster 1913, "...the act of doing or representing in an excessive manner;..." and Minimization as defined in Oxford Living Dictionary accessed June 2018, "1.1 Represent or estimate at less than the true value or importance" idea(cciv)

Signal: HTML Usage

Ref Definition (Template) Tags
a65e3e73 [website] front page uses syntactically valid HTML sometag(CSVDemo)