Minutes: Credible Web CG SUBGROUP 1 on Inspection (02 July 2018)


  1. update on credco
  2. inbound links
  3. future meetings



# [Sandro Hawke] Agenda-ish

# [An Xiao Mina] For our meeting:

# .. minutes seem to be missing some named indicators

# .. I’ll add before we approve

# .. contains ads.txt, group agreed it’s useful being so nascent

# .. credco study will use that

# .. aggressive social shares, distinct from aggressive ads,

# .. example: in body content

# .. finally: agressive ads, popups, follow-mouse, full-screen

# .. maybe not most useful but worth studying

update on credco

# .. this week we’re kicking off the credco study

# .. 30-50 article

# .. ad indicators, citation indicttors, and som eothers out of consultation

# .. 6 annotators

# .. get more sense of annotation issues

# .. Q: what constitutes scientific validation?

# .. we’ll be correlating indicators for gold std, and IRR

# .. but we haven’t discussed what the bar should be, as a community

# Sandro Hawke says: my sense is that it depends on what one is going to do with it

# .. nobody would claim these indicators are going to tell you with high confidence that a random article is true or not. they’re validated for a particular kind of article, and even then - what would we do with an indicator that crosses bar?

# Sandro Hawke says: I’m inclined to keep a confidence score with each indicator. Not sure each indicator could go beyond that, i.e., one that tells you with high levels of confidence

# Davide Ceolin says: There should also be confidence score… some indicators are much more deterministic, easier to compute – like presence of ads.txt – but eval aggressiveness of document, that’s much more vague. So level of confidence within the indicator.

# Sandro Hawke says: I think that shows in the IRR

# Davide Ceolin says: The strength in an indicator might show in how it plays with other indicators

# .. I think that would need a bigger data set

# Sandro Hawke says: sounds like IRR validates each, but relation to truth might not be visible as a solo indicator

# An Xiao Mina says: even if individual studies don’t show results, later studies on same data might pull things out

# .. and different table when we get more studies, eventually

# .. for ads.txt we’re not using humans, so no real IRR

# Sandro Hawke says: I cant think of strong reasons to do it one way or another. Might be good to note if an indicator requires no human judgment.

# .. for a yes/no question and all humans agree, IRR = 1. But that’s different from having machines test

# An Xiao Mina says: : some things machine be machine checkable, like emotional tone.

# Sandro Hawke says: I think Easy-Machine-Check is different from this ML-type checking which we’ll have more and more of

# An Xiao Mina says: subset of extrinsic indicators

# .. things that require stepping outside the content

# .. previously we talked about how theoretically they are more difficult to game

# .. requires link-farm type techniques to game

# .. fact-check-by-IFCN member, but hard to check

# .. want to go beyond IFCN

# .. eg CDC makes assessment

# .. looking for broader typology of inbound links

# [An Xiao Mina]

# Sandro Hawke says: for intrinsic indicators, we could theoretically train systems to detect these over time

# .. a fact check out there is a completely different game - it’s a matter of determining if content makes the same claim as a fact check

# .. computational linguistics challenge, not a machine learning thing

# .. wouldn’t expect results to be feeding into weighting algorithm in nearly the same way, weight on this would be so much higher than other kinds of indicators

# .. you could align a claimreview with a source that’s trusted

# .. IIUC, FB uses claim data from partners, and Google doesnt exactly use IFCN. they have their own computation about whether a site is reliable. will accept claimreview from non IFCN members in theory. not saying IFCN is exactly the gold standard. very high signal into legitimacy

# An Xiao Mina says: I can see how some of this might fall under area-2, under claims

# .. is it about Claims, or all inbound

# Sandro Hawke says: : I think only Claims go into Area-2, and inbound links from wikipedia go in Area-3 Reputation

# An Xiao Mina says: that’s a thing you could code up

# .. inbound links for domains, as well as pages

# Sandro Hawke says: this sounds like reputation logic

# .. the kind of code you’d use, not human readers

# .. are there extrinsic indicators you do want to have the reviews look at?

# An Xiao Mina says: Verdict from FC website, or Verdict from high reputation source, where students manually go look

# .. ratio of comments to likes (on facebook), linkes from social media, links from other news sites, FB shares and engagements

# .. not likely to ask students to do, more automation-friendly

# .. also FB comment sentiment

# Davide Ceolin says: how sources refer to other sources, inbound links might be positive or negative mention, sentiment of the mention

# Sandro Hawke says: could be done with human reviewers seeing a snip of the mention; maybe sentiment analysis code is good enough

# Sandro Hawke says: of course on twitter as a human I can’t tell what’s sarcastic

# An Xiao Mina says: there is literature around this

# .. is this under reputation?

# An Xiao Mina says: The conversation around an article, vs the conversation about a source

# Sandro Hawke says: not exactly reputation. if one of my friends says the NYT sucks, that’s clearly reputation. if someone says an article in the NYT sucks, that’s not really reputation

# .. just one inbound link with content about it, analysis of an article, that’s complicated

# Sandro Hawke says: : maybe this is another area, because it’s hugely useful, but doesn’t exactly line up with existing articles

# An Xiao Mina says: : Maybe this is another credco study

# Sandro Hawke says: some kind of study where you show ten tweets linking to article

# .. my sense is given rep of twitter feeds that’d be good

# .. sounds like some inbound belongs more the ClaimReview

# .. some about reputation

# .. and one that could be a good study

future meetings

# [Sandro Hawke] no Monday meeting next week, conflict, but week after is good

# Sandro Hawke says: thinking about scoping of this meeting as Area 1 meeting

# [Sandro Hawke] so 2 weeks, Monday, go through other Intrinsic indicators and think about them as a group

# [Sandro Hawke] adjourned

# [Cheryl Langdon-Orr] Present +