# [Sandro Hawke] Agenda-ish https://credweb.org/agenda/20180702.html
# [An Xiao Mina] For our meeting: https://credweb.org/cciv/
# .. minutes seem to be missing some named indicators
# .. I'll add before we approve
# .. contains ads.txt, group agreed it's useful being so nascent
# .. credco study will use that
# .. aggressive social shares, distinct from aggressive ads,
# .. example: in body content
# .. finally: agressive ads, popups, follow-mouse, full-screen
# .. maybe not most useful but worth studying
# .. this week we're kicking off the credco study
# .. 30-50 article
# .. ad indicators, citation indicttors, and som eothers out of consultation
# .. 6 annotators
# .. get more sense of annotation issues
# .. Q: what constitutes scientific validation?
# .. we'll be correlating indicators for gold std, and IRR
# .. but we haven't discussed what the bar should be, as a community
# Sandro Hawke says: my sense is that it depends on what one is going to do with it
# .. nobody would claim these indicators are going to tell you with high confidence that a random article is true or not. they're validated for a particular kind of article, and even then - what would we do with an indicator that crosses bar?
# Sandro Hawke says: I'm inclined to keep a confidence score with each indicator. Not sure each indicator could go beyond that, i.e., one that tells you with high levels of confidence
# Davide Ceolin says: There should also be confidence score... some indicators are much more deterministic, easier to compute -- like presence of ads.txt -- but eval aggressiveness of document, that's much more vague. So level of confidence within the indicator.
# Sandro Hawke says: I think that shows in the IRR
# Davide Ceolin says: The strength in an indicator might show in how it plays with other indicators
# .. I think that would need a bigger data set
# Sandro Hawke says: sounds like IRR validates each, but relation to truth might not be visible as a solo indicator
# An Xiao Mina says: even if individual studies don't show results, later studies on same data might pull things out
# .. and different table when we get more studies, eventually
# .. for ads.txt we're not using humans, so no real IRR
# Sandro Hawke says: I cant think of strong reasons to do it one way or another. Might be good to note if an indicator requires no human judgment.
# .. for a yes/no question and all humans agree, IRR = 1. But that's different from having machines test
# An Xiao Mina says: : some things machine be machine checkable, like emotional tone.
# Sandro Hawke says: I think Easy-Machine-Check is different from this ML-type checking which we'll have more and more of
# An Xiao Mina says: subset of extrinsic indicators
# .. things that require stepping outside the content
# .. previously we talked about how theoretically they are more difficult to game
# .. requires link-farm type techniques to game
# .. fact-check-by-IFCN member, but hard to check
# .. want to go beyond IFCN
# .. eg CDC makes assessment
# .. looking for broader typology of inbound links
# [An Xiao Mina] https://credweb.org/cciv/#inbound-references
# Sandro Hawke says: for intrinsic indicators, we could theoretically train systems to detect these over time
# .. a fact check out there is a completely different game - it's a matter of determining if content makes the same claim as a fact check
# .. computational linguistics challenge, not a machine learning thing
# .. wouldn't expect results to be feeding into weighting algorithm in nearly the same way, weight on this would be so much higher than other kinds of indicators
# .. you could align a claimreview with a source that's trusted
# .. IIUC, FB uses claim data from partners, and Google doesnt exactly use IFCN. they have their own computation about whether a site is reliable. will accept claimreview from non IFCN members in theory. not saying IFCN is exactly the gold standard. very high signal into legitimacy
# An Xiao Mina says: I can see how some of this might fall under area-2, under claims
# .. is it about Claims, or all inbound
# Sandro Hawke says: : I think only Claims go into Area-2, and inbound links from wikipedia go in Area-3 Reputation
# An Xiao Mina says: that's a thing you could code up
# .. inbound links for domains, as well as pages
# Sandro Hawke says: this sounds like reputation logic
# .. the kind of code you'd use, not human readers
# .. are there extrinsic indicators you do want to have the reviews look at?
# An Xiao Mina says: Verdict from FC website, or Verdict from high reputation source, where students manually go look
# .. ratio of comments to likes (on facebook), linkes from social media, links from other news sites, FB shares and engagements
# .. not likely to ask students to do, more automation-friendly
# .. also FB comment sentiment
# Davide Ceolin says: how sources refer to other sources, inbound links might be positive or negative mention, sentiment of the mention
# Sandro Hawke says: could be done with human reviewers seeing a snip of the mention; maybe sentiment analysis code is good enough
# Sandro Hawke says: of course on twitter as a human I can't tell what's sarcastic
# An Xiao Mina says: there is literature around this
# .. is this under reputation?
# An Xiao Mina says: The conversation around an article, vs the conversation about a source
# Sandro Hawke says: not exactly reputation. if one of my friends says the NYT sucks, that's clearly reputation. if someone says an article in the NYT sucks, that's not really reputation
# .. just one inbound link with content about it, analysis of an article, that's complicated
# Sandro Hawke says: : maybe this is another area, because it's hugely useful, but doesn't exactly line up with existing articles
# An Xiao Mina says: : Maybe this is another credco study
# Sandro Hawke says: some kind of study where you show ten tweets linking to article
# .. my sense is given rep of twitter feeds that'd be good
# .. sounds like some inbound belongs more the ClaimReview
# .. some about reputation
# .. and one that could be a good study
# [Sandro Hawke] no Monday meeting next week, conflict, but week after is good
# Sandro Hawke says: thinking about scoping of this meeting as Area 1 meeting
# [Sandro Hawke] so 2 weeks, Monday, go through other Intrinsic indicators and think about them as a group
# [Sandro Hawke] adjourned
# [Cheryl Langdon-Orr] Present +