1. Stance Classification for Fact-Checking
Lecture: Web Science
Luca Brandt

2. Table of Content Introduction Motivation Fact-Checking Process
Paper 1 – Fake News Challenge
Paper 2 – Relevant Document Discovery for Fact-Checking Articles
Future Work
Luca Brandt Stance Classification Web Science 2019

3. Introduction What are Fake News ? Problem with Fake News
Propaganda (not objective information)
Deliberate disinformation/hoaxes
Reporters paying sources for stories
Made up stories
Problem with Fake News
Spreading misinformation
Reduces trust in news media
Manipulating society
Luca Brandt Stance Classification Web Science 2019

4. Introduction What is Fact-Checking? Claim
Process to check the veracity and correctness of a claim/statement
In non-fictional texts
A classic journalism task
Claim
Statement made by a politician
Story published by another journalist
Rumor on Social Media
Etc.
Luca Brandt Stance Classification Web Science 2019

5. Example Claim
Luca Brandt Stance Classification Web Science 2019

6. Motivation Why Fact-Checking? Fact-Checking and AI
Goal is to provide a verdict if a claim is true, false or mixed
Fight misinformation
Identify Fake News
Providing context to users to understand information better
Fact-Checking and AI
Automatically detect Fake News
Automatically gather documents relevant to a claim
We want to automate the detection of fake news because it is Time consuming
For 1 article to be fact checked 1 person needs approx 1 day
Luca Brandt Stance Classification Web Science 2019

7. Fact-Checking Process
Given a claim/statement
Find documents relevant to claim
Classic Information Retrieval task
Understand the stance of relevant documents
Stance Classification, Classification Problem
Give verdict if claim is true or false
Classification Problem
Understand stance: this is Stance Detection -> given a document and a claim -> what stance has the document regarding the claim is it true or false or mixed
Luca Brandt Stance Classification Web Science 2019

8. Fake News Challenge (FNC-1)
Foster development of AI tech to detect Fake News
50 teams participated from industry and academia
Task: Stance Detection of an entire document
Learn classifier f: (document, headline) -> stance {AGR,DSG,DSC,UNR}
Dataset: 300 Topics – claims with 5-20 documents each
Every document summarized to a headline
Each document matched with every headline to generate dataset
2017
Stance of entire document -> earlier stance classification of a single sentence
Luca Brandt Stance Classification Web Science 2019

9. A Retrospective Analysis of the Fake News Challenge Stance Detection Task
By Hanselowski et al.
13. June 2018
Main Contributions:
First summarizing and analyzing paper for the FNC-1
Reproduction of results of top-3 performers
Proposed a new evaluation metric
Proposed a new model
Year of publication
Luca Brandt Stance Classification Web Science 2019

10. Top-3 Performers 1. TalosComb 2. Athene 3. UCL Machine Reading (UCLMR)
Weighted average model of deep conv net and gradient-boosted decision tree
TalosCNN uses pre-trained word2vec embeddings
TalosTree based on word count, TF-IDF, sentiment, and word2vec embeddings
2. Athene
Multilayer perceptron (MLP) with 6 hidden layers, with handcrafted features
Unigrams, cosine similarity, topic models
3. UCL Machine Reading (UCLMR)
MLP as well but only 1 layer
Term frequency vectors of 5000 most frequent unigrams
Cosine Similarity between TF-IDF vectors of headline and doc
TalosCNN contains several Conv layers and three fully contected layers + softmax to obtain probabilities
UCL
Cosine similarity of word embeddings of nouns and verbs in headline and document
Luca Brandt Stance Classification Web Science 2019

11. Problem with the metric and dataset
Hierarchical metric
.25 points if classified correctly as related {AGR, DSG, DSC} or unrelated {UNR}
.75 points if classified correctly as AGR, DSG, or DSC
But related class is imbalanced
Not difficult to predict related or unrelated (best systems reach 99% UNR)
Correctly predicting related vs unrelated and always picking DSC would achieve a FNC-1 score of > better than the winner
.25 points if document classified correctly given a headline
Dataset
So it favors models which correctly predict majority classes, but it is more interesting to correctly predict the minority classes AGR or DSG in the case of factchecking!
Luca Brandt Stance Classification Web Science 2019

12. Their model and metric F1m metric StackLSTM
Class-wise F1 scores and macro average to F1m score
F1 =
Not affected by large size of majority class
Naive approach predicting UNR and always DSC -> F1m = .444
StackLSTM
Combines best features of their feature test
Concatenate GloVe Wordembeddings fed through 2 stacked LSTMs
Understand the meaning of whole sentence
Hidden state of LSTMs fed through 3 layers NN
Softmax to obtain probabilities
Combines best features of their test:
they tested the features of the top 3 models, how much they contribute and what are the most promising features
GloVe Embeddings highlight
Concatenate up tp 100 tokens of the 50-dim glove embedd. then fed through the LSTMs
With this technique they produce sequences of words which hold more semantic meaning than uni and bi grams and can capture more complex patterns
Luca Brandt Stance Classification Web Science 2019

13. Reproduction of Results
Reproduction of all results successfull without significant deviation
TalosComb is combination of CNN and Tree so they included all three
FNC is originall metric
F1m is their proposed metric
Class wise F1 scores
Regarding FNC score Athene won
But Talos Tree actually performs better than TalosComb
stackLSTM is best but not significant
But is significantly better than every other model in terms of DSG class the minority class -> which is important for fact checking
Luca Brandt Stance Classification Web Science 2019

14. Pros & Cons Pros: Cons: First paper to summarize and analyze the FNC-1
First paper to reproduce the results
Proposed a better metric
Proposed a new model better than the state of the art
Cons:
Proposed new model still has low accuracy of Disagreeing class
Luca Brandt Stance Classification Web Science 2019

15. Relevant Document Discovery for Fact-Checking Articles
Paper by Wang et al.
23. April 2018
Main Contributions:
End-to-end system for relevant document discovery for fact-cheking articles
Better than state of the art stance classification
Better than state of the art relevance classification
Better state of the art
Luca Brandt Stance Classification Web Science 2019

16. Fact-Checking Articles
Adopted Schema.org ClaimReview Markup
Provides structure of an article
Key Fields on top of content:
Claim
Claimant
Verdict
Structured fields cannot provide documents relevant to claim
Identifying claim relevant documents extremely usefull
Luca Brandt Stance Classification Web Science 2019

17. Fact Checking Article & Claim Relevant Doc
Luca Brandt Stance Classification Web Science 2019

18. Overview of their system
Start with a fact checking article
Based on the factchecking article they try to find a set of related articles
Binary classifier: check whether a related document is relevant or not
Use another classifier to predict the stance of relevant documents wrt the fact checking article and its claim
Difference to the systems of other paper
Here they start of a fact-checking article
In the other paper they started with a document headline pair.
In this paper they do a pre classification to eliminate the unrelevant documents
In the other paper they had a lot of unrelated instances.
Luca Brandt Stance Classification Web Science 2019

19. Candidate Generation Via Navigation Via Search with Google
Outgoing links from fact-checking article
But most of them not relevant
Via Search with Google
Key challenge: generate the right set of queries
Texts from title and claim
Title and claim text transformed with entity annotations
Click graph queries
Combining both generating about 2400 related documents
Finding the related documents
Because most of navigational sources not relevant -> generate candidates through search
Entity annotations: extract important information from title and claim -> only entities
Click graph queries: queries which led a user to click on the fact checking article they collect up to 50 queries
Luca Brandt Stance Classification Web Science 2019

20. Relevance Classification
Classifier M: (f,d) -> {relevant, irrelevant}
Features:
Building confidence vectors of entities
Cosine similarity between confidence vectors of
Claim and text/sentence/paragraph of related doc
Sentence of fact-checking article and sentence of related doc
And whole documents
Publication date
Gradient Boosted Decision Tree
Combines all features – predicts relevant or irrelevant
Confidence vectors: how often a word is found in fact-checking article and the entire related doc
Luca Brandt Stance Classification Web Science 2019

21. Stance Classification
Build model M: (f, d) -> {contradict, support}
Similarity not good for stance classification
Find key contradicting patterns in context similar to the claim
Collected 3.1k (claim, contradicting statement) pairs
Built 900-dim lexicon from uni- and bi-grams with grams of highest probability
Uni-grams: hoax, fake, purportedly, rumor
Bi-grams: made-up, fact check, not true, no evidence
Classifier input: fact-checking article and relevant doc
Crowd sourcing labeling: found out 80% of relevant docs are contradict or support
They say number of neither/discuss articles are small in the fact checking domain so they dont lay focus on this.
Difference to the systems of the other paper
There was the class Discussing which is left out here
Two texts can have much lexical overlap but a single not in the sentence can change the whole meaning
Where the contradicting statement was mostly the headline because usually the headline is a contradicting statement
Luca Brandt Stance Classification Web Science 2019

22. Stance Classification
from relevant doc use title, headline, text and prune away text whose similarity is smaller than a threshold
Concatenate text with one sentence before and after the text-> key components
Extract uni- and bi-grams of key components -> final feature vector
Using a Gradient boosted decision tree for prediction
Luca Brandt Stance Classification Web Science 2019

23. Results of their model Interesting their model vs the FNC Winner.
Here original FNC TalosComb was evaluated on their dataset -> performed poorly
And then FNC TalosComb was trained and evaluated on their dataset, but only the gradient boosted decision tree, because TalosCNN model was not trainable -> performed better but still their model was better.
But you have to pay attention to the results:
91% accuracy in stance classification seems to good to be true, especially when you take the 81% accuracy of the relevance classification into account which is the pre processing for the stance classification
They evaluated each part of their system in isolation. So the approx 20% error of the relevance classification is not playing a role in the stance classification
If they would have evaluated the whole system the score for the stance classification would have been lower.

24. Pros & Cons
Pros:
New state of the art stance classification algorithm
Proposed whole end-to-end system for relevant document discovery
Cons:
Not providing their dataset
Not providing the distribution of the dataset
Not providing the per class scores
Evaluation in isolation
Ignored the Discussing class
Distribution not provided
Per class scores
Isolation evaluation
Discussing class still exists
Luca Brandt Stance Classification Web Science 2019

25. Conclusion What are Fake News What is Fact-Checking
Fake News Challenge
Top-3 Performers
Stack LSTM
End-to-End System for Relevant Document Discovery
The Disagreeing Class is not really well predicted
What we showed
Luca Brandt Stance Classification Web Science 2019

26. Future Work All features text based
Including non-textual data as features
Videos -> Image & Speech Recognition
Social Media Pages -> embedded image/graphic information
Develop ML techniques with deeper semantic understanding
Not relying on lexical features
Disagreeing class has to be predicted better
Need to higher accuracy on disagree class
Luca Brandt Stance Classification Web Science 2019

27. Thank you for your attention
Any Questions?

28. Dataset of Paper 2
Unlabled Corpus: fact-checking articles by highly reputable fact- checkers
Relevance-labeled corpus: using their candidate generation algorithm related documents per fact-checking article 33.5M
With crowd-workers and balancing for positive and negative examples a total of (claim, doc) pairs
Crowd-working question: does this doc address the claim?
Stance-labeled corpus: randomly sampled 1200 of the positive instances of relevance-labled corpus and crowdsourced -> support, contradict, neither, can‘t tell
For 12% workers couldn‘t agree -> removed
Manual Corpus: to measure candidate generation, randomly sampled 450 fact-checking articles and let crowdworkers search for candidates
Luca Brandt Stance Classification Web Science 2019

29. Features Description Bag of Words : Topic Models:
1- and 2-Grams with 5000 token vocabulary for headline and doc
Added a negation flag „_NEG“ as prefix to every word between special negation words like „not“, „never“, „no´“ until next punctuation mark
Topic Models:
Non-negative matrix factorization
Latent semantic indexing
Latent dirichlet allocation
Similarity between topic models and headlines and bodies
Luca Brandt Stance Classification Web Science 2019

30. Feature Description II
Lexicon-based features
Based on lexicons which hold the sentiment/polarity for each word
Computed seperately for headline and body
Count positive and negative polarized words -> features
Find maximum positive and negative polarity -> features
Last word with negative or positive polarity -> feature
Refuting words list („fake“, „hoax“) -> features
Concatenating all features from above
Readability features
Measured with different metrics (e.g. SMOG, Flesch-Kincaid, Gunning fog)
Luca Brandt Stance Classification Web Science 2019