Multimodal Alignment of Scholarly Documents and Their Presentations Bamdad Bahrani JCDL 2013 Submission Feb 2013.

Slides:



Advertisements
Similar presentations
Yansong Feng and Mirella Lapata
Advertisements

Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
Lesson 15 Presentation Programs.
Document Summarization using Conditional Random Fields Dou Shen, Jian-Tao Sun, Hua Li, Qiang Yang, Zheng Chen IJCAI 2007 Hao-Chin Chang Department of Computer.
Road-Sign Detection and Recognition Based on Support Vector Machines Saturnino, Sergio et al. Yunjia Man ECG 782 Dr. Brendan.
Title of Presentation Author 1, Author 2, Author 3, Author 4 Abstract Introduction This is my abstract. This is my abstract. This is my abstract. This.
TTI's Gender Prediction System using Bootstrapping and Identical-Hierarchy Mohammad Golam Sohrab Computational Intelligence Laboratory Toyota.
Text Classification Using Stochastic Keyword Generation Cong Li, Ji-Rong Wen and Hang Li Microsoft Research Asia August 22nd, 2003.
Software Documentation Written By: Ian Sommerville Presentation By: Stephen Lopez-Couto.
Abstract This template explains how to prepare Power Point slide of your short paper for the Proceedings of the ICServ2014 so that its appearance is clear.
Slide Image Retrieval: A Preliminary Study Guo Min Liew and Min-Yen Kan National University of Singapore Web IR / NLP Group (WING)
The use of machine translation tools for cross-lingual text-mining Blaz Fortuna Jozef Stefan Institute, Ljubljana John Shawe-Taylor Southampton University.
What are they and what should you know about them?
MIEL 2012 Title of Your Talk Author name(s) Affiliation.
Crawling and Aligning Scholarly Presentations and Documents from the Web By SARAVANAN.S 09/09/2011 Under the guidance of A/P Min-Yen Kan 10/23/
Logical Structure Recovery in Scholarly Articles with Rich Document Features Minh-Thang Luong, Thuy Dung Nguyen and Min-Yen Kan.
INF5181 / Project Fall 2013 Project – Details of submission Deadline: November 18 th, Format: PDF Together with the project report you must submit a declaration.
IR Homework #3 By J. H. Wang May 4, Programming Exercise #3: Text Classification Goal: to classify each document into predefined categories Input:
Personal Project. Topic Modeling and Presenting Data from a Publication Objectives –Using XML related techniques to model and present data from a publication.
Creating Accessible Content in Microsoft Office 2010 NC Basic Skills Instructor Training Academy 2012.
AN EFFECTIVE STATISTICAL APPROACH TO BLOG POST OPINION RETRIEVAL Ben He Craig Macdonald Iadh Ounis University of Glasgow Jiyin He University of Amsterdam.
Citation Provenance FYP/Research Update WING Meeting 28 Sept 2012 Heng Low Wee 1/5/
Text Based Similarity Metrics and Delta for Semantic Web Graphs Krishnamurthy Koduvayur Viswanathan Monday, June 28,
Feb 21-25, 2005ICM 2005 Mumbai1 Converting Existing Corpus to an OAI Compliant Repository J. Tang, K. Maly, and M. Zubair Department of Computer Science.
Jean-Yves Le Meur - CERN Geneva Switzerland - GL'99 Conference 1.
Week 4: 6/6 – 6/10 Jeffrey Loppert. This week.. Coded a Histogram of Oriented Gradients (HOG) Feature Extractor Extracted features from positive and negative.
Formal Report Strategies. Types of Formal Reports Informational Presents Info Analytical Presents Info Analyses info and draws conclusions Recommendation.
1 Bilinear Classifiers for Visual Recognition Computational Vision Lab. University of California Irvine To be presented in NIPS 2009 Hamed Pirsiavash Deva.
CMGT 430 Week 4 Learning Team Presentation Check this A+ tutorial guideline at Week-4-Learning-Team-Presentation.
A Simple Approach for Author Profiling in MapReduce
Accessibility issues: Images and the use of ALT Text
Your Poster Title Goes Here
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
Deep Compositional Cross-modal Learning to Rank via Local-Global Alignment Xinyang Jiang, Fei Wu, Xi Li, Zhou Zhao, Weiming Lu, Siliang Tang, Yueting.
Software Documentation
Author name(s) Affiliation
Extraction, aggregation and classification at Web Scale
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
Using Transductive SVMs for Object Classification in Images
Poster Name [Align Center]
Author’s Name/s Goes Here
Experience with XML – based production of publications Case of « Statistical yearbook 2005 and 2006  » Guy Zacharias Centralisation et Diffusion STATEC.
Author’s Name/s Goes Here
Your Poster Title Goes Here
Your Poster Title Goes Here
Your Poster Title Goes Here
Your Poster Title Goes Here
Your Poster Title Goes Here
Your Poster Title Goes Here
Author’s Name/s Goes Here
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
Your Poster Title Goes Here
Presenter’s Name (if case)
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
Your Poster Title Goes Here
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
Your Poster Title Goes Here
Title Introduction: Discussion & Conclusion: Methods & Results:
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
Include your personal presentation if necessary
Extracting Why Text Segment from Web Based on Grammar-gram
WAYNE STATE UNIVERSITY – DETROIT, MICHIGAN
Your Poster Title Goes Here
TITLE PRESENTATION Include your personal presentation if necessary.
Your Poster Title Goes Here
Presentation transcript:

Multimodal Alignment of Scholarly Documents and Their Presentations Bamdad Bahrani JCDL 2013 Submission Feb 2013

Motivation 0 How many papers do you read every week? 0 How many you read deeply? 0 How many you just skim? 0 Title, abstract and conclusion  Enough? 0 A summary of the paper  Most important issues IntroductionAnalysisMethodExperiment & ResultConclusion 2

Motivation 0 Slide Presentation as a summary 0 It includes important contents from paper 0 It is made by the same author 0 But 0 Not detailed enough 0 Misses some technical parts of the paper IntroductionAnalysisMethodExperiment & ResultConclusion 3

Introduction 0 The Paper 0 and its Slide Presentation 0 Alignment map IntroductionAnalysisMethodExperiment & ResultConclusion 4

Previous Works 0 Hayama et al Japanese technical papers and presentation sheets 0 Using HMM 0 Kan SlideSeer 0 Crawling of paper-presentation pairs, aligning them and GUI 0 Beamer and Girju Detailed analysis of different similarity measures 5 IntroductionAnalysisMethodExperiment & ResultConclusion Only Textual Content

Slide Analysis IntroductionAnalysisMethodExperiment & ResultConclusion 6

Error Analysis Slide TypeIncorrectly aligned in baseline Common reason Nil64%Doesn’t know where to align  align to best fit Outline36%Name of some sections in it  align to longest one Image81%Very little text available Drawing53%Noisy data: lots of shapes and text boxes Table50%Little text, noisy data Around 70% are showing “Evaluation and Result” IntroductionAnalysisMethodExperiment & ResultConclusion 7

Alignment Modals 0 Text Similarity 0 Between each slide and each section 0 The core aligner unit 0 The baseline 0 A cosine similarity measure: TF. IDF 0 Linear Ordering 0 Ordering between slides and sections are monotonic 0 Visual appearance of slides MotivationAnalysisMethodExperiment & ResultDiscussion 8

Text Extraction Unit 0 Presentation 0 Paper MS PowerPoint VB compiler Slides 1.Slide Title text 2.Slide Body text 3.Slide Number PDFx PDF Parser (via Python) XML 1.Section Title 2.Section Body IntroductionAnalysisMethodExperiment & ResultConclusion 9

Slide Image Classifier Unit Take Snapshot Slides 1. Text 2. Outline 3. Drawing 4. Results Image Classifier Image IntroductionAnalysisMethodExperiment & ResultConclusion 10

Image Class Instructions 0 1. Text 0 Text similarity alignment weight  Increase 2/ Outline 0 Text similarity alignment weight  Decrease 1/3 0 Linear ordering alignment weight  Decrease 1/ Drawing 0 Uniform probability for all weights 0 4. Result 0 Exceptional rule: Align directly to “Experiment and Result” section IntroductionAnalysisMethodExperiment & ResultConclusion 11

Image Classifier experiment and result Manually annotated slides 0 Linear SVM 0 Feature extraction: Histogram of Oriented Gradiants 0 Blurring filters 0 Normalization 0 10 fold cross validation Image ClassTextOutlineDrawingResultAverage Correctly Classified 86%95%83%84%87.2% IntroductionAnalysisMethodExperiment & ResultConclusion 12

Experiments 0 Experiment 1: 0 Baseline 0 Paragraph-to-slide alignment 0 Only textual data 0 Experiment 2: 0 Section-to-slide alignment 0 Only textual data IntroductionAnalysisMethodExperiment & ResultConclusion 13

Experiments 0 Experiment 3: 0 The effect of Linear Ordering alignment was added. 0 Textual data and ordering information 0 Experiment 4: 0 The effect of Image Classification was added. 0 Textual data, ordering information and visual content IntroductionAnalysisMethodExperiment & ResultConclusion 14

Results BaselineSection Ordering Image Class IntroductionAnalysisMethodExperiment & ResultConclusion 15 25%

Conclusion 0 Many slides with images and drawings 0 Textual data is not enough 0 Taking advantage of graphical features of slides IntroductionAnalysisMethodExperiment & ResultConclusion 16

Future Tasks 0 Bigger dataset 0 More efficient text similarity measures 0 Differentiate between Title and Body text weights 0 Support more input file format 0 A GUI to view aligned documents IntroductionAnalysisMethodExperiment & ResultConclusion 17

Thank you…! 18 IntroductionAnalysisMethodExperiment & ResultConclusion

System Architcture Input: Presentation Text Extraction Textual Similarity Input: Document nil Linear Ordering 1. Text 3. Drawing 2. Index 4. Results Multimodal Fusion Slide Image Classifier Output: Alignment 19