1. Speaker: Yun-Nung Chen 陳縕儂 Advisor: Prof. Lin-Shan Lee 李琳山 National Taiwan University Automatic Key Term Extraction and Summarization from Spoken Course Lectures 課程錄音之自動關鍵用語擷取及摘要

2. Introduction 2 Master Defense, National Taiwan University Target: extract key terms and summaries from course lectures

3. Key TermSummary O Indexing and retrieval O The relations between key terms and segments of documents 3 Introduction Master Defense, National Taiwan University O Efficiently understand the document Related to document understanding and semantics from the document Both are “Information Extraction”

4. 4 Master Defense, National Taiwan University

5. Definition O Key Term O Higher term frequency O Core content O Two types O Keyword O Ex. “ 語音 ” O Key phrase O Ex. “ 語言 模型 ” 5 Master Defense, National Taiwan University

6. Automatic Key Term Extraction 6 ▼ Original spoken documents Archive of spoken documents Branching Entropy Feature Extraction Learning Methods 1)AdaBoost 2)Neural Network ASR speech signal ASR trans Master Defense, National Taiwan University

7. Automatic Key Term Extraction 7 Archive of spoken documents Branching Entropy Feature Extraction Learning Methods 1)AdaBoost 2)Neural Network ASR speech signal ASR trans Master Defense, National Taiwan University

8. Automatic Key Term Extraction 8 Archive of spoken documents Branching Entropy Feature Extraction Learning Methods 1)AdaBoost 2)Neural Network ASR speech signal ASR trans Master Defense, National Taiwan University

9. Phrase Identification Automatic Key Term Extraction 9 Archive of spoken documents Branching Entropy Feature Extraction Learning Methods 1)AdaBoost 2)Neural Network ASR speech signal First using branching entropy to identify phrases ASR trans Master Defense, National Taiwan University

10. Phrase Identification Key Term Extraction Automatic Key Term Extraction 10 Archive of spoken documents Branching Entropy Feature Extraction Learning Methods 1)AdaBoost 2)Neural Network ASR speech signal Key terms entropy acoustic model : Then using learning methods to extract key terms by some features ASR trans Master Defense, National Taiwan University

11. Phrase Identification Key Term Extraction Automatic Key Term Extraction 11 Archive of spoken documents Branching Entropy Feature Extraction Learning Methods 1)AdaBoost 2)Neural Network ASR speech signal Key terms entropy acoustic model : ASR trans Master Defense, National Taiwan University

12. Branching Entropy 12 O Inside the phrase hidden Markov model How to decide the boundary of a phrase? represent is can : : is of in : : Master Defense, National Taiwan University

13. Branching Entropy 13 O Inside the phrase hidden Markov model How to decide the boundary of a phrase? represent is can : : is of in : : Master Defense, National Taiwan University

14. Branching Entropy 14 hidden Markov model boundary Define branching entropy to decide possible boundary How to decide the boundary of a phrase? represent is can : : is of in : : O Inside the phrase O Boundary of the phrase Master Defense, National Taiwan University

15. Branching Entropy 15 hidden Markov model O Definition of Right Branching Entropy O Probability of x i given X O Right branching entropy for X X xixi How to decide the boundary of a phrase? represent is can : : is of in : : Master Defense, National Taiwan University

16. Branching Entropy 16 hidden Markov model O Decision of Right Boundary O Find the right boundary located between X and x i where X boundary How to decide the boundary of a phrase? represent is can : : is of in : : Master Defense, National Taiwan University

17. Branching Entropy 17 hidden Markov model How to decide the boundary of a phrase? represent is can : : is of in : : Master Defense, National Taiwan University

18. Branching Entropy 18 hidden Markov model How to decide the boundary of a phrase? represent is can : : is of in : : Master Defense, National Taiwan University

19. Branching Entropy 19 hidden Markov model How to decide the boundary of a phrase? represent is can : : is of in : : Master Defense, National Taiwan University boundary Using PAT tree to implement

20. Phrase Identification Key Term Extraction Automatic Key Term Extraction 20 Archive of spoken documents Branching Entropy Feature Extraction Learning Methods 1)AdaBoost 2)Neural Network ASR speech signal Key terms entropy acoustic model : Extract prosodic, lexical, and semantic features for each candidate term ASR trans Master Defense, National Taiwan University

21. Feature Extraction 21 O Prosodic features O For each candidate term appearing at the first time Feature Name Feature Description Duration (I – IV) normalized duration (max, min, mean, range) Speaker tends to use longer duration to emphasize key terms using 4 values for duration of the term duration of phone “a” normalized by avg duration of phone “a” Master Defense, National Taiwan University

22. Feature Extraction 22 O Prosodic features O For each candidate term appearing at the first time Higher pitch may represent significant information Feature Name Feature Description Duration (I – IV) normalized duration (max, min, mean, range) Master Defense, National Taiwan University

23. Feature Extraction 23 O Prosodic features O For each candidate term appearing at the first time Higher pitch may represent significant information Feature Name Feature Description Duration (I – IV) normalized duration (max, min, mean, range) Pitch (I - IV) F0 (max, min, mean, range) Master Defense, National Taiwan University

24. Feature Extraction 24 O Prosodic features O For each candidate term appearing at the first time Higher energy emphasizes important information Feature Name Feature Description Duration (I – IV) normalized duration (max, min, mean, range) Pitch (I - IV) F0 (max, min, mean, range) Master Defense, National Taiwan University

25. Feature Extraction 25 O Prosodic features O For each candidate term appearing at the first time Higher energy emphasizes important information Feature Name Feature Description Duration (I – IV) normalized duration (max, min, mean, range) Pitch (I - IV) F0 (max, min, mean, range) Energy (I - IV) energy (max, min, mean, range) Master Defense, National Taiwan University

26. Feature Extraction 26 O Lexical features Feature NameFeature Description TFterm frequency IDFinverse document frequency TFIDFtf * idf PoSthe PoS tag Using some well-known lexical features for each candidate term Master Defense, National Taiwan University

27. Feature Extraction 27 O Semantic features O Probabilistic Latent Semantic Analysis (PLSA) O Latent Topic Probability Key terms tend to focus on limited topics D i : documents T k : latent topics t j : terms Master Defense, National Taiwan University

28. Feature Extraction 28 O Semantic features O Probabilistic Latent Semantic Analysis (PLSA) O Latent Topic Probability Feature NameFeature Description LTP (I - III) Latent Topic Probability (mean, variance, standard deviation) non-key term key term Key terms tend to focus on limited topics describe a probability distribution Master Defense, National Taiwan University

29. Feature Extraction 29 O Semantic features O Probabilistic Latent Semantic Analysis (PLSA) O Latent Topic Significance Within-topic to out-of-topic ratio Feature NameFeature Description LTP (I - III) Latent Topic Probability (mean, variance, standard deviation) non-key term key term Key terms tend to focus on limited topics within-topic freq. out-of-topic freq. Master Defense, National Taiwan University

30. Feature Extraction 30 O Semantic features O Probabilistic Latent Semantic Analysis (PLSA) O Latent Topic Significance Within-topic to out-of-topic ratio Feature NameFeature Description LTP (I - III) Latent Topic Probability (mean, variance, standard deviation) LTS (I - III) Latent Topic Significance (mean, variance, standard deviation) non-key term key term Key terms tend to focus on limited topics within-topic freq. out-of-topic freq. Master Defense, National Taiwan University

31. Feature Extraction 31 O Semantic features O Probabilistic Latent Semantic Analysis (PLSA) O Latent Topic Entropy Feature NameFeature Description LTP (I - III) Latent Topic Probability (mean, variance, standard deviation) LTS (I - III) Latent Topic Significance (mean, variance, standard deviation) non-key term key term Key terms tend to focus on limited topics Master Defense, National Taiwan University

32. Feature Extraction 32 O Semantic features O Probabilistic Latent Semantic Analysis (PLSA) O Latent Topic Entropy Feature NameFeature Description LTP (I - III) Latent Topic Probability (mean, variance, standard deviation) LTS (I - III) Latent Topic Significance (mean, variance, standard deviation) LTEterm entropy for latent topic non-key term key term Key terms tend to focus on limited topics Higher LTE Lower LTE Master Defense, National Taiwan University

33. Phrase Identification Key Term Extraction Automatic Key Term Extraction 33 Archive of spoken documents Branching Entropy Feature Extraction Learning Methods 1)AdaBoost 2)Neural Network ASR speech signal ASR trans Key terms entropy acoustic model : Using supervised approaches to extract key terms Master Defense, National Taiwan University

34. Learning Methods 34 O Adaptive Boosting (AdaBoost) O Neural Network Automatically adjust the weights of features to train a classifier Master Defense, National Taiwan University

35. Automatic Key Term Extraction 35 Master Defense, National Taiwan University

36. Experiments 36 O Corpus O NTU lecture corpus O Mandarin Chinese embedded by English words O Single speaker O 45.2 hours O ASR System O Bilingual AM with model adaptation [1] O LM with adaptation using random forests [2] Master Defense, National Taiwan University LanguageMandarinEnglishOverall Char Acc (%)78.1553.4476.26 [1] Ching-Feng Yeh, “Bilingual Code-Mixed Acoustic Modeling by Unit Mapping and Model Recovery,” Master Thesis, 2011. [2] Chao-Yu Huang, “Language Model Adaptation for Mandarin-English Code-Mixed Lectures Using Word Classes and Random Forests,” Master Thesis, 2011.

37. Experiments 37 O Reference Key Terms O Annotations from 61 students who have taken the course O If the an annotator labeled 150 key terms, he gave each of them a score of 1/150, but 0 to others O Rank the terms by the sum of all scores given by all annotators for each term O Choose the top N terms form the list O N is average number of key terms O N = 154 key terms O 59 key phrases and 95 keywords O Evaluation O 3-fold cross validation Master Defense, National Taiwan University

38. Experiments 38 O Feature Effectiveness O Neural network for keywords from ASR transcriptions Each set of these features alone gives F1 from 20% to 42% Prosodic features and lexical features are additiveThree sets of features are all useful 20.78 42.86 35.63 48.15 56.55 Pr: Prosodic Lx: Lexical Sm: Semantic F-measure Master Defense, National Taiwan University

39. Experiments 39 O Overall Performance (Keywords & Key Phrases) Baseline Branching entropy performs well F-measure Master Defense, National Taiwan University N-Gram TFIDF Branching Entropy TFIDF Branching Entropy AdaBoost Branching Entropy Neural Network key phrase keyword 23.44 52.60 57.68 62.70

40. The performance of manual is slightly better than ASR Experiments 40 O Overall Performance (Keywords & Key Phrases) 55.84 Baseline 62.39 67.31 32.19 Supervised learning using neural network gives the best results F-measure Master Defense, National Taiwan University N-Gram TFIDF Branching Entropy TFIDF Branching Entropy AdaBoost Branching Entropy Neural Network key phrase keyword 23.44 52.60 57.68 62.70

41. 41 Master Defense, National Taiwan University

42. Introduction 42 O Extractive Summary O Important sentences in the document O Computing Importance of Sentences O Statistical Measure, Linguistic Measure, Confidence Score, N-Gram Score, Grammatical Structure Score O Ranking Sentences by Importance and Deciding Ratio of Summary Master Defense, National Taiwan University Proposed a better statistical measure of a term

43. Statistical Measure of a Term 43 O LTE-Based Statistical Measure (Baseline) O Key-Term-Based Statistical Measure O Considering only key terms O Weighted by LTS of the term Master Defense, National Taiwan University T k-1 TkTk T k+1 … … Key terms can represent core content of the document Latent topic probability can be estimated more accurately t i key

44. Importance of the Sentence 44 O Original Importance O LTE-based statistical measure O Key-term-based statistical measure O New Importance O Considering original importance and similarity of other sentences Master Defense, National Taiwan University Sentences similar to more sentences should get higher importance

45. Random Walk on a Graph 45 O Idea O Sentences similar to more important sentences should be more important O Graph Construction O Node: sentence in the document O Edge: weighted by similarity between nodes O Node Score O Interpolating two scores O Normalized original score of sentence S i O Scores propagated from neighbors according to edge weight p(j, i) Master Defense, National Taiwan University Nodes connecting to more nodes with higher scores should get higher scores score of S i in k-th iter.

46. Random Walk on a Graph 46 O Topical Similarity between Sentences O Edge weight sim(S i, S j ): (sentence i  sentence j ) O Latent topic probability of the sentence O Using Latent Topic Significance Master Defense, National Taiwan University SjSj t LTS SiSi … … TkTk T k+1 tjtj T k-1 titi tktk

47. Random Walk on a Graph 47 O Scores of Sentences O Converged equation O Matrix form O Solution dominate eigen vector of P’ O Integrated with Original Importance Master Defense, National Taiwan University

48. Automatic Summarization 48 Master Defense, National Taiwan University

49. Experiments 49 O Same Corpus and ASR System O NTU lecture corpus O Reference Summaries O Two human produced reference summaries for each document O Ranking sentences from “the most important” to “of average importance” O Evaluation Metric O ROUGE-1, ROUGE-2, ROUGE-3 O ROUGE-L: Longest Common Subsequence (LCS) Master Defense, National Taiwan University

50. Evaluation 50 Master Defense, National Taiwan University ASR ROUGE-1 LTE Key ROUGE-2 ROUGE-3 ROUGE-L Key-term-based statistical measure is helpful

51. Evaluation 51 Master Defense, National Taiwan University ROUGE-1ROUGE-2 ROUGE-3 ROUGE-L Random walk can help the LTE-based statistical measure ROUGE-1 ROUGE-2 ROUGE-3 ROUGE-L Random walk can also help the key-term-based statistical measure LTE LTE + RW Key Key + RW ASR Topical similarity can compensate recognition errors

52. Evaluation 52 Master Defense, National Taiwan University ROUGE-1ROUGE-2 ROUGE-3 ROUGE-L ASR LTE LTE + RW Key Key + RW Manual Key-term-based statistical measure and random walk using topical similarity are useful for summarization

53. 53 Master Defense, National Taiwan University

54. Automatic Key Term ExtractionAutomatic Summarization The performance can be improved by ▫ Key-term-based statistical measure ▫ Random walk with topical similarity  Compensating recognition errors  Giving higher scores to sentences topically similar to more important sentences  Considering all sentences in the document 54 The performance can be improved by ▫ Identifying phrases by branching entropy ▫ Prosodic, lexical, and semantic features together Conclusions Master Defense, National Taiwan University

55. Published Papers: [1] Yun-Nung Chen, Yu Huang, Sheng-Yi Kong, and Lin-Shan Lee, “Automatic Key Term Extraction from Spoken Course Lectures Using Branching Entropy and Prosodic/Semantic Features,” in Proceedings of SLT, 2010. [2] Yun-Nung Chen, Yu Huang, Ching-Feng Yeh, and Lin-Shan Lee, “Spoken Lecture Summarization by Random Walk over a Graph Constructed with Automatically Extracted Key Terms,” in Proceedings of InterSpeech, 2011. 55 Master Defense, National Taiwan University