1 Sentence-extractive automatic speech summarization and evaluation techniques Makoto Hirohata, Yosuke Shinnaka, Koji Iwano, Sadaoki Furui Presented by Yi-Ting Chen

2 Reference Makoto Hirohata, Yosuke Shinnaka, Koji Iwano and Sadaoki Furui, Sentence-extractive automatic speech summarization and evaluation techniques, Speech Communication, In Press, Corrected Proof,, Available online 5 June 2006

3 Outline Introduction Sentence extraction methods Objective evaluation metrics Experiments Conclusion

4 Introduction Why summarize? –Recognition errors cause transcriptions of obtained from spontaneous speech to include irrelevant or incorrect information –Spontaneous speech is ill-formed and usually includes redundant information –Direct transcriptions are therefore not useful They have proposed a two-stage summarization method, but it was confirmed that sentence extraction is more important than sentence compaction This paper investigates and evaluates sentence extraction- based speech summarization techniques under 10% summarization ratio

5 Sentence extraction methods

6 Extraction using sentence location –One hundred and sixty nine presentation were used in the analysis –This result shows that human subjects tend to extract sentence from introduction and conclusion segments under 10% summarization ratio –This is no such tendency at 50%

7 Sentence extraction methods Extraction using sentence location –The introduction and conclusion segments are estimated based on the Hearst method using sentence cohesiveness –The cohesiveness C(r) for each segmentation boundary r is measured by a cosine value: –The segmentation boundary for the end of an introduction part is the beginning of the presentation where the cohesiveness becomes lower than a preset threshold set to 0.1

8 Sentence extraction methods Extraction using confidence and linguistic scores – Confidence score: a logarithmic value of the posterior probability for each transcribed word Linguistic score

9 Sentence extraction methods Extraction using a significance scores Extraction using latent semantic analysis

10 Sentence extraction methods Extraction using dimension reduction based on SVD SVD Dimension reduction Weighted word-frequency vector Weighted singular-value vector Reduced dimension vector

11 Objective evaluation metrics Summarization accuracy –All the human summaries are merged into a single word network –Word accuracy of the automatic summary is then measured as a summarization accuracy in comparison with the closest word string extracted from the word network (SumACCY) –Problem: the variation between manual summaries is so large that the network accepts inappropriate summaries –Using word accuracy obtained by using the manual summaries individually was proposed (SumACCY-E, SumACCY-E/Max, SumACCY-E/avg)

12 Objective evaluation metrics Sentence F-measure –Sentence recall/precision or F-measure is commonly used in evaluating sentence-extraction text summarization. –Since sentence boundaries are not explicitly indicated in input speech –Extraction of a sentence in the recognition result is considered as extraction of one or multiple sentence in the manual summary having an overlap of 50% or more words –F-measure/max, F-measure/ave.

13 Objective evaluation metrics N-gram recall –ROUGE-N is an N-gram recall between an automatic summary and a set of manual summaries. – –S H is a set of manual summaries, S is an individual manual summary, g n is an N-gram, C(g n ) is the number of g n ’s in the manual summary, and C(g n ) is the number of co-occurrences of g n in the manual summary and automatic summary. –1-grams, 2-grams, and 3-grams

14 Experiments Experimental conditions –30 presentations by 20 males and 10 females in the CSJ were automatically summarized at 10% summarization ratio. –Mean word recognition accuracy was 69%. –Sentence boundaries in the recognition results were automatically determined using language models, which achieved 72% recall and 75% precision. Subjective evaluation –180 automatic summaries (30 presentations X 6 summarization methods) were evaluated by 12 human subjects –Methods: SIG, LSA, DIM, SIG+IC, LSA+IC, and DIM+IC –DIM: k set to 5

15 Experiments Correlation between subjective and objective evaluation results

16

17 Experiments Correlation between subjective and objective evaluation results

18

19 Experiments objective evaluation results for various summarizations –The weight factors were optimized a posteriori so that the objective evaluation score got maximized –The parameter K was also set a posteriori to 1 Word recognition accuracy

20 Experiments objective evaluation results for various summarizations Word recognition accuracy

21 Conclusion A sentence extraction method using dimension reduction based on SVD and another method using sentence location information are proposed Among the objective evaluation metrics, summarization accuracy, sentence F-measure, 2 and 3-gram recall were found to the effective under 10% summarization ratio It was confirmed that the summarization method using sentence location improves summarization results