Fidel Castro: Information Retrieval for 37 years of Socialist Oratory Yevgeni Berzak, Carsten Ehler, Michal Richter & Todd Shore Project Seminar: Text Mining/NLP for Historical Documents Caroline Sporleder & Michael Schreiber Computational Linguistics & Phonetics Saarland University 19.06.2018 http://upload.wikimedia.org/wikipedia/commons/thumb/e/ed/02.Trinidad_%2859%29.JPG/800px-02.Trinidad_%2859%29.JPG

Berzak, Ehrler, Richter & Shore Overview Introduction Goals Methods C onclusion http://lanic.utexas.edu/project/castro/image/fidel3.jpg 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 1) Introduction U of Texas DB: “includes speeches, interviews, etc by Fidel Castro from 1959 – 1996” Organised by date, annotated with basic metadata: date, location, document type (e.g. 'speech', 'interview') Keyword search http://www.nndb.com/people/118/000023049/fidel-castro-sm.jpg 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 1) Introduction Issues: Keyword-based search is unsophisticated, not designed specifically for historians/intelligent searching Relative lack of structure within documents makes standard methods of retrieving document info (e.g. categorisation, summarisation) difficult No explicit link between documents in corpus 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 2) Goals Index corpus by named entities Calculate similarity of documents to each other based on named entities and lexical similarity Implement metadata-driven document retrieval system Provide linking between results of retrieval system Represent similarity of documents based on one type of entity (e.g. persons or locations) or many Search for documents related to designated named-entity keyword(s), input by user Interactive visualisation of results 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 3) Methods Download data, extract text Extract metadata from header Date, location, document type (e.g. “speech”, “interview”) Generate database 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 3) Methods Stanford Named Entity Recognizer The visits to these places have made unforgettable impressions on us. Murmansk, for example, and Moscow, Volgograd, Uzbekistan, Tashkent, and Samarkand--in each of these places we have seen the efforts by the Soviet people, and how the Soviet workers, led by the Communist Party, are creating great things. We have seen regions separated by great distances... 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 3) Methods Stanford Named Entity Recognizer The visits to these places have made unforgettable impressions on us. Murmansk, for example, and Moscow, Volgograd, Uzbekistan, Tashkent, and Samarkand--in each of these places we have seen the efforts by the Soviet people, and how the Soviet workers, led by the Communist Party, are creating great things. We have seen regions separated by great distances... 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 3) Methods Stanford Named Entity Recognizer Fidel Castro Fidel CASTRO Dr. Fidel Castro Castro Fidel CASTRO Dr. Castro FidelCastro Dr Fidel Castro 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 3) Methods Calculate: NE similarities with p-spectrum string kernels Calculate overall lexical similarity of documents Fidel Dr. Fidel Allende 1 0.8 Allendre NOTE: Mock data 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 3) Methods NE counts include not only NEs actually in document, but also equivalent NEs NE query mechanism includes all NEs similar to query, not only exact string matches For each pair of documents, similarity measurement is produced: either for named entities or lexical similarity 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 4) Methods Interface: Interactive GUI, allowing for visualisation of custom queries based on different combinations of similarity measures Graphical representation Node: Document Edge: Similarity measure 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 4) Methods 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore 4) Conclusion Representation of similarity information enables further information to be inferred than through metadata alone (e.g. “topics”, correlation to other variables – time, location) NE similarity info can be used to find related documents Guided exploration of information tailored to (historical) researcher's interest based on similarities is possible and effective 19/06/2018 Berzak, Ehrler, Richter & Shore

Berzak, Ehrler, Richter & Shore Works cited Castro Speech Database. Retrieved 1 Mar 2010, from University of Texas at Austin, LANIC website: http://lanic.utexas.edu/la/cb/cuba/castro.html. Stanford Named Entity Recognizer. Retrieved 4 Mar 2010, from Stanford University, The Stanford NLP (Natural Language Processing) Group website: http://nlp.stanford.edu/software/CRF-NER.shtml. 19/06/2018 Berzak, Ehrler, Richter & Shore http://upload.wikimedia.org/wikipedia/commons/5/55/Trinidad_(Kuba)_02.jp g

Berzak, Ehrler, Richter & Shore 1) Introduction “Castro Warns Against Complacency” “Speaking here tonight I am presented with one of the most difficult obligations in this long struggle which began on Nov. 30, 1956, in Santiago. The people are listening, the revolutionaries are listening, and the soldiers whose destinies are in other hands are listening also. this is a decisive moment in our history: The tyranny has been overthrown, but there is still much to be done. Let us not fool ourselves into believing that the future will be easy; perhaps everything will be more difficult in the future.” –Fidel Castro, 9 Jan, 1959 http://lanic.utexas.edu/project/castro/db/1959/19590109.html 19/06/2018 Berzak, Ehrler, Richter & Shore