Building Topic/Trend Detection System based on Slow Intelligence Chia-Chun Shih & Ting-Chun Peng Institute for Information Industry Taipei, Taiwan Presented at DMS’10 special session on Slow Intelligence Systems

Agenda Introduction Topic/Trend Detection System Topic/Trend Detection System with Slow Intelligence Conclusion

Introduction

Introduction Social media is prevailing Blog Posts Twitter Posts Facebook Users Social media is prevailing Social media is a reflection of real-world An experiment from HP Social Computing Lab shows: Twitter-rate time series can accurately predict box-office movie sales with Adjusted R2 = 0.973 (amazing!!) The emerging market for Social Media Monitoring Service E.g., Nielsen Buzzmetrics, Radian6

Introduction Topic Detection and Tracking (TDT) (cont’d) Initiated by DARPA at 1996 discover the topical structure in unsegmented streams of news reporting as it appears across multiple media Tasks: Topic Detection Topic Tracking First Story Detection Story Segmentation Link Detection

Knowledge-based Controller Introduction (cont’d) Slow Intelligence provides a software development framework for systems with insufficient computing resources to gradually adapt to environments to handle complexities Environment Knowledge-based Controller Problem Solution 1 2 3 4 Enumerator Adaptor Eliminator Concentrator Slow Intelligence System

Introduction (cont’d) In this paper, we propose a design of online topic/trend detection system for Social Media with the advantages of Slow Intelligence. Four complexities of designing online topic/trend detection systems are identified, along with corresponding Slow Intelligence solutions.

Topic/Trend Detection System

Topic/Trend Detection System Objective Detect current hot topics and to predict future hot topics based on data collected from Social Media Three components Crawler & Extractor: Collect data and extract information from Social Media Topic Extractor: Detect hot topics from a set of text documents Trend Detector: Detect trends (future hot topics) based on currently available data Current Hot topics Social Media Crawler & Extractor Topic Extractor Trend Detector Future Hot topics

Topic/Trend Detection System (cont’d) Crawler & Extractor Social Media User’s Keywords of Interests HTML documents Web Crawler Text documents Web data DB Information Extractor Topic Extractor * Extract articles and metadata (title, author, content, etc) from semi-structured web content Crawler & Extractor

Topic/Trend Detection System (cont’d) Topic Extractor Web data DB Current topics Topic Word Extraction Topic Word Clustering Apply TF-IDF scheme to generate Top-N topic words for each document Apply clustering algorithm to cluster topic words into topic groups. The topic groups are treated as “topics” Current Hot topics Hot topic extraction Apply aging theory to find hot topics Topic Extractor

Topic/Trend Detection System (cont’d) Trend Detector Current topics Trend Estimation Algorithms Topic Trend (Future Hot Topics) Trend Detector The Trend Estimation Algorithm is a black box now, however, it will “find its way” when Slow Intelligence is involved in the system

Topic/Trend Detection System with Slow Intelligence

T/TD System with Slow Intelligence Four complexities of designing online topic/trend detection systems 1. It is unlikely to collect all web data based on limited amount of computing resources. The system needs to develop data collection strategies which can concentrate limited resources on collecting important web data. Crawler & Extractor

T/TD System with Slow Intelligence (cont’d) 2. Many computation methods are available for estimating trends. If parameter settings are also taken into account, there are too many combinations to choose. Furthermore, Internet is a changing environment, which means current best solution may not perform well in the future. The system needs to automatically (or at least quasi-automatically) find best solution from many alternatives in a changing environment. Trend Detector

T/TD System with Slow Intelligence (cont’d) 3. The crawler needs to revisit websites to collect up-to-date data in hourly or daily intervals. Each site has different amount of to-be-update data and different policy to restrict frequent access, which are unknown beforehand. The system needs to find feasible data collection schedule based on past experience. Crawler & Extractor

T/TD System with Slow Intelligence (cont’d) 4. Any changes in web pages may disrupt Extractors. It needs automatic repair mechanism for Extractors if many websites are being monitored. The repair mechanism needs to detect errors of Extractors, find alternatives, and choose the best solution from alternatives to fix the disrupted Extractors. Crawler & Extractor

T/TD System with Slow Intelligence (cont’d) 1. SIS to help restrict the range of data collection Knowledge of data Knowledge of algorithm

T/TD System with Slow Intelligence (cont’d) 2. SIS to help select and adapt trend detection algorithms

T/TD System with Slow Intelligence (cont’d) 3. SIS to help scheduling Crawler

T/TD System with Slow Intelligence (cont’d) 4. SIS to help adapt Extractors

Conclusion

Conclusion An online trend detection system requires careful resource allocation and automatic algorithm adaptation to process huge size of heterogeneous data. This research adopts Slow Intelligence, which provides a framework for systems with insufficient computing resources to gradually adapt to environments, to response the challenges. Four Slow Intelligence subsystems are proposed, and each subsystem targets a challenge in designing online topic/trend detection systems.

If you have any questions, please e-mail us chiachun@iii.org.tw (Chia-Chun Shih) markpeng@iii.org.tw (Ting-Chun Peng)