Internet Searching and Browsing in a Multilingual World An Experiment on the Chinese Business Intelligence Portal Acknowledgment: NSF/NIJ Grant

2 Outline Motivation The Chinese Business Intelligence Portal –System Description –Results of Usability Study Conclusions

Introduction

4 Motivation As the Internet grows in popularity worldwide, more users want to access Web content in their native languages –The majority of the total global online population (63.5%) lives in non-English-speaking areas (Global-Reach, 2002) –Such population is estimated to grow rapidly, much faster than English-speaking population However, existing search engines may not serve their needs, because most technologies have been developed for English-speaking users

5 This Presentation The following slides present our efforts in creating and evaluating intelligent Web portals that address the above needs –The Chinese business information serves as our research testbed Through the studies, we aim to achieve better understanding of human interaction and analysis with automated systems developed for Internet searching and browsing in a multilingual world

The Chinese Business Intelligence Portal (CBizPort)

7 CBizPort The Chinese Business Intelligence Portal (CBizPort) –Two versions of user interface: Simplified Chinese and Traditional Chinese –URLs Introduction: Portal: –Each version has the same user interface and provides the same functions Encoding conversion Meta searching major Chinese information sources Summarization, Categorization Providing links to major Chinese business Web resources –The following slides show the system architecture and screen shots of CBizPort

8

9 Keywords: Meta searches 8 major information sources of Mainland China, Hong Kong, and Taiwan Provides links to major Chinese business Web sites and resources Provides both Simplified and Traditional Chinese versions of user interface Allows input of multiple key terms

10 Search PageResult Page Categorizer A two-sentence summary on left, original page on right Summarizer Web pages grouped by key phrases extracted by mutual information algorithm (non- exclusive categorization)

11 Evaluation of CBizPort Objectives 1.To evaluate the performance of summarizer as a preview function and categorizer as an overview function 2.To compare CBizPort with regional Chinese search engines to study its effectiveness and usability 3.To evaluate, in comparison with existing regional Chinese search engines, the information quality obtained from CBizPort and its capability of searching for cross- regional business information

12 Experimental Design Searching and browsing were studied Scenario-based, culturally oriented tasks, e.g., –A search task (4 min): “Find two cities in mainland China that Motorola has set up its manufacturing operations” –A browse task (5 min): “Describe, in a number of distinct themes, the economic impacts of removing trade barriers between mainland China and Taiwan towards Hong Kong ” Theme identification method (Chen et al., 2001) –Pilot test: 3 subjects used up all the time in most tasks  only focused on effectiveness but not efficiency

13 10 Tasks in the Experiment (1 hour) Subject’s Origin ToolSettingHong KongTaiwanChina CBizPortBasic searching (with neither summarizer nor analyzer) SO1SO2SO3 BO1BO2BO3 Basic searching + with summarizer only SM1 BM1 Basic searching + with categorizer only SA1 BA1 Regional Chinese SE General searching and browsing SG1 BG1 Cross-regional searching and browsing SC1SC2SC3 BC1BC2BC3 S = search task; B = browse task; O = Basic searching (with neither summarizer nor analyzer); M = Basic searching + with summarizer only; A = Basic searching + with categorizer only; G = General searching and browsing; C = Cross-regional searching and browsing; same number signals the same question across different regions (Random assignment of tasks is used for different settings)

14 Comparisons Search Browse Search Browse Openfind YahooHK Sina.com or CBizPort With or without summarizer With or without categorizer Compare

15 Subjects 30 subjects, 10 from each region, were recruited –Rationale: equal influence of regional impacts Each subject used CBizPort and another search tool according to his/her origin Subject’s originSearch toolCBizPort version Hong KongYahooHKTraditional Chinese TaiwanOpenFindTraditional Chinese Mainland ChinaSina.comSimplified Chinese

16 Experts Three experts, one from each region, were recruited to provide answers to all browse tasks –First, the experts identify the set of relevant answers (organized into themes) to a browse task –Then, they modified the answers by adding some of subjects’ responses that they judged as relevant –The above two steps are repeated for all the other browse tasks Bla bla bla

17 Hypotheses Three sets of hypotheses were tested –CBizPort’s Enhanced Analysis Capabilities Searching and browsing With or without summarizer/categorizer –SE Performance Comparison Searching and browsing capabilities Individual settings and combination* –Users’ Subjective Evaluation Information quality cross-regional searching capability overall satisfaction –Auxiliary hypotheses: Performance of the three regions are not significantly different We tried to mimic a situation that each subject was allowed to use both CBizPort and benchmark search engine together to solve the same problem

18 CBizPort Experts’ answers Benchmark SE

19

20 Performance Measures Accuracy = Percentage of correct answers Precision = number of correct themes identified by users / total number of themes identified by users Recall = number of correct themes identified by users / total number of themes identified by an expert F value = 2*Recall*Precision / (Precision + Recall) Information quality: accessibility, appropriateness of amount, believability, completeness, …, etc. (Wang & Strong, 2002) Subjective evaluation: cross-regional searching capability, overall satisfaction, protocol analysis, post- hoc test (to study whether the three SEs yield significantly different results)

21 Accuracy of search tasks

22 Precision of browse tasks

23 Recall of browse tasks

24 F value of browse tasks

25 Information Quality

26 Users’ Subjective Evaluation

27 Subjects’ Verbal Comments Subjects liked summarizer and categorizer –Subj.#15: “… good performance in summarization and categorization, more focused results can be found”; #26: “… very handy”; #6: “…useful tools to enhance the searching ability” (11 subjects) CBizPort provides a wide coverage and variety of searching options –Subj.#2: “… Yahoo Search Engine is more limited when search certain term in a specific region … While CBizport can fulfill what Yahoo couldn’t do.”; #4: “… more search engines to choose from” (4 subjects)

28 Subjects’ Verbal Comments (2) Subjects are familiar with benchmark SEs –Subj#27: “I am familiar with the format of Openfind. So that's the reason that I am more satisfied with it than CBizPort.”; (4 subjects) Benchmark SEs are not good at cross- regional information searching –Subj#15: “Sina gives many results but they are not focused, and is poor at searching HK and Taiwan results”; #5: “ provide more accurate regional searching ” CBizPort is user friendly but slow –#3: “Yahoo not as precise as CBizPort”; #28: “… easier to search” (7 subjects); “slow” (3 subjects)

29 Conclusions CBizPort’s summarizer and categorizer provide helpful analysis capabilities for users’ search and browse tasks –CBizPort’s searching and browsing performance is comparable to that of regional Chinese search engines CBizPort can significantly augment the searching and browsing ability of regional Chinese search engines, thus improving human integration of regional information and analysis –Information quality, cross-regional searching capability and overall satisfaction of CBizPort are comparable to those of regional Chinese search engines CBizPort is better than regional Chinese search engines in terms of analysis functions, cross-regional searching capabilities and user-friendliness