Evaluating Visual and Statistical Exploration of Scientific Literature Networks Robert Gove 1,3, Cody Dunne 1,3, Ben Shneiderman 1,3, Judith Klavans 2, and Bonnie Dorr 2,3 1 HCIL, 2 CLIP, 3 Computer Science University of Maryland, College Park, USA

How do we help users explore scientific literature domains?

Commonly available capabilities Document statistics Keyword summary Citation visualization Corpus statistics Full-text search

Three types of systems Search engines & digital libraries Reference managers Summarization & recommender systems

Taxonomy of System Functionalities

Action Science Explorer Visual and statistical exploration

Evaluation Corpus of 147 papers on Dependency Parsing Exploration-based usability testing – 4 domain experts, two hours each 1.Training 2.Predefined tasks 3.Exploration questions 4.Participant-defined tasks

1. Training minutes Showed videos demonstrating Action Science Explorer Answered participant questions Gave think-aloud instructions

2. Predefined tasks Number of participants Capability Task 1: Identifying important authors and papers Task 2: Collecting evidence Attribute Ranking ███ 3██ 2 Find Communities █ 1 Plot Nodes In-Cite Text █ 1 In-Cite Summary Search █ 1 Sorting █ 1 Groups ██ 2█ 1

3. Participants’ questions How has the field evolved? What are subtopics within the field? What are the most accessible papers? What are the relationships between authors? What are the methods and applications?

4. Participant-defined tasks Number of participants Capability All participant-defined tasks Attribute Ranking ██ 2 Find Communities █ 1 Plot Nodes In-Cite Text █ 1 In-Cite Summary Search █ 1 Sorting █ 1 Groups █ 1

Answering participants’ questions 2/4 questions were answered – 1 unanswered question was uncompletable – 1 unanswered question not well supported

Design recommendations for exploration Confirmed design decisions – Provide metrics for ranking documents – Support user-created groups and document annotation – Give overview of corpus Suggested new capabilities – Allow control over corpus refinement – Incorporate undo/redo functionality

Conclusion A taxonomy of capabilities in current tools An evaluation of Action Science Explorer A list of literature exploration questions A list of system design recommendations Robert Gove -

Acknowledgements Supported by NSF grant IIS and Booz Allen Hamilton Dragomir Radev and Vahed Qazvinian, University of Michigan David Zajic and Michael Whidby, University of Maryland