Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman & Desney Tan oronto r

In situ search for digital ink InkSeine :  Ink +  Seine  -n: a fishing net  -v: to fish with a seine  = insanely cool ink search! InkSeine lets the user fish for useful information directly from ink notes. Fishermen catching salmon on the Columbia River using a seine. (courtesy Wikipedia)

Active note taking XLibris—pen-based system for active reading (reading + critical thinking + learning) InkSeine combines pen-based note taking with searching, linking, collecting & sensemaking:  Sketch designs  Reflect on a topic  Capture & extend creative ideas

Core interaction pattern: Ink-Search-Gather Ink – excels at capturing user’s thoughts & ideas with minimum distraction from formatting issues Ink Search Gather

Ink-Search-Gather interaction pattern Search – digging through the file system on a tablet is really a pain  Active note taking benefits from low overhead access to supporting materials & related docs  Focus on personal search, but support Web as well Ink Search Gather

Ink-Search-Gather interaction pattern Gather – Make it easy to use what you find by pulling content directly into unstructured notes  Thumbnail hyperlinks to documents  Clippings from documents  Queries that can be revisited later Ink Search Gather

Ink-Search-Gather interaction pattern InkSeine unifies these three elements in a fluid user experience that is integrated directly with inking Ink Search Gather

In situ search Leverage preexisting ink to initiate search  No context switch to “search app” – stay in the flow  No tedious transcription of text to a “search box” Queries as first class objects, commingled with ink notes  Flag some ink for search for later  Visible and salient search history as part of notes  Queries can be copied, pasted, moved, etc. Interleave inking, searching & gathering  No barrier between inking / searching – note ideas while searching, side-by-side searching, span app boundaries for content gathering Tightly couple queries with application content  Queries naturally persist in originating context  Easy re-visitation of prior queries  Results of queries become new content

Rich tradeoffs in cost structure of sensemaking Continuum of time cost – can defer at any point:  1 second: Capture thought “I should search on this”  5 seconds: Trigger search & see initial results list  s: Open a doc from result list or drag result into notes  20 s-2 minutes:  Scroll through list and inspect details of results  Apply filters, revise query  Grab a snapshot from a short document  Correct occasional ink reco errors – Tablet’s reco is amazingly good!  Longer: sufficient depth for common info needs; not mired by last 10% of complex queries that require full featured search tools  Inspect result documents to see if they meet information needs  Sideways searches based on Title, Date, or Author of a result

Video

Paper Prototype

Findings Existing desktop search tool difficult to use  Difficult text entry  Enter and correct text

Findings Existing desktop search tool difficult to use  Difficult text entry  Enter and correct text  Annoying context switch and flow interruption  Switch among apps and context

Findings Existing desktop search tool difficult to use  Difficult text entry  Enter and correct text  Annoying context switch and flow interruption  Switch among apps and context  Tedious information gathering  Difficult to link to documents and gather pieces of information

Notebook Control

In Situ Search

Breadcrumb

Trigger Personal Search

Search Panel

Query Area (CueTIP)

Result List

Focus Result

File and Timeline Filters Filters

Gestures for Filters Tap - Select Cross – Multi-select Disjunctive cross – Multi-select Cross & Turn Right – Exclude

Directed Work Flow Selection Correction Search

Occlusion

Crossing Widgets

Resize Search Panel

Examine Document

Document & Tracking Menu

Multiple Search Panels

Design Issue 1: Discovery Gesture discovery problem Highlighter hints

Design Issue 2: Occlusion Context preservation & occlusion problem Smart positioning of search panel

Design Issue 3: Poor Reco Recognition problem for out-of-vocabulary words Personalized vocabulary from desktop search index UIST List?, Dist?, Gist?, Mist?, Hist? ?

Summary In situ search experience Tailored to pen input Optimum workflow / maximum flexibility Multiple ways to gather content Minimize search screen real estate Span application boundaries

Future Directions Pull in materials for searches from notes

Future Directions Pull in materials for searches from notes Support vector-based searches

Questions?