Microsoft Research Faculty Summit 2007

When Is The Pen Mightier Than The Keyboard? Andries van Dam Professor of Computer Science Brown University

4 Ivan Sutherland’s Sketchpad (1963) Bert Sutherland’s Logic and circuit diagrams (1966) First Wave: Research ‘60s and ‘70s Setting context History Research demos

5 Robert Anderson’s 2D Math Reco on Rand Tablet (1967) First Wave: Research ‘60s and ‘70s Setting context History Research demos

6 Alan Kay’s Dynabook (1968) Setting context History Research demos First Wave: Research ‘60s and ‘70s

7 Wang Freestyle (1988)GRIDPad (1989) Newton (1993) GO +PenPoint (1991) Slate (1992) PenWindows (1991) Palm Pilot’s Graffiti (1994) Anoto (1999) Setting context History Research demos Second Wave: Commercialization (‘80s and 90’s)

8 RIP (Adapted from Bill Buxton) Freestyle Palm Pilot’s Graffiti (1994) Anoto (1999) Grid Pen for Windows GO Slate Newton Crosspad Setting context History Research demos

9 Third Wave: Research And Deployment (2000 …) Much more powerful hardware Tablet PC Digitizers Wacom Cintiq Smartboard Much improved software support Tablet SDK Handwriting recognition Speech recognition Character recognizers Better recognition algorithms Machine learning (use those cycles!) Setting context History Research demos

Pen-Centric Computing: My Definition Takes advantage of the pen More than a high resolution mouse for picking and trailing digital ink Interprets digital ink in appropriate context for character/symbol, gesture, and diagram recognition Best if used within a multi-modal interface, e.g., with speech recognition Uses pen, finger, multi-touch, … HP TouchSmart IQ770 Smartboard Microsoft Surface Setting context History Research demos

Music Notepad 11 Distinguished Lecture - 15 March Setting context History Research demos

12 Demo: MathPad2 Overview Mathematical sketching Combine handwritten math, simple FoC, and freeform diagrams 2D math expression recognizer Graphing; symbolic and numeric solutions Uses MATLAB® as underlying math engine Diagrams animate according to associated math expression(s) Fully gestural interface for editing Expressions can be deleted, edited, and re-recognized Modeless operation Setting context History Research demos

13 MathPad2 Sketch-based interface dataflow Raw Stroke DataPreprocessingSegmentation FeatureExtractionAndAnalysis Classification Parsing SketchUnderstanding Make Inferences Setting context History Research demos

Math Error Visualization Techniques Always typeset After interaction pause, show typeset Can disrupt input, but is best for reading Adjusted handwriting Replace with clear handwriting after pause Less disruptive, but less clear Typeset offset Interactive fit-to-width typeset below ink Easy to read, but takes up a lot of space Typeset mini Interactive fixed-size typeset below ink Harder to read, but less distracting Setting context History Research demos

15 Demo: ChemPad Overview Learning tool for Organic Chemistry Students sketch molecules to generate 3D structures 150 users at Brown in last two years Free download Work in progress Reaction helper Interface for chemists Contemporary molecule drawing format Export conversion for external chemistry software Setting context History Research demos

16 ChemPad Architecture Ink Recognition System 3D Model Builder Reaction Engine 3D ViewerInk Collector ChemPad User Interface Draws a molecule sketch Sketch is passed from collector to recognizer Parsed sketch is created Parsed sketch is passed to model builder 3D model of molecule is generated Model is displayed to user Chemist gains insight Student Chemist Ink Recognition System 3D Model Builder Reaction Engine Setting context History Research demos

Distinguished Lecture - 15 March

Reaction Engine Ink Recognition System 18 ChemPad Architecture 3D Model Builder ChemPad User Interface Chemist gains insight Makes a gesture at the functional group Gesture is passed to recognition system Reaction Engine is queried Student Chemist User selects reaction Options are presented to user 3D Model Builder Setting context History Research demos

MSpcc An invitation Center to be international resource for pen-computing community Open source software Website as portal for information Institutions and people Projects and applications Publications, demos, and tutorials Research Wiki Workshops We invite your inquiries and contributions!

20 “To Infinity And Beyond…”

© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.