1 Pen-Centric Shorthand Interfaces Charles C. Tappert Seidenberg School of CSIS, Pace University

2 Themes of Presentation Online Handwriting Recognition and Pen Computing Tutorial Historical Research – undertaken for the Palm-Xerox Patent Infringement Lawsuit Recent Research - Enhanced Pen-Centric Shorthand Interfaces can have benefits DPS dissertation could extend M.S. thesis

3 Enhanced Pen-Centric Shorthand Interfaces Can use word/phrase shorthand to speed text input Can provide critical infrastructure for many pen-centric applications Can enhance natural pen-centric interactions for many applications Will have greatest impact on the utility of applications running on small mobile devices

4 Part 1: Online (Pen-Centric) Handwriting Recognition Written Languages and Handwriting Properties The Fundamental Property of Writing Handwriting Recognition Difficulties Online (Pen-Centric) Handwriting Recognition Online more accurate than Offline Recognition Online Info Can Complicate Recognition Process Design Tradeoffs / Design Decisions Computer Problems in English

5 Written Language and Handwriting Properties Alphabet Letters, digits, punctuation, special symbols Writing is a time sequence of strokes Stroke – writing from pen down to pen up Usually complete one character before beginning the next Spatial order – e.g., in English left to right

6 Fundamental Property of Writing Differences between different characters are more significant than differences between different drawings of the same character This makes handwritten communication possible Can there be exceptions – say, different characters written identically?

7 Fundamental Property of Writing in English Property holds within subalphabets of uppercase, lowercase, and digits, but not across them “I”, “l”, and “1” written with single vertical stroke “O” and “0” written similarly with an oval

8 Handwriting Recognition Difficulties Shape, size, and slant variation Similarly shaped characters – U and V Careless writing in the extreme, almost illegible writing Resolving difficult ambiguities requires sophisticated recognition algorithms, syntax/semantics

9 Electronic tablets invented in late 1950s Digitizer and display in separate surfaces Pen Computers arrived in 1980s Combined digitizer and display Brought input and output into one surface Immediate feedback via electronic ink Created paper-like interface Online (Pen-Centric) Handwriting Recognition

10 Tablet Digitizers – Dynamic Information Pen down – indication of inking X-Y coordinates as function of time Sampling rate: 100 points per second Resolution: 200 points per inch

11 Early Pen-Centric Interface Different surfaces for input and output Rand system, about 1959

12 Pen Computers IBM vision Paper-like interface, 1992 Microsoft Tablet PC Launched, 2001

13 Pen-Centric PDAs Early Palm Pilot Palm Tungsten T3 and Sony Clié TH55

14 Online (Pen-Centric) Handwriting Recognition Machine recognizes the writing as the user writes Digitizer equipment captures the dynamic information of the writing Stroke number, order, direction, speed A stroke is the writing from pen down to pen up

15 Online (Pen-Centric) more accurate than Offline (Static) Recognition Can use both dynamic and static information Can often distinguish between similarly shaped characters E.g., 5 versus S where the 5 is usually written with two strokes and the S with one stroke

16 Online Information Can Complicate Recognition Process Large number of possible variations E can be written with one, two, three, or four strokes, and with various stroke orders and directions A four-stroke E has 384 variations (4! stroke orders x 2 4 stroke directions)

17 Online Information Can Complicate Recognition Process Other variations

18 Online Information Can Complicate Recognition Process Segmentation ambiguities character-within-character problem lowercase d might be recognized as a cl if drawn with two strokes that are somewhat separated from one another

19 Design Tradeoffs/Decisions No constraints on the user Machine recognizes user's normal writing User severely constrained Must write in particular style such as handprint Must write strokes in particular order, direction, and graphical specification

20 English Writing Styles Handprint Uppercase – about 2 strokes per letter Lowercase- about 1 stroke per letter Cursive Script Usually less than 1 stroke per letter Delayed crossing and dotting strokes

21 Computer Problems in English Constrained Handprint Printing one symbol per box – form filling Printing on lines – symbols can touch or overlap Unconstrained Handprint No lines and symbols can touch or overlap Cursive Script Mixed Printing and Cursive

22 Computer Problems in English

23 Part 2 Shorthand in Pen-Centric PDAs Famous Uses of Shorthand Historical Shorthand Alphabets Pen-Centric Shorthand Alphabets Pen-Centric Word/Phrase Shorthand Allegro/Chatroom Shorthand System M.S. thesis that could be extended into a DPS dissertation

24 Background Famous writings throughout history were effectively written in a style of shorthand Cicero’s orations Martin Luther’s sermons Shakespeare’s and George Bernard Shaw’s plays Samuel Pepys’ diary Sir Isaac Newton’s notebooks

25 Historical Shorthand Alphabets We first review the history of shorthand systems prior to pen computing Shorthand is “a method of writing rapidly by substituting characters, abbreviations, or symbols for letters, words, or phrases” Shorthand can be traced back to the Greeks in 400 B.C.

26 Historical Shorthand Alphabets We focus on shorthand alphabets that might be appropriate for PDAs We review two types of shorthand Geometric shorthand Small number of basic shapes Shapes reused in multiple orientations Non-geometric shorthand shorthand

27 Historical Shorthand Alphabets Ancient Greeks – 400 BC Tironian Alphabet – 63 BC John Willis’s Stenography – 1602 Gabelsberger Alphabet – 1834 Moon Alphabet – 1845

28 Tironian Alphabet, 63 B.C. Non-Geometric

29 Stenography Alphabet, 1602

30 Stenography Alphabet, 1602 Basic Shapes and Orientations

31 Gabelsberger Cursive-Style, 1834 Non-Geometric Alphabet

32 Moon Geometric Alphabet, 1845

33 Other Historical Shorthand Systems Phonetic alphabets Pitman (1837), was popular in UK Gregg (1888), was popular in USA Systems for the blind Braille (1821)

34 Pen-Centric Shorthand Alphabets Some of the earliest were for CAD/CAM symbols represent graphical items and commands Others developed for text input on small consumer devices like PDAs that have limited computing power We review geometric and non-geometric shorthands appropriate for small devices

35 Pen-Centric Shorthand Alphabets Historical alphabets presented above could be used for machine recognition symbols drawn with a single stroke (except “K” in Tironian and “+” in Stenography) In addition to shape and orientation, online systems can use stroke direction to differentiate among symbols

36 Pen-Centric Shorthand Alphabets Geometric Pen-Centric Shorthands Organek – 1991 Allen – filed 1991, patent 1993 Goldberg (Xerox) – filed 1993, patent 1997 Non-Geometric Pen-Centric Shorthands Graffiti (Palm Computing) – 1995 Allegro (Papyrus) – 1995

37 Organek Alphabet, 1991

38 Organic Alphabet, 1991 Basic Shapes and Orientations One shape in 4 orientations. This gives 8 directions that together with 3 lengths provide 24 symbols. A second wheel provides additional symbols.

39 Allen patent, filed 1991

40 Allen patent, filed 1991 Basic Shapes and Orientations

41 Goldberg patent, filed 1993 (“unistroke symbols”)

42 Goldberg patent, filed 1993 Basic Shapes and Orientations

43 Goldberg patent, filed Basic shapes 4 Orientations 2 Stroked Directions 40 Possible Symbols Designed for Speed of Input and Maximum Symbol Separation

44 Shorthand Alphabet Design How would you design a shorthand alphabet? What would be the design criteria?

45 Design of Graffiti Alphabet for the Palm Pilot Small alphabet Uppercase, digits, special symbols One stroke per symbol to avoid segmentation difficulty Separate writing areas for letters and digits to avoid same-shape confusions

46 Graffiti Alphabet, 1995

47 Graffiti Mimics Keyboard Input Character by character input Mode shifts for Uppercase Special characters Eyes can focus on application’s insertion point rather than on input area

48 Graffiti Alphabet Design What was the additional design criterion?

49 Graffiti Alphabet Design Designed for ease of learning 20 letters exactly match the Roman alphabet 6 remaining ones match partially

50 Graffiti Alphabet: 11 of 26 characters have alternate variations

51 Frequently Confused Characters

52 Other Low Performance Characters

53 Symbol Overlap Comparison

54 Graffiti Recognition Accuracy Study

55 Allegro Alphabet (Papyrus), 1995 (now Microsoft)

56 Simplified Design Tradeoffs/Decisions for Graffiti and Allegro PDA Alphabets Small alphabet one case rather than both upper and lowercase One stroke per character (character = stroke) allows machine to recognize each character upon pen lift Small number of writing variations per letter preferably only one Separate writing areas for letters and digits avoids confusion of similarly shaped letters and digits High correspondence to Roman alphabet for ease of learning non-geometric, might not actually qualify as shorthand

57 Commercially Successful Shorthands Similar to the Roman alphabet Easy to learn Graffiti used in Palm OS devices notably the Palm Pilot & Handspring models Allegro used in Microsoft Windows devices Geometric alphabets not successful

58 Current Commercial Systems Company/SystemWriting Style Palm Computing/Graffiti*Special Shorthand Alphabet Microsoft/Papyrus AllegroSpecial Shorthand Alphabet CIC/JotRelatively Unconstrained Handprint Microsoft Relatively Unconstrained Handprint and Cursive *A few years ago Palm switched from Graffiti to Graffiti2, Graffiti2 is basically Jot licensed from CIC.

59 Jeff HawkinsJeff Hawkins, innovatorinnovator 1979 BSEE Cornell, worked at Intel and GRiD ABD BioPhysics doctoral program, U.C. Berkeley back at GRiD he created GRiDPAD, first pen computer 1992 formed Palm Computing, 1993 created first handwriting reco software product for a mobile handheld - Casio’s Zoomer 1995 Palm Computing bought by U.S. Robotics 1996 created PalmPilot, first PDA with Graffiti shorthand alphabet (over a million shipped in 18 months, a 66% market share, and the fastest growth of any computing product in history, faster than the TV and the VCR) 1997 U.S. Robotics bought by 3Com (sued by Xerox for patent infringement) 1998 left Palm to form Handspring, 1999 launched the Visor handheld 2000 Palm Computing spun off by 3ComPalm Computing 2002 created what is now the Redwood Center for Theoretical NeuroscienceRedwood Center for Theoretical Neuroscience 2003 Handspring (with Hawkins, et al.) acquired by Palm Computing 2005 Founded Numenta to build the ultimate brain-like machineNumentaultimate brain-like machine

60 Palm-Xerox Patent Infringement Lawsuit The nine-year old battle between Palm and Xerox over handwriting recognition ends in 2006, see article.see article Palm pays Xerox $22.5 million for a fully paid-up license for Xerox patents covering its text input Unistrokes technology Xerox first sued Palm predecessor Palm Computing back in April 1997, claiming that the Graffiti text-entry system used in its PDAs infringed on patents for Unistrokes, which allows users to input letters and numbers into personal data units with basic, one stroke movements.

61 Conclusions Palm-Xerox Patent Infringement Lawsuit Invalidity Historical research showed that Goldberg alphabet not so unique Even though the patent was accepted as valid, these arguments narrowed the scope of the patent Infringement Analyses and comparisons of the Goldberg and Graffiti alphabets showed major differences Result was favorable settlement for Palm

62 Pen-Centric Word/Phrase Shorthand such as Chatroom Shorthand Further increase speed of text entry Potential applications Where input speed important Where word/phrase abbreviations occur frequently – e.g.,

63 Chatroom Shorthand Examples CUSee you, or Cracking up CMCall the end of the day ^5High five 2niteTonight LOLLaughing out loud ASAPAs soon as possible B/C or BCBecause

64 Allegro/Chatroom Shorthand System Developed for M.S. dissertation Student was hearing impaired Developed as output component of communication system Handwriting to text to speech Two input writing areas One for Allegro (all-purpose) One for chatroom-like or user-defined words/phrases

65 Allegro/Chatroom Shorthand System

66 Allegro/Chatroom Shorthand System

67 Allegro/Chatroom Shorthand System M.S. Thesis Experimental Results Allegro/Chatroom pen-centric shorthand input is faster than typing text and is comparable to typing text and chatroom shorthand characters

68 Conclusions Pen-Centric Shorthands Pen-centric interfaces should use shorthand, and especially word/phrase shorthand where appropriate, for fast text input Benefit of shorthand interfaces Provides critical infrastructure for many pen-centric applications Enhances natural pen-centric interactions for many applications Has greatest impact on the utility of applications running on small mobile devices

69 Conclusions Handwriting Recognition Graffiti and Allegro greatly simplified the recognition problem Handprint problem not completely solved Even with IBM’s ThinkWrite, CIC’s Jot, and Microsoft products Cursive script problem clearly not solved

70 References W.B. Huber, S.-H. Cha, C.C. Tappert, and V.L. Hanson, "Use of Chatroom Abbreviations and Shorthand Symbols in Pen Computing," Proc. 9th Int. Workshop on Frontiers in Handwriting Recognition, IWFHR 2004, Tokyo, Japan, October W. Huber, V. Hanson, S. Cha, and C.C. Tappert, "Common Chatroom Abbreviations Speed Pen Computing," Proc. 11th Int. Conf. Human- Computer Interaction, Las Vegas, NV, July C.C. Tappert and S. Cha, "Handwriting Recognition Interfaces," Chapter 6, pp , in Text Entry Systems, Scott MacKenzie and Kumiko Tanaka-Ishii (Eds.), Morgan Kaufmann, C.C. Tappert, C.Y. Suen, and T. Wakahara, "The state-of-the-art in on- line handwriting recognition," IEEE Trans. Pattern Analysis Machine Intelligence, Vol. PAMI-12, pp , August C.C. Tappert and J.R. Ward, "Pen-Centric Shorthand Handwriting Recognition Interfaces," Proc. 1st Int. Workshop on Pen-Based Learning Technologies, Catania, Italy, May 2007.