Tablet PC Platform Advanced Topics Software Design Engineer/Test Lead

Topics 1. Tablet Hardware API 2. Overview of the Tablet SDK a. Ink, Stroke and related objects b. Renderer 3. Clipboard Support 4. Ink Recognition 5. Gesture Support

Tablet PC Platform Hardware API

Examining Tablet Hardware Electromagnetic digitizers High resolution, High throughput There are two types of digitizers Integrated and Non-Integrated Multiple Digitizers are supported The mouse is also represented as a digitizer Digitizers support various capabilities # of stylus buttons, pressure, angle or rotation attributes Ink Stroke PacketData varies between digitizers

Tablet Hardware API Tablet Object representation of a digitizer Query for digitizer’s supported properties Get property metrics E.g.: How many levels of pressure are supported? Tablets Returns collection of all connected digitizers Including the mouse

Tablet Hardware API Cont. Cursor Represents a stylus Allows for access to stylus properties Inverted, Button state, Name Available in all InkCollector/Overlay events CursorInRange / CursorOut0fRange CursorDown / CursorUp Cursors Collection of Cursor objects

Ink Overlay & Ink Collector Main function is to collect ink Ink Collector provides core functionality Ink Overlay is an Ink Collector with the addition of easy to use Editing Modes (Ink, Select, Erase) Contains an Ink object Full set of event notifications

InkCollector Top level object for collecting ink Attaches to any HWND Handles input from all attached Tablets (digitizers) and Cursors (pens) Intercepts cursor movements CollectionMode property Ink Only Gesture Only Ink and Gesture

InkOverlay Superset of InkCollector Adds Editing Modes Ink – Real-time inking mode Select – Selection mode Delete – Eraser mode (both point and stroke)

Ink Collection Events CursorInRange/CursorOutOfRangeNewInAirPacketsCursorDownNewPacketsStrokeGestureSystemGesture Mouse Events

Ink Object Primary Data Object in the Tablet Platform Container for: Stroke data Metadata Controls all means of persistence Extensible via ExtendedProperties collection

Stroke Object Represents a single stroke Single pen down, pen move, pen up sequence ExtendedProperties Allows applications to access and store custom data DrawingAttributes Controls the rendering of the stroke Rich API PacketData, BezierPoints, PolylineCusps, SelfIntersections

Strokes Collections What are Stroke collections? Collections of references to strokes Not the stroke data itself Important so you can pass a subset to the Recognizers Stroke Collections can share strokes Both Stroke collections include Stroke 3 Strokes Stroke (1) Strokes Stroke (2)Stroke (3)Stroke (4)Stroke (5)

DrawingAttributes Object Controls the appearance of Ink Color, width, transparency Smoothing, anti-aliasing Height, width Pen Tip: Ball or Rectangle DefaultDrawingAttributes property Specifies the DrawingAttributes for all Cursors Can be applied to Stroke Strokes collection Cursor (Pen)

Renderer Object Two coordinate systems on a Tablet PC Device coordinates Ink coordinates (HIMETRIC units) Renderer object Map between coordinate systems Draw methods for rendering Manipulation methods Object and View transforms supported Helper methods: Scale, Move, Rotate

Clipboard Support

Use the Ink API to copy and paste ClipboardCopyClipboardPaste Multiple formats supported ISF – Ink Serialized Format HTML Bitmap, Metafile Applications interacting with the clipboard can choose the format

Ink Recognition Interpret pen movements and/or strokes as Text (Handwriting) Gestures Shapes and symbols Language Support Synchronous and Asynchronous recognition supported Provides a full set of recognition results Extensibility architecture

Recognizer Context Object for accessing handwriting recognizers Binds ink data to a particular language for recognition Performs synchronous or asynchronous handwriting recognition Recognition results and alternates returned Attributes used to bias recognition results

RecognitionResult Recognizers contain a set of structures Each structure called a “lattice” TopString property TopConfidence property TopAlternate property RecognitionAlternates object GetAlternatesFromSelection method Access to alternate paths through the lattice

Gesture Support

Gestures Collected through InkCollector, InkOverlay, or RealTimeStylus Two Types: System or Application System Gestures Mimic mouse events Cannot be disabled Application Gestures Can be single or multi-stroke Can be turned on/off individually

System Gestures Available Gestures TapDoubleTapRightTapDragRightDragHoldEnterHoverEnterHoverLeave Coordinates are Window Coordinates Can’t be cancelled

Application Gestures CollectionMode determines support InkOnlyInkAndGestureGestureOnly Single vs. Multi-Stroke Gestures Inform your users about gesture support Good UI feedback Consistent use (i.e. Scratchout)

Gestures Focus: Scenario driven: don’t over useScenario driven: don’t over use

The Tablet and Mobile PC Developer Center

Tablet PC Developer Center The latest technical articles Downloads Developer resources Regular columns.Newsgroups Featured partners and more……

Attend a free chat or web cast List of newsgroups MS Community Sites Locate Local User Groups and Community Sites

© 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Appendix

RealTimeStylus Plug-in Interfaces

RealTimeStylus 1. Performance De-couples rendering and collection of ink 2. Flexibility You can control exactly how and when the ink is collected and rendered 3. Extensible You can create your own plug-ins to render and collect ink

RealTimeStylus 1. Performance Gives you the opportunity to ensure users see no lag when creating ink Collection of stylus input is moved off the UI thread and is collected asynchronously Provides direct access to stylus input even before it is rendered Queued events allow for separation of Real Time Collection thread and UI thread Minimizes thread usage (From 5 per instance to 1 per process)

RealTimeStylus Focus: No lag renderingNo lag rendering

RealTimeStylus 2. Flexibility Plug-in based architecture You determine how you render and collect ink based on which plug-ins you choose Plug-ins can be dynamically added, removed or re-ordered

RealTimeStylus 3. Extensible Allows for the creation of new Real Time components Custom dynamic rendering, gesture recognition, disabled regions, etc. Customize your ink collection and rendering to suit your need

Architecture (Overview) Pen Service RealTimePlugIn (IStylusSyncPlugin) RealTimeStylus queue RealTimePlugIn (IStylusSyncPlugin) “Inking” Thread UI Thread Ink Collecting Object (IStylusAsyncPlugin) storage InkCollector

RealTimeStylus Plug-ins Object that implements a synchronous or asynchronous interface SyncPlugin: fast custom rendering AsyncPlugin: ink data collection Can manipulate packet data Read, Modify, Add, or Ignore Tablet OS includes two plug-ins Dynamic Rendering Gesture Recognition SDK includes two sample plug-ins Custom Rendering Constrained Region

RealTimeStylus Plugin Focus: Flexibility in rendering inkFlexibility in rendering ink Flexibility in collecting stroke dataFlexibility in collecting stroke data

Journal Note Reader

Provides Read-only access to Journal Notes Exposes all attributes in Journal files Ink, Images, Title, Stationary, etc. Enables your application to consume common note files JNT file is presented as an XML document Native and Managed versions DownloadDownload as an SDK update Download

Journal Reader with Ink Divider Focus: Easy to consume Journal filesEasy to consume Journal files