9/20/20151 Interaction Devices CIS 577 Bruce R. Maxim UM-Dearborn

9/20/20152 User Communication Keyboard remains the user’s primary input device Most GUI systems expect pointing devices as well –Light pen (direct) –Touch screen (direct) –Mouse –Trackball –Joystick –Graphics tablet –Touch pad –Foot controls (novel) –Gloves (novel) –Eye movement detectors (novel)

9/20/20153 Pointing Device Tasks Selection Positioning objects Orienting objects on screen Path tracing Quantify Text entry/editing (not easily done)

9/20/20154 Direct vs Indirect Light pens and touch screens allow the fastest user responses, but are the least accurate (finger vs pixel level resolution) Graphics tablets with a stylus allows good resolution and long periods of use without switching to the key board (with good software design – often menu driven)

9/20/20155 Comparison Keyboard –Fast control, require memorization and recall Mouse –Best choice for pointing to objects in arbitrary positions, can be as fast as pointing with a finger Joystick –Poor cursor control devices, actually slower than mouse (only use where they are game realistic)

9/20/20156 Speech Technology Discrete word recognition Continuous speech recognition Speech store and forward Speech generation

9/20/20157 Discrete Word Recognition 90 to 98% reliability for small vocabulary Usually requires speaker dependent training Most people would rather type than dictate

9/20/20158 When should you use it? Speakers hands are busy Mobility required Speakers eyes are occupied Harsh or cramped conditions prevent use of key board

9/20/20159 Speech Store and Forward Voice mail type technology Video games Low cost Resource intensive

9/20/ When to use computer generated speech? Message is simple Message is short Message will not be referred to later Message deals with events in time Message requires immediate response Visual communications channels are overloaded Environment lighting is bad User must move around User subjected to high G forces or lack of oxygen

9/20/ Display Technologies CRT –raster scan cathode ray tube LCD –flicker free, lightweight, limited viewing angles Plasma –early touch screens LED –some resolution limitations Projectors –room lighting and resolution restrictions

9/20/ Video Display Features Rapid operation Reasonable size (24 by 80 or 66 by 166 chars) Reasonable resolution (768 by 1024 minimum) Quiet operation No waste paper Low cost (< $100) Reliable Good for highlighting Good for graphics and animation

9/20/ Image Technologies Scanners (< $100) Digital cameras (< $200) Digital video (< $200) 12 inch video disk CD ROM (600 MB data, 1 hour music) DVD (3 GB data, 2 hours video) Motion Capture

9/20/ Hardcopy Devices Printers –ink jet, laser, thermal, dot matrix Plotters –seem to be vanishing (slow, high resolution) Photographic printers –ink jet technology, special paper

9/20/ Color Can soothe or strike the eye Adds accents to an uninteresting display Facilitates subtle discriminations in complex displays Emphasizes the logical organization of information Draws attention to warnings Can evoke strong emotional reactions of joy, excitement, fear, or anger

9/20/ Color Display Problems Colors can't be represented perfectly on digital computer systems People don’t have the same perceptions of colors Color blindness Eye fatigue can be caused by using bad color combinations

9/20/ Color Description Schemes Computer Industry Approach (RBG) –Red, Green, Blue (0 – 255), additive model Ink jet Approach (CMY or CMYK) –Cyan, Magenta, Yellow, subtractive model Broadcast TV (YIQ) –Y = luminence, I, Q = chromacity Psychological Dimensions (HSB or HSV) –Brightness (amount of light energy) –Hue (predominant wave length) –Saturation (color purity)

RGB Model

Note that for the "complementary" colors the sum of the values equals white light (1, 1, 1). For example: red (1, 0, 0) + cyan (0, 1, 1) = white (1, 1, 1) green (0, 1, 0) + magenta (1, 0, 1) = white (1, 1, 1) blue (0, 0, 1) + yellow (1, 1, 0) = white (1, 1, 1)

CMY Model

Cyan = Green + Blue, so light reflected from a cyan pigment has no red component, i.e., the red is absorbed (or subtracted) by cyan. Magenta subtracts Green Yellow subtracts Blue. Printers usually use four colors: Cyan, Yellow, Magenta and Black. Cyan, Yellow, and Magenta together produce a dark gray rather than a true Black.

CMY Conversions RGB to CMY C = 1 – R M = 1 – G Y = 1 – B CMY to RGB R = 1 – C G = 1 – M B = 1 – Y For printing (CKYK) K = min(C,M,Y) C = C – K M = M – K Y = Y – K Where K = Black

RGB to YIQ |Y| | | |R| |I| = | –0.321| * |G| |Q | |0.212 – | |B| For YIQ to RGB use inverse matrix

HSV Model

Pure Red is H=0, S = 1, V = 1 White is S = 0, V = 1 (H is irrelevant) Black is V = 0 (both H and S are irrelevant) Adding white is done by decreasing S and leaving V unchanged Changing shades is done by decreasing V and keeping S unchanged Create tones by decreasing both S and V

Using Color Effectively - 1 Limit total number of colors Recognize power of color to speed up or slow down a task Ensure color coding supports user task Have color coding appear with minimal user effort Place color coding under user control Design for good monochrome display first Use color to help formatting

9/20/ Using Color Effectively - 2 Use consistent color coding Be alert to user expectations for certain colors (e.g. red means stop or danger) Use color changes to indicate state changes Use color in graphic displays to increase information density Be alert to bad color pairings (e.g. saturated blue and red) Consider needs of color impaired users

9/20/ Direct-Input 1.Create main DirectInput interface using CreateIDirectIntput8( ) 2.(optional) Query for device GUID’s for all devices (mouse, keyboard, joystick) 3.Create each device using CreateDevice( ) please note: GUID_SysKeyboard and GUID_SysMouse are built in 4.Set cooperation level for each device using SetCooperativeLevel( ) 5.Set data format for each device using SetDatFormat( )

9/20/ Direct-Input 6.Set properties of each device using IDirectIntputDevice8::SetProperty( ) 7.Acquire each device using IDirectIntputDevice8::Acquire( ) 8.(optional) Poll devices using IDirectIntputDevice8::Poll( ) 9.Read device using IDirectIntputDevice8::GetStateDevice( )

9/20/ Input DirectX can request input in two ways –Immediate input (default) –Buffered input LaMothe only uses immediate input DirectX can cut Windows out of the message loop altogether, so be careful when you set each device cooperation level

9/20/ LaMothe Example

9/20/ Keyboard Global Declarations #include // directX includes #include #include "T3DLIB1.H" // directinput globals LPDIRECTINPUT8 lpdi = NULL; // dinput object LPDIRECTINPUTDEVICE8 lpdikey = NULL; // dinput keyboard LPDIRECTINPUTDEVICE8 lpdimouse = NULL; // dinput mouse LPDIRECTINPUTDEVICE8 lpdijoy = NULL; // dinput joystick GUID joystickGUID; // guid for main joystick char joyname[80]; // name of joystick // these contain the target records for all di input packets UCHAR keyboard_state[256]; // contains keyboard state table DIMOUSESTATE mouse_state; // contains state of mouse DIJOYSTATE joy_state; // contains state of joystick

9/20/ Game_Init( ) // initialize directdraw DDraw_Init(SCREEN_WIDTH, SCREEN_HEIGHT, SCREEN_BPP); // first create the direct input object if (DirectInput8Create(main_instance,DIRECTINPUT_VERSION, IID_IDirectInput8, (void **)&lpdi,NULL)!=DI_OK) return(0); // create a keyboard device ////////////////////////////////// if (lpdi->CreateDevice(GUID_SysKeyboard, &lpdikey, NULL)!=DI_OK) return(0); // set cooperation level if (lpdikey->SetCooperativeLevel(main_window_handle, DISCL_NONEXCLUSIVE | DISCL_BACKGROUND)!=DI_OK) return(0); // set data format if (lpdikey->SetDataFormat(&c_dfDIKeyboard)!=DI_OK) return(0); // acquire the keyboard if (lpdikey->Acquire()!=DI_OK) return(0);

9/20/ Game_Shutdown( ) // kill the reactor Destroy_Bitmap(&reactor); // kill skelaton Destroy_BOB(&skelaton); // release keyboard lpdikey->Unacquire(); lpdikey->Release(); lpdi->Release(); // shutdonw directdraw DDraw_Shutdown();

9/20/ Game_Main( ) // get player input // get the keyboard data lpdikey->GetDeviceState(256, (LPVOID)keyboard_state); // reset motion flag player_moving = 0; // test direction of motion if (keyboard_state[DIK_RIGHT] && keyboard_state[DIK_UP]) { // move skelaton skelaton.x += 2; skelaton.y -= 2; dx=2; dy =- 2; // set motion flag player_moving = 1; // check animation needs to change if (skelaton.curr_animation != SKELATON_NEAST) Set_Animation_BOB(&skelaton,SKELATON_NEAST); } // end if …

9/20/ Mouse Detects changes in X, y position (Mickeys) Specifies “absolute” position Can specify velcoity range Good for selecting objects, moving objects, or changing head position (view) Hard to center if used as joystick replacement Has small number of buttons (2 or 3

9/20/ LaMothe Example

9/20/ Mouse Global Declarations #define BUTTON_SPRAY 0 // defines for each button #define BUTTON_PENCIL 1 #define BUTTON_ERASE 2 #define BUTTON_EXIT 3 // directinput globals LPDIRECTINPUT8 lpdi = NULL; // dinput object LPDIRECTINPUTDEVICE8 lpdikey = NULL; // dinput keyboard LPDIRECTINPUTDEVICE8 lpdimouse = NULL; // dinput mouse LPDIRECTINPUTDEVICE8 lpdijoy = NULL; // dinput joystick GUID joystickGUID; // guid for main joystick char joyname[80]; // name of joystick // these contain the target records for all di input packets UCHAR keyboard_state[256]; // contains keyboard state table DIMOUSESTATE mouse_state; // contains state of mouse DIJOYSTATE joy_state; // contains state of joystick

9/20/ Game_Init( ) // first create the direct input object DirectInput8Create(main_instance,DIRECTINPUT_VERSION,IID_IDirectInput8, (void **)&lpdi,NULL); // create a mouse device ///////////////////////////////////// lpdi->CreateDevice(GUID_SysMouse, &lpdimouse, NULL); // set cooperation level lpdimouse->SetCooperativeLevel(main_window_handle, DISCL_NONEXCLUSIVE | DISCL_BACKGROUND); // set data format lpdimouse->SetDataFormat(&c_dfDIMouse); // acquire the mouse lpdimouse->Acquire();

9/20/ Game_Init( ) // set the global mouse position mouse_x = screen_height/2; mouse_y = screen_height/2; // load the master bitmap in with all the graphics Load_Bitmap_File(&bitmap8bit, "PAINT.BMP"); Set_Palette(bitmap8bit.palette); // make sure all the surfaces are clean before starting DDraw_Fill_Surface(lpddsback, 0); DDraw_Fill_Surface(lpddsprimary, 0); // create the pointer bob Create_BOB(&pointer,mouse_x,mouse_y,32,34,1, BOB_ATTR_VISIBLE | BOB_ATTR_SINGLE_FRAME,DDSCAPS_SYSTEMMEMORY); // load the image for the pointer in Load_Frame_BOB(&pointer,&bitmap8bit,0,0,2,BITMAP_EXTRACT_MODE_CELL);

9/20/ Game_Init( ) // set clipping rectangle to screen extents so mouse cursor // doens't mess up at edges RECT screen_rect = {0,0,screen_width,screen_height}; lpddclipper = DDraw_Attach_Clipper(lpddsback,1,&screen_rect); // hide the mouse ShowCursor(FALSE);

9/20/ Game_Shutdown( ) // first unacquire the mouse lpdimouse->Unacquire(); // now release the mouse lpdimouse->Release(); // release directinput lpdi->Release();

9/20/ Game_Main( ) // move the mouse cursor mouse_x+=(mouse_state.lX); mouse_y+=(mouse_state.lY); // test bounds // first x boundaries if (mouse_x >= screen_width) mouse_x = screen_width-1; else if (mouse_x < 0) mouse_x = 0; // now the y boundaries if (mouse_y >= screen_height) mouse_y= screen_height-1; else if (mouse_y < 0) mouse_y = 0;

9/20/ Game_Main( ) // position the pointer bob to the mouse coords pointer.x = mouse_x - 16; pointer.y = mouse_y - 16; // test what the user is doing with the mouse if ((mouse_x > 3) && (mouse_x < 500-3) && (mouse_y > 3) && (mouse_y < SCREEN_HEIGHT-3)) { // mouse is within canvas region // if left button is down then draw if (mouse_state.rgbButtons[0]) { … };

9/20/ Game Pads Simplified keyboard Buttons have good stimulus/response compatibility Lets you use two hand Good for multiplayer games Can require weird combinations of buttons Don’t get to rely on fastest finger (index) Doesn’t support variable input

9/20/ Joysticks Lots of devices look like joysticks (digital – 8 positions, analog x,y deflection) Allow variable input plus buttons Great for specifying changes Can be used to specify “rate in a direction” Not good for specifying absolute positions (need to “hold position steady) Not good for many arcade games

9/20/ Using Joysticks & Game Pads 1.Scan for all devices and record GUID’s 2.Create joystick device using one GUID 3.Use interface from step 2 and create another interface (release the first) 4.Set cooperation level 5.Set data format and properties 6.Acquire joystick 7.Read joystick using Poll and GetDevice

9/20/ LaMothe Example

9/20/ Joystick Global Declarations // directinput globals LPDIRECTINPUT8 lpdi = NULL; // dinput object LPDIRECTINPUTDEVICE8 lpdikey = NULL; // dinput keyboard LPDIRECTINPUTDEVICE8 lpdimouse = NULL; // dinput mouse LPDIRECTINPUTDEVICE8 lpdijoy = NULL; // dinput joystick GUID joystickGUID; // guid for main joystick char joyname[80]; // name of joystick // these contain the target records for all di input packets UCHAR keyboard_state[256]; // contains keyboard state table DIMOUSESTATE mouse_state; // contains state of mouse DIJOYSTATE joy_state; // contains state of joystick

9/20/ DI_Enum_Joysticks( ) // this function enumerates the joysticks, but // stops at the first one and returns the // instance guid of it, so we can create it *(GUID*)guid_ptr = lpddi->guidInstance; // copy name into global strcpy(joyname, (char *)lpddi->tszProductName); // stop enumeration after one iteration return(DIENUM_STOP);

9/20/ Game_Init( ) // joystick creation section //////////////////////////////// // first create the direct input object if (DirectInput8Create(main_instance,DIRECTINPUT_VERSION, IID_IDirectInput8, (void **)&lpdi,NULL)!=DI_OK) return(0); // first find the f***ing GUID of your particular joystick lpdi->EnumDevices(DI8DEVCLASS_GAMECTRL, DI_Enum_Joysticks, &joystickGUID, DIEDFL_ATTACHEDONLY); if (lpdi->CreateDevice(joystickGUID, &lpdijoy, NULL)!=DI_OK) return(0); // set cooperation level if (lpdijoy->SetCooperativeLevel(main_window_handle, DISCL_NONEXCLUSIVE | DISCL_BACKGROUND)!=DI_OK) return(0);

9/20/ Game_Init( ) // set data format if (lpdijoy->SetDataFormat(&c_dfDIJoystick)!=DI_OK) return(0); // set the range of the joystick DIPROPRANGE joy_axis_range; // first x axis joy_axis_range.lMin = -24; joy_axis_range.lMax = 24; joy_axis_range.diph.dwSize = sizeof(DIPROPRANGE); joy_axis_range.diph.dwHeaderSize = sizeof(DIPROPHEADER); joy_axis_range.diph.dwObj = DIJOFS_X; joy_axis_range.diph.dwHow = DIPH_BYOFFSET; lpdijoy->SetProperty(DIPROP_RANGE,&joy_axis_range.diph);

9/20/ Game_Init( ) // now y-axis joy_axis_range.lMin = -24; joy_axis_range.lMax = 24; joy_axis_range.diph.dwSize = sizeof(DIPROPRANGE); joy_axis_range.diph.dwHeaderSize = sizeof(DIPROPHEADER); joy_axis_range.diph.dwObj = DIJOFS_Y; joy_axis_range.diph.dwHow = DIPH_BYOFFSET; lpdijoy->SetProperty(DIPROP_RANGE,&joy_axis_range.diph); // and now the dead band (zone where movements are ignored) DIPROPDWORD dead_band; // here's our property word dead_band.diph.dwSize = sizeof(dead_band); dead_band.diph.dwHeaderSize = sizeof(dead_band.diph); dead_band.diph.dwObj = DIJOFS_X; dead_band.diph.dwHow = DIPH_BYOFFSET;

9/20/ Game_Init( ) // 10% will be used on both sides of the range +/- dead_band.dwData = 1000; // finally set the property lpdijoy->SetProperty(DIPROP_DEADZONE,&dead_band.diph); dead_band.diph.dwSize = sizeof(dead_band); dead_band.diph.dwHeaderSize = sizeof(dead_band.diph); dead_band.diph.dwObj = DIJOFS_Y; dead_band.diph.dwHow = DIPH_BYOFFSET; // 10% will be used on both sides of the range +/- dead_band.dwData = 1000; // finally set the property lpdijoy->SetProperty(DIPROP_DEADZONE,&dead_band.diph); // acquire the joystick if (lpdijoy->Acquire()!=DI_OK) return(0);

9/20/ Game_Shutdown( ) // release joystick lpdijoy->Unacquire(); lpdijoy->Release(); lpdi->Release(); // shutdonw directdraw DDraw_Shutdown();

9/20/ Game_Main( ) // get joystick data lpdijoy->Poll(); // this is needed for joysticks only lpdijoy->GetDeviceState(sizeof(DIJOYSTATE), (LPVOID)&joy_state); // lock the back buffer DDraw_Lock_Back_Surface(); // draw the background reactor image Draw_Bitmap(&playfield, back_buffer, back_lpitch, 0); // unlock the back buffer DDraw_Unlock_Back_Surface(); // is the player moving? blaster.x+=joy_state.lX; blaster.y+=joy_state.lY;

9/20/ Game_Main( ) // is player firing? if (joy_state.rgbButtons[0]) Start_Missile(); // display joystick and buttons 0-7 sprintf(buffer,"Joystick Stats: X-Axis=%d, Y-Axis=%d, buttons(%d,%d,%d,%d,%d,%d,%d,%d)", joy_state.lX,joy_state.lY, joy_state.rgbButtons[0], joy_state.rgbButtons[1], joy_state.rgbButtons[2], joy_state.rgbButtons[3], joy_state.rgbButtons[4], joy_state.rgbButtons[5], joy_state.rgbButtons[6], joy_state.rgbButtons[7]); // print out name of joystick sprintf(buffer, "Joystick Name & Vendor: %s",joyname); Draw_Text_GDI(buffer,0,SCREEN_HEIGHT-40,RGB(255,255,50),lpddsback);

9/20/ Force Feedback Gameplay can influence input device by producing some resistive force Requires bi-directional communication between interaction device and computer (e.g. joysticks, steering wheels, yokes, etc.) Examples –Engine-hum (sine-wave) –Machine gun (square-wave vibrations)

9/20/ Force Feedback Lags > 25ms between visual event and feedback are noticable Designers need to anticipate effects of conditions (e.g. wind) and surface texture (e.g. bumps and grit) in planning game play Can be mesmerizing to players (adds to realism) LaMothe has example, can’t run on my laptop (my FFB device is not USB)