Hardware that Enables Multimedia Input and Output Devices Virtual Reality Devices Modems and Network Interfaces

Input and Output Devices Monitors Speakers and MIDI interfaces VR helmets and immersive displays Keyboards and OCR devices Digital cameras, scanners, & CD-ROMs MIDI keyboards and microphones Video cameras and frame grabbers Mice, track balls, joysticks, and VR gloves and wands Input

Monitors A simplified cathode ray tube (CRT)

Monitors Pixel — a picture element; a dot of color on the screen Three different phosphors at each pixel to create the color RGB (Red, green, blue) CYM (Cyan, yellow, magenta)

Monitors Raster Scanning

Monitors Refresh rate — the frequency at which the phosphors are excited Normally the refresh rate is given in Hertz For flicker-free images 75 Hz or faster is desirable The refresh rate for a projector needs to be coordinated with the monitor

Monitors The digitized image to be displayed must be stored in a buffer The stored image is said to be “bit-mapped,” because, for monochrome images, the map used just one bit per pixel Multimedia monitors use 24 bits per pixel (8 for each color); can define >16 million colors

A Good Multimedia Monitor Large enough for comfortable viewing, probably 15” or greater Pixel size of no more than 0.28mm Refresh rate of at least 75 Hz Capable of displaying 24-bit color Designed for the CPU and operating system Ergonomically comfortable and attractive

Speakers and MIDI Interfaces Storage of digitized sound files Reproduction via digital-to-analog conversion sent to a loudspeaker Built-in speakers often do not have sufficient fidelity Low-powered (3- to 5-watt) external speakers or head-phones will serve a single user and provide excellent fidelity

Speakers and MIDI Interfaces Storage of synthesizer command files Create the sounds by sending the commands to a synthesizer Musical Instrument Digital Interface (MIDI) standard (1982) MIDI includes both a hardware and a message standard

Speakers and MIDI Interfaces MIDI hardware standard defines cables, connectors, circuits, and electrical signals MIDI message standard defines a Device number for multiple device systems Control segment that gives an instrument-specific command Data segment containing the information needed by that instrument for that command

Alphanumeric Keyboards For entering commands, text, and data Each key is a switch that closes when it is depressed, sending a code to the CPU The arrangement of the keys may vary The most common is QWERTY Another arrangement is Dvorak

Choosing a Keyboard Does it include all of the needed characters, including command keys? Is it ergonomically comfortable and safe, preventing repetitive stress syndrome?

Optical Character Recognition (OCR) Hardware — scans the text image Software — systematically checks the entire image for patterns of light and dark that it recognizes as alphabetic, numeric, or punc- tuation characters OCR software entails pattern recognition, a sophisticated logic problem

Optical Character Recognition (OCR) E e E E e e E E e E e e E E E e e E e e It is relatively easy for a human to recognize each of these characters as the letter “e.” For the pattern recognition logic in OCR software, this is very difficult.

Digital Cameras and Scanners Real Image — a portion of what is physically present in nature Digital Image — a representation of a real image in which individual points are encoded to represent the wavelength and intensity of light at that point Still Image — a single snapshot of an instant; may be real or digital Motion Image — a sequence of images that, when viewed consecutively at the appro- priate rate, gives the impression of con- tinuous motion; may be digital or analog

Scanners Schematic Drawing of a Scanner

Digital Cameras Schematic Drawing of a Digital Camera

Digital Cameras and Scanners Image quality depends on the: Quality of the optics and the scanning mechanism, which determines focus Precision of the photosensitive cells, which determines the accuracy of the encoding of intensity and wavelength data Resolution of the instrument in dots per inch, which determines graininess Amount of storage available, which deter- mines the total size of an image that can be digitized

Inputting Images Memory required to store a 5” x 7” snapshot Dots/inch resolution of snapshot image 100 300 600 1200 Bytes required for storage 1.05 Mb 9.45 Mb 37.8 Mb 151.2 Mb Assuming no compression, 24 bits per pixel Memory required to store a 5” x 7” snapshot

Video Cameras and Frame Grabbers Video cameras are similar to digital cameras Except that a video camera takes image after image continuously The output from many video cameras is analog and requires digitizing circuitry to make the image usable in a computer Digital camcorders are now available Frame grabber software allows the capture of a single still image from the video stream Frame grabbed images are of rather low resolution, however, <80-90 dots/inch

Microphones and MIDI Keyboards For input of sound Microphones capture sound waves from the air as an analog signal The analog signal must be digitized to be stored and then replayed by the computer Digitizing at <10,000 Hz is adequate for speech; 20,000 Hz is needed for music MIDI keyboards usually look like piano key- boards with extra switches and controls MIDI keyboards encode and transmit musical information according to the MIDI standard

Inputting Positional Information Mice Trackballs Track pads Joysticks Drawing tablets

Inputting Positional Information Specifying a point on a surface requires two dimensions, as with latitude and longitude A third dimension could be added, as with altitude For multimedia, what is commonly needed is position on the monitor in terms of left-right (X) and up- down (Y) distances

Inputting Positional Information X and Y coordinates are obtained relative to a fixed point, usually one corner of the screen The coordinates are entered in analog form as output from roll- ing wheels inside a device such as a mouse The analog values are digitized to specify the X and Y coordinates

The Mechanism of a Mouse

Using a Drawing Tablet

Collage by Janet Anderson Created in part using a drawing tablet © Janet Anderson

CD-ROMs, DVDs, and Video Disks Media for external storage and transport of data Compact disk—read-only memory Rewritable compact disk (CD-RW) DVD Video disk (laser disk); analog format

CD-ROMs Digital format Write once, read many times A rewritable version (CD-RW) is available, but not in common use Information is “written” by burning tiny holes in the disk surface with a laser The hole pattern is read by a laser and inter- preted as the bits comprising the data Can store 600-700 megabytes of data; about 300,000 pages of double-spaced text, or more than an hour of high fidelity sound

Creating Multimedia CD-ROMs Requires a hard disk large enough to store ~650 megabytes of data to be written to the CD Requires a CD-ROM recorder that writes the data to the blank CD using a laser The developer creates the multimedia mater- ial, stores it on the hard disk, and then tests it as completely as possible When the material is in final form, it is written to the blank CD as if it were being copied from one disk to another

Video Disks or Laser Disks Much larger than a CD-ROM; ~12” in diameter Hold ~54,000 video frames per side Hold ~30 minutes of video per side Read-only Analog format Requires a conversion board to be used with a computer Excellent for large-scale, video-based multimedia projects

Virtual Reality Devices Non interactive Slow image update rate Simple image Nonengaging content and presentation No sound Basic Screen display Low resolution image Monoscopic image Small field of view No head tracking No body motion sensing No tactile feedback Highly interactive Fast image update rate Highly complex image Highly engaging content and presentation Three-dimensional sound Head-mounted display High resolution image Stereoscopic image Full field of view Full head tracking Full body motion sensing Full tactile feedback Factors affecting the degree of immersion in virtual reality

NCSA’s CAVE

NCSA’s CAVE Virtual Reality Room with stereo glasses and magnetic head/hand tracking Fully immersive using three of four walls to display the graphics Uses an SGI Power Onyx with Reality Engine 2 software

Miniature version of NCSA’s CAVE NCSA’s ImmersaDesk Miniature version of NCSA’s CAVE

Drafting-table format virtual prototyping device NCSA’s ImmersaDesk Drafting-table format virtual prototyping device

Uses CAVE’s stereo glasses and magnetic head/hand tracking NCSA’s ImmersaDesk Uses CAVE’s stereo glasses and magnetic head/hand tracking

Semi-immersive, fills the user’s field of vision NCSA’s ImmersaDesk Semi-immersive, fills the user’s field of vision

NCSA’s ImmersaDesk Uses an identical SGI Power Onyx with the same Reality Engine 2 software as the CAVE

VR Head-Mounted Display

VR Head-Mounted Display Limitations: Liquid Crystal Displays (LCDs) pixels not as small as a CRT pixels not as bright as a CRT cannot change as quickly as a CRT short focal distance makes precision, high resolution, and rapid response even more essential Muscle receptor feedback confusion light rays indicate “distant” muscles indicate “very close”

VR Head-Mounted Display Parallax — the apparent change in position of a stationary object when viewed from slightly different positions A person’s eyes each see a slightly different view of an object As the brain receives these two images, it interprets the the distance to the object in terms of the difference in position of the object in the two images Parallax can be used to fool the brain into “seeing” images as being at various distances

Demonstrating Parallax Pencil Demonstrating Parallax Looking at a pencil aligned What is seen using with the corner of a room both eyes What is seen with What is seen with right eye covered left eye covered

Stereoscope Courtesy of Special Collections, M. I. King Library, University of Kentucky

Parallax Problems with VR Head-Mounted Displays Images may not be perfectly realistic, especially with motion images When the observer’s head moves and the eyes are refocused, muscle receptor feedback data does not correlate with visual cues The perspective is always that of the camera, never the viewer’s eyes A viewer motion feedback mechanism is needed to change the perspective This all contributes to “cybersickness”

VR Aural Output Refer to the discussion in chapter 2 regarding the perception of sound Two key factors Localization Identification The brain interprets differences in the signals it receives from the two ears in a manner analogous to binocular vision For multimedia sound to be completely realistic, it requires head-position sensing feedback and enormous computational power — not practical for most multimedia

VR Input Devices The terminology of three-dimensional motion y x z Yaw Roll Origin z Pitch The terminology of three-dimensional motion

VR Position Sensing A point in space is defined in terms of distance along three mutually perpen- dicular axes, usually termed X, Y, and Z Motion is defined in terms of changes in position, which requires six parameters Devices that can sense and record motion are termed six-degrees-of-freedom (6-DOF) devices

VR Position Sensing Sensor output from a 6-DOF device can be continuous polled, or sent only upon request Parameters to consider in evaluating a tracking device: Lag or Latency — the delay between the actual time of the motion and when it is available as input data; should be <50 mSec Update rate — Rate at which measurements are made; should be as fast as possible Precision or accuracy of the measurements Range over which the sensors operate Rejection of interference

VR Voice Input Speech Recognition Complications due to variations in Pitch Timbre Volume Speed of Delivery Inflection Accent Natural language processing

Natural Language Processing Put out the light. Turn off the light. Close the light, please. The light, turn it off. Please, shut the light. Kill the lights.

Natural Language Processing Only the son praised his sister. (The rest of the family did not.) The only son praised his sister. (There was just one son.) The son only praised his sister. (He never found fault with her.) The son praised only his sister. (But never anyone else.) The son praised his only sister. (He had just one sister.) The son praised his sister only. (In this instance, he praised no one but her.)

Modems and Network Interfaces Serial and Parallel Serial — the bits arrive sequentially Parallel — the bits arrive simultaneously Character encoding ASCII — American standard code for information interchange EBCDIC — Extended binary-coded decimal interchange code Unicode

ASCII and Unicode ASCII is limited because it is only a 7- or 8-bit code; even using “escape sequences” only a small number of characters can be encoded Unicode is a 16-bit code that can encode many primary scripts plus special character sets known as secondary scripts

Unicode Scripts Primary scripts: Arabic Georgian Hebrew Malayalam Armenian Greek Hiragana Oriya Bengali Gujarati Kannada Phonetic Bopomofo Gurmkhi Katakana Tamil Cyrillic Han Latin Telugu Devanagari Hangul Lao Thai Secondary scripts: Numbers General diacritics General symbols Miscellaneous symbols General Punctuation Technical symbols Dingbats Mathematical symbols Presentation forms Arrows, blocks, box drawing forms, and geometric shapes

Modems and Network Interfaces Start bits Opposite bit from the system idle state Necessary to alert the receiver to the beginning of a new character Stop bits Provide a short delay at the end of each character to give the receiver enough time to convert from serial to parallel Error-checking codes Parity bits, CRC bits, etc. Discussed in chapter 6

Modems and Network Interfaces Transmission rate Internal transfer rates are much faster than data rates over networks The interface needs to “interrupt” the computer when it has new data, not keep it from doing other processing while data is being received Transmission form Connection via a telephone line requires a modem (MOdulator-DEModulator) to translate the internal data transfer format into an audio signal, and vice-versa

Modems and Network Interfaces Use of modems and telephone lines for connectivity