REQUIREMENTS FOR MULTIMEDIA PROCESSING MULTIMEDIA SYSTEMS IREK DEFEE

WHAT ARE THE RQUIREMENTS FOR HANDLING MULTIMEDIA DATA ? 1. PERCEPTUAL REQUIREMENTS For the user we would like to provide maximal perceptual stimulation THIS REQUIRES: - OPTIMAL STIMULATION OF EACH SENSE - OPTIMAL COORDINATION OF ALL

SENSES - INVOLVEMENT OF POSSIBLE MANY SENSES (ACTIVE) THIS MEANS INVOLVEMENT OF: - ACOUSTICAL (HEARING,VOICE) - VISUAL - HAPTIC (TOUCH, MOTORIC, VESTIBULAR (orientation in space) ) ON FULL SCALE (e.g. SPATIAL PROPERTIES)

THE PROBLEM IS HOW THIS CAN BE DONE TECHNICALLY: - ACOUSTICAL STIMULATION THIS IS NOW RELATIVELY EASY (SPATIAL (MULTICHANNEL SOUND) - VISUAL STIMULATION SPATIAL (3-D) IMAGES ARE PROBLEM, HIGH RESOLUTION TOO

HAPTIC STIMULATION THERE ARE MANY INPUT DEVICES (JOYSTICKS), BUT FULL STIMULATION INCLUDING MOTOR SENSE, SKIN SENSORS AND VESTIBULAR SYSTEM IS DIFFICULT IN FACT WE CAN TALK ABOUT TWO TYPES OF SYSTEMS FOR STIMULATION:

- CLOSELY COUPLED TO SENSORY SYSTEM (BY GLOVES, UNIFORMS, EYE DISPLAYS, HEADPHONES, MOVING SIMULATORS) THEY ALLOW ”FULL IMMERSION”, OR CREATION OF VIRTUAL REALITY MIGHT BE TIRING TO USE - LOOSELY COUPLED TO SENSORY SYSTEM (NORMAL SCREENS, SPEAKERS, JOYSTICKS ETC.) MIGHT BE MORE COMFORTABLE

IN THIS COURSE WE ARE TALKING ABOUT ’LOOSELY’ COUPLED MULTIMEDIA SYSTEMS WITH STANDARD SENSORY INTERFACES VIRTUAL REALITY IS A TOPIC OF OTHER COURSES

WHAT ARE THE TECHNICAL REQUIREMENTS FOR MULTIMEDIA SYSTEMS ? 1.HANDLING OF LARGE DATA SETS E.G. AUDIO 5 CHANNELS X 16-BIT X SAMPLES/SEC = BITS/SEC ~450 KB/sec = 27 MB/min=1,6GB/h

SECOND EXAMPLE, VIDEO RESOLUTION OF STANDARD TV PICTURE IS 720X576 POINTS, EACH POINT IS 24-BIT (8-BIT R, G, B) THERE ARE 25 PICTURES PER SECOND SO THE DATA RATE IS 720X576X25X24~ 250 Mb/sec = 1.8GB/min = 108 GB/h FOR HIGHER RESOLUTIONS:

A FORMAT FOR HIGH-DEFINITION TV IS PROPOSED PICTURE SIZE IS 1920X1080 POINTS IT IS ABOUT 5X TIMES LARGER THAN STANDARD TV. DATA RATES ARE THEN, 1.3 Gb/sec, 9 GB/min, 540 GB/h DATA RATES FOR AUDIO AND VIDEO ARE NOT PRACTICAL, COMPRESSION IS REQUIRED

- COORDINATION AND SYNCHRONIZATION - HUMAN SENSORY SYSTEM IS VERY SENSITIVE TO COORDINATION OF EVENTS IN TIME AND SPACE MULTIMEDIA DATA, ESPECIALLYAUDIO AND VIDEO DEPEND CRITICALLY ON TIME AND SPACE

EXAMPLES: - LIP SYNCHRONIZATION IN TV AND CINEMA, VOICE AND LIP MOVEMENT MUST BE SYNCHRONIZED, OTHERWISE IT IS VERY ANNOYING SYNCHRONIZATION ERROR FOR BIG PICTURES MUST BE IN THE LIMIT OF +20 ; –40 MILISECONDS

- SPATIAL SOUND HEARING SYSTEM ESTABLISHES LOCATION OF SOUND SOURCES BASED ON DIFFERENCE OF TIME OF ARRIVAL AT EARS PRECISION REQUIRED IS ON THE ORDER OF SEVERAL 10’s OF MICROSECONDS.

THIRD REQUIREMENT: FAST RESPONSE WHENEVER WE RESPOND (E.G. BY CLICKING), WE EXPECT IMMEDIATE RESPONSE. IMMEDIATE MEANS ”NO NOTICEABLE DELAY” FOR PEOPLE. IT MEANS LESS THAN 300 ms=0.3 sec

THUS, MULTIMEDIA DATA REQUIRE HIGH-DATA RATE & TIME/SPACE CRITICAL PROCESSING WITH FAST RESPONSE TO EXTERNAL EVENTS THESE ARE ATTRIBUTES OF REAL-TIME PROCESSING

DEMANDS FOR COMPUTATIONAL POWER ARE HIGH ONLY RECENTLY THIS BECAME POSSIBLE

Multimedia systems requirements CONTENT DELIVERY DISTRIBUTION CONSUMER Recall the multimedia system is for service offering:

What users expect from a service they use? -They expect satisfying Quality of Service – QoS QoS includes technical aspects which are most important for engineers (e.g. good communication link) QoS in broad sense includes all other aspects which make customer satisfied (e.g. support, price)

What is QoS ? QoS is degree to which customer expectations are satisifed QoS includes technical parameters for quality: e.g. reliability of communication QoS may also include other parameters for satisfaction (attraction, support, price, etc.)

What service owners expect? They expect proper environment that they can run their service Especially that their buisness can run without problems Unfortunately for network buisnesses there are big problems with this What is absolutely necessary is SERVICE PROTECTION

Good buiness over the network needs as the most critical requirement STRONG SERVICE PROTECTION CONTENT DELIVERY DISTRIBUTION CONSUMER Every part of the system has to be strongly protected against those who want to destroy the service integrity

Overall architecture of multimedia system There are two ”layers”: Hardware layer: servers, networks and terminals Software layer: content, service, provisioning, networking, middleware, applications

Multimedia system Composed of hardware/software components Goal: Creation and delivery of media content from production to consumer at proper QoS in an economic way This means business is sustainable and consumers are satisfied

Examples of multimedia systems Content delivery system examples: Television/(radio?) (old, basic system) Web/Internet type Mobile (radio networks) Content distribution type examples: Music Movies (audio/visual content Games Virtual worlds (Check Second life on the Web) eBooks? eMagazines? eNewspapers?