Annual Project Review 12 October 2000 IST

Introduction Nick Lodge ITC

Presentation Structure Introduction Project Management Overview Linguistic Advances for Deaf People Progress on Internet Services Progress on Broadcast Services Commercial Strategies Video of Counter Services

Project Management Overview John Glauert UEA, Norwich, UK

Project Structure User Focus Application Focus Technology Focus

WWW High Street Broadcast Language Animation EvaluationExploitation

Application Focus Internet –Information and Education for Deaf People (IvD, IDGS) High Street –Counter Services – Post Office (UKPO) Television –Regulation and Standards (ITC, INT, IRT) WWW High Street Broadcast

User Focus Evaluation –Close Involvement of Deaf People (RNID) Exploitation –Broadcasters and Service Providers (ITC, UKPO, Televirtual) EvaluationExploitation

Technology Focus Sign Language Linguistics –Use of natural sign language (IDGS, UEA, IvD, RNID) Animation –Increased realism in sign generation (Televirtual, INT, UEA) Language Animation

Workplan Structure Project Phases Technology Transfer Prototype Applications Advanced Applications

Technology Transfer First 12 Months Exchange Existing Prototypes Establish Collaborative Working Largely Complete

Prototype Applications Months 9 to 24 Enhancement of Prototypes Extended Flexibility Focus on Well Defined Domains Excellent Progress

Advanced Applications Months 18 to 36 Increased Animation Realism Flexible Sign Generation Advanced Linguistic Processing Address Wide Ranging Domains

Workpackage Progress Achievements in Review Period –All Deliverables achieved –Milestones achieved or rescheduled with no impact Progress on Deliverables –Reported here for Prototype Applications Planned Changes –Response to Exploitation Opportunities will be presented

Management System Workpackage Leadership Consortium Meetings Workpackage Meetings

Workpackage Leadership 9 Partners 8 Workpackages Workpackage Leadership –Coordination of Partners –Workplan in Single Partner Tasks –Lightweight use of Microsoft Project

Consortium Meetings Team of Partner Leaders Quarterly Meetings –First day for Management and Planning –Second day allows for Technical Discussions

Workpackage Meetings Meetings as required by Workpackage Leader Extensive use of Website for Document Archive and Exchange

Addressing the needs of Deaf people using natural sign language - Overview of linguistic processing in ViSiCAST Thomas Hanke, IDGS, U Hamburg

Why Sign Language? Natural language of most Deaf, i.e. not an artificial language 1 in Europe pre-lingually deaf, only 10% of them Deaf of Deaf Most of them leave school functionally illiterate

What is Sign Language? A language with its own syntax & lexicon Not a 1:1 translation from spoken language as is SSE Not universal, but much closer to each other than spoken languages

What is so special about Sign Language? Parallel use of articulators Use of space to encode grammatical function Iconicity and productivity

Sign Language and Text No established written form for any sign language phonetic transcription systems such as HamNoSys Glossing: GIVE-1-3a

HamNoSys Flexible enough to handle at least the European sign languages, will be verified for DGS, NGT, BSL Further developed in the project to include nonmanual activities Embedded into XML: SiGML

SiGML Three Levels: –Glosses: mapping more or less 1:1 to the animation machines database of signs –HamNoSys: phonetic description –Motion capture files: for uninterpreted direct transmission thru SiGML

SiGML (2) – – – …

How does it fit in? Web pages English Spoken lang to text English text to Sign Notation Transport & Animation

Standard AI Machine Translation Approach Semantic Structure English textSign Notation (XML) Lexicon Grammar Lexicon Grammar

To reduce complexity Restricted Domains –Handling objects in a kitchen –Appartment descriptions Semi-automatic process –User can help both with signs and word order

Progress on Internet Services Margriet Verlinden IvD, The Netherlands

Workpackage 2 objective –produce tools that will allow a deaf citizen to access multimedia resources through sign language deliverables –Browser plug-in (month 12) –Web-page-sign (month 30) –Signing tutor (month 30)

Signing on the WWW What will it look like? How does it work? How is it created? How far are we now?

daily updated link to project info. signing avatar controls for signing What will it look like? PLAY PAUSE STOP MOCK-UP

How does it work? text from meteo- rological institute standard sentences playlist of motion data SERVER CLIENT Internet-page with SiGML Internet-browser + plugin for motion player (=avatar)

How is it created? text from meteorological institute standard sentences playlist of motion data Internet-page with SiGML model for weather forecasts mapping from stand. senten- ces to sign sequences (3x) recording + post-editing of signs (3x) XML for signs: SiGML user-interface (webpage+plug-in) update-facility

How far are we? basic functionality implemented motion data is being post-edited additional functionality in progress DEMO

Progress on Broadcast Services Mark Wells Televirtual, Norwich, UK

The Problem: –Legislation (UK, soon European) requires on-screen signing – Open Signing (ever present) is obtrusive –Even Closed Signing uses bandwidth: Broadcasters run a tight bandwidth economy

The Solution: Virtual Human Signing –Can be Open or Closed –Picture Quality not compromised by compression –Potential for User Interaction –Very Cheap in terms of bandwidth

Virtual Humans / Avatars

Entertainment Communications Transactions Guides

Virtual Humans / Avatars Entertainment

Virtual Humans / Avatars Communications

Signing Virtual Humans

Recording Signing

Real time Face, Hand, Body, Tracking

Recording Signing Real time Face, Hand, Body, Tracking - & Playback

On Television: Transmit the Movement - not the Video

Motion Data + Calibration IHoIst COM Mesh Mesh Attachment Description Bone Set Renderer Avatar Codec DSP Compressed Motion Stream DSP Avatar Codec Compressed Motion Stream Bone Set MPEG 2 Broadcast Stream Scene

Transmission System - The Demo...

Integrated TX system for broadcast to STBs – demonstrator complete end of 2000 Implementing virtual human s/w in STB Optimising compression algorithm Building multiplexer for carriage in MPEG-2 SiGML streaming Broadcast VH Signing:

Low transmission rate < 30 kbit/s Compatibility with signing on other media and foreign deaf languages Precise, sharp representation of signer Many display options Can conform to MPEG-4 standards Future-proof: –cost saving –allows vast no. of signed programmes –no transition from video-based to VH signing VH on TV: The Advantages

Commercial Strategies Nick Lodge ITC

What is the Product or Service? Diversity of applications –Broadcast - information displays, plays, movies –Multimedia - dictionaries, EPGs, learning tools –Face-to-face - videoconferencing, UMTS phones Diversity of languages –German, Dutch, English signs –Future: ASL, Makaton, Paget Gorman

Theatre / Cinema Application Display allows visibility of external world and signer

What is the Product or Service? Diversity of applications –Broadcast - information displays, plays, movies –Multimedia - dictionaries, EPGs, learning tools –Face-to-face - videoconferencing, UMTS phones Diversity of languages –German, Dutch, English signs –Future: ASL, Makaton, Paget Gorman

What is the Product or Service? Increase awareness of sign languages Promote standards for signing avatars Advance linguistic representation & processing of deaf languages Provide consultancy support

Building Brand Awareness Establish acceptability of virtual humans Establish presence of ViSiCAST across different media Establish Visia as a character with personality and encourage familiarity

The Asymmetric Market Only 0.05% - 0.1% of population sign Millions of companies wish to sell to deaf people or Wish to be seen to be aware of deaf customers

Legislative Influences Disability Discrimination Directive UK Broadcasting Act 1996 UK Communications Act %Content of DTT channel

WWW strategy Give away basic web browser Sell SiGML authoring tool De facto standard

Face-to-face Product Stand-alone technology Licensed speech recognition Sell ViSiCAST signing software Systems require s/w & h/w support

Broadcasting Production & Transmission Technology Body suits combining capture technologies Digital compression systems Multiplexers for combination with MPEG-2 services Studio practices for handling signing data Encouraging use of the system within DVB

The Set-top Box Strategy Develop special set-top box for deaf users by November 2001 Work with STB manufacturer to develop concept box with VH interface –speech synthesis, speech recognition Explore human factors with elderly users Develop speech for blind users

Progress on Counter Services TESSA Video of Deliverable

Example The weather forecast of the KNMI, drawn up on Monday Oct. 9, and valid till midnight. First a shower, in the west sometimes with thunder. Furthermore now and then sunny. In the afternoon more cloudy and in the evening rain from the west. Temperature in the afternoon around 14 degrees. The weather forecast of the KNMI, drawn up on Monday Oct. 9, and valid till midnight. (p.1) In the morning a shower. (p.7) Sometimes in the west thunder. (p.8) Sometimes sunny. (p.7) In the afternoon increasingly cloudy.(p.7) Tonight from the west rain. (p.8) In the afternoon 14 degrees. (p.11)