1. Annual Project Review 12 October 2000 IST-1999-10500

2. Introduction Nick Lodge ITC

3. Presentation Structure Introduction Project Management Overview Linguistic Advances for Deaf People Progress on Internet Services Progress on Broadcast Services Commercial Strategies Video of Counter Services

5. Project Management Overview John Glauert UEA, Norwich, UK

6. Project Structure User Focus Application Focus Technology Focus

7. WWW High Street Broadcast Language Animation EvaluationExploitation

8. Application Focus Internet –Information and Education for Deaf People (IvD, IDGS) High Street –Counter Services – Post Office (UKPO) Television –Regulation and Standards (ITC, INT, IRT) WWW High Street Broadcast

9. User Focus Evaluation –Close Involvement of Deaf People (RNID) Exploitation –Broadcasters and Service Providers (ITC, UKPO, Televirtual) EvaluationExploitation

10. Technology Focus Sign Language Linguistics –Use of natural sign language (IDGS, UEA, IvD, RNID) Animation –Increased realism in sign generation (Televirtual, INT, UEA) Language Animation

11. Workplan Structure Project Phases Technology Transfer Prototype Applications Advanced Applications

12. Technology Transfer First 12 Months Exchange Existing Prototypes Establish Collaborative Working Largely Complete

13. Prototype Applications Months 9 to 24 Enhancement of Prototypes Extended Flexibility Focus on Well Defined Domains Excellent Progress

14. Advanced Applications Months 18 to 36 Increased Animation Realism Flexible Sign Generation Advanced Linguistic Processing Address Wide Ranging Domains

15. Workpackage Progress Achievements in Review Period –All Deliverables achieved –Milestones achieved or rescheduled with no impact Progress on Deliverables –Reported here for Prototype Applications Planned Changes –Response to Exploitation Opportunities will be presented

16. Management System Workpackage Leadership Consortium Meetings Workpackage Meetings

17. Workpackage Leadership 9 Partners 8 Workpackages Workpackage Leadership –Coordination of Partners –Workplan in Single Partner Tasks –Lightweight use of Microsoft Project

18. Consortium Meetings Team of Partner Leaders Quarterly Meetings –First day for Management and Planning –Second day allows for Technical Discussions

19. Workpackage Meetings Meetings as required by Workpackage Leader Extensive use of Email Website for Document Archive and Exchange

21. Addressing the needs of Deaf people using natural sign language - Overview of linguistic processing in ViSiCAST Thomas Hanke, IDGS, U Hamburg

22. Why Sign Language? Natural language of most Deaf, i.e. not an artificial language 1 in Europe pre-lingually deaf, only 10% of them Deaf of Deaf Most of them leave school functionally illiterate

23. What is Sign Language? A language with its own syntax & lexicon Not a 1:1 translation from spoken language as is SSE Not universal, but much closer to each other than spoken languages

24. What is so special about Sign Language? Parallel use of articulators Use of space to encode grammatical function Iconicity and productivity

25. Sign Language and Text No established written form for any sign language phonetic transcription systems such as HamNoSys Glossing: GIVE-1-3a

26. HamNoSys Flexible enough to handle at least the European sign languages, will be verified for DGS, NGT, BSL Further developed in the project to include nonmanual activities Embedded into XML: SiGML

27. SiGML Three Levels: –Glosses: mapping more or less 1:1 to the animation machines database of signs –HamNoSys: phonetic description –Motion capture files: for uninterpreted direct transmission thru SiGML

28. SiGML (2) – – – …

29. How does it fit in? Web pages English Spoken lang to text English text to Sign Notation Transport & Animation

30. Standard AI Machine Translation Approach Semantic Structure English textSign Notation (XML) Lexicon Grammar Lexicon Grammar

31. To reduce complexity Restricted Domains –Handling objects in a kitchen –Appartment descriptions Semi-automatic process –User can help both with signs and word order

33. Progress on Internet Services Margriet Verlinden IvD, The Netherlands

34. Workpackage 2 objective –produce tools that will allow a deaf citizen to access multimedia resources through sign language deliverables –Browser plug-in (month 12) –Web-page-sign (month 30) –Signing tutor (month 30)

35. Signing on the WWW What will it look like? How does it work? How is it created? How far are we now?

36. daily updated link to project info. signing avatar controls for signing What will it look like? PLAY PAUSE STOP MOCK-UP

37. How does it work? text from meteo- rological institute standard sentences playlist of motion data SERVER CLIENT Internet-page with SiGML Internet-browser + plugin for motion player (=avatar)

38. How is it created? text from meteorological institute standard sentences playlist of motion data Internet-page with SiGML model for weather forecasts mapping from stand. senten- ces to sign sequences (3x) recording + post-editing of signs (3x) XML for signs: SiGML user-interface (webpage+plug-in) update-facility

39. How far are we? basic functionality implemented motion data is being post-edited additional functionality in progress DEMO

41. Progress on Broadcast Services Mark Wells Televirtual, Norwich, UK

42. The Problem: –Legislation (UK, soon European) requires on-screen signing – Open Signing (ever present) is obtrusive –Even Closed Signing uses bandwidth: Broadcasters run a tight bandwidth economy

43. The Solution: Virtual Human Signing –Can be Open or Closed –Picture Quality not compromised by compression –Potential for User Interaction –Very Cheap in terms of bandwidth

44. Virtual Humans / Avatars

45. Entertainment Communications Transactions Guides

46. Virtual Humans / Avatars Entertainment

47. Virtual Humans / Avatars Communications

48. Signing Virtual Humans

50. Recording Signing

51. Real time Face, Hand, Body, Tracking

52. Recording Signing Real time Face, Hand, Body, Tracking - & Playback

53. On Television: Transmit the Movement - not the Video

54. Motion Data + Calibration IHoIst COM Mesh Mesh Attachment Description Bone Set Renderer Avatar Codec DSP Compressed Motion Stream DSP Avatar Codec Compressed Motion Stream Bone Set MPEG 2 Broadcast Stream Scene

55. Transmission System - The Demo...

56. Integrated TX system for broadcast to STBs – demonstrator complete end of 2000 Implementing virtual human s/w in STB Optimising compression algorithm Building multiplexer for carriage in MPEG-2 SiGML streaming Broadcast VH Signing:

57. Low transmission rate < 30 kbit/s Compatibility with signing on other media and foreign deaf languages Precise, sharp representation of signer Many display options Can conform to MPEG-4 standards Future-proof: –cost saving –allows vast no. of signed programmes –no transition from video-based to VH signing VH on TV: The Advantages

59. Commercial Strategies Nick Lodge ITC

60. What is the Product or Service? Diversity of applications –Broadcast - information displays, plays, movies –Multimedia - dictionaries, EPGs, learning tools –Face-to-face - videoconferencing, UMTS phones Diversity of languages –German, Dutch, English signs –Future: ASL, Makaton, Paget Gorman

61. Theatre / Cinema Application Display allows visibility of external world and signer

62. What is the Product or Service? Diversity of applications –Broadcast - information displays, plays, movies –Multimedia - dictionaries, EPGs, learning tools –Face-to-face - videoconferencing, UMTS phones Diversity of languages –German, Dutch, English signs –Future: ASL, Makaton, Paget Gorman

63. What is the Product or Service? Increase awareness of sign languages Promote standards for signing avatars Advance linguistic representation & processing of deaf languages Provide consultancy support

64. Building Brand Awareness Establish acceptability of virtual humans Establish presence of ViSiCAST across different media Establish Visia as a character with personality and encourage familiarity

65. The Asymmetric Market Only 0.05% - 0.1% of population sign Millions of companies wish to sell to deaf people or Wish to be seen to be aware of deaf customers

66. Legislative Influences Disability Discrimination Directive UK Broadcasting Act 1996 UK Communications Act 2001 1999 2009 5 2 3 4 1 %Content of DTT channel

67. WWW strategy Give away basic web browser Sell SiGML authoring tool De facto standard

68. Face-to-face Product Stand-alone technology Licensed speech recognition Sell ViSiCAST signing software Systems require s/w & h/w support

69. Broadcasting Production & Transmission Technology Body suits combining capture technologies Digital compression systems Multiplexers for combination with MPEG-2 services Studio practices for handling signing data Encouraging use of the system within DVB

70. The Set-top Box Strategy Develop special set-top box for deaf users by November 2001 Work with STB manufacturer to develop concept box with VH interface –speech synthesis, speech recognition Explore human factors with elderly users Develop speech for blind users

72. Progress on Counter Services TESSA Video of Deliverable

74. Example The weather forecast of the KNMI, drawn up on Monday Oct. 9, and valid till midnight. First a shower, in the west sometimes with thunder. Furthermore now and then sunny. In the afternoon more cloudy and in the evening rain from the west. Temperature in the afternoon around 14 degrees. The weather forecast of the KNMI, drawn up on Monday Oct. 9, and valid till midnight. (p.1) In the morning a shower. (p.7) Sometimes in the west thunder. (p.8) Sometimes sunny. (p.7) In the afternoon increasingly cloudy.(p.7) Tonight from the west rain. (p.8) In the afternoon 14 degrees. (p.11)