1. RTP Payload format for Application and Desktop Sharing
Omer Boyaci & Henning Schulzrinne
November 18, 2008
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

2. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Application Sharing
Sharing an application with multiple users
There is only one copy of the application
Participants do not need application itself
Briefly, participants
receive screen updates
send keyboard and mouse events
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

3. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Terminology
Application Host (AH)
Participant
Remoting protocol
Human Interface Protocol (HIP)
Different modes of remote access
Remote Desktop Connection
(Full or partial)Screen Sharing
Application Sharing
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

4. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Screenshot
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

5. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Screenshot (2)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

6. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Architecture
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Remoting protocol

7. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Architecture
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Human Interface Protocol (HIP)

8. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Architecture
Binary Floor Control Protocol (BFCP) manages the ownership of AH side human interface devices.
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

9. Multimedia Support (Movies)
Composite image comparing JPEG and PNG: notice artifacts in JPEG versus solid PNG background.
draft-boyaci-avt-png-00
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

10. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Comparison of Sharing Systems in terms of web page visiting performance
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

11. Multimedia Support (Movies)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

12. Comparison of Sharing Systems in terms of multimedia performance
Unlimited Bandwidth
BASS-Theora
BASS-JPEG
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

13. Comparison of Sharing Systems in terms of multimedia performance
3Mb/s Bandwidth
BASS-Theora
BASS-JPEG
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

14. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Remoting Messages
Application host (AH) to participants
WindowStateInfo
RegionUpdate
MoveRectangle
MousePointerInfo
Participants to AH (NACK messages)
PLI (Picture Loss Indication)
NACK request
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

15. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
HIP Messages
MousePressed
MouseReleased
MouseMoved
MouseWheelMoved
KeyPressed
KeyReleased
KeyTyped
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

16. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
VNC problems
Client-pull based
No multicast support
Same encoding for all updates
CPU usage increases for each new user
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

17. TeleTeachingTool (VNC based)
Problems
Modified VNC protocol and clients (TTT server is pushing updates)
No compression (only hextile encoding) because packets can get lost
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

18. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Open Issues
Transport Protocol
RTP
MSRP
Custom Made
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

19. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Open Issues
How to do retransmission requests for UDP clients
Current proposal
NACK-based solution
NACK suppression to prevent floods
RTP-library level retransmissions
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

20. Current Internet Drafts
draft-boyaci-avt-app-sharing-00
draft-boyaci-avt-png-00
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

21. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Backup Slides
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

22. Protocol Message Structures
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

23. Common remoting/hip Header
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

24. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
A Window Record
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

25. A Region Update Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

26. The MoveRectangle Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

27. The MousePressed Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

28. The KeyPressed Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

29. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
The KeyTyped Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

30. Comparison of Sharing Systems
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

31. Screenshot (Overlapped Windows)1 4 2 3
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

32. Screenshot (Multicast App. Tool)1 4 2 3 4
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

33. Screenshot (Ultra VNC)1 4 2 3 4
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

34. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Screenshot (BASS) 1 4 2
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

35. Comparison of Sharing Systems
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

36. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
VNC problems
Client-pull based
No multicast support
Same encoding for all updates
CPU usage increases for each new user
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

37. TeleTeachingTool (VNC based)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

38. Information about BASS
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

39. Supported Platforms/OS
*Client is Java based.
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

40. Client (Viewer) Architecture
Client can
Connect to server
Wait for incoming connections
Client supports
TCP
UDP (+Multicast)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

41. BASS Windows Server Architecture
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

42. BASS Windows Server Architecture
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

43. BASS Windows Server Architecture
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

44. BASS Windows Server Architecture
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

45. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Challenges
Different client bandwidths/speeds
Late Joiner
The effects of packet loss
Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

46. Multimedia Support (Movies)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

47. Multimedia Support (Movies)
Our system uses PNG to compress and transmit the region updates
PNG is lossless and effective for computer generated images but ineffective for real world captures like pictures or movies
JPG is more suitable for photographic images
However, JPG is lossy and not effective for computer generated images (text, line, shapes,...)
Our system should use both
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

48. Multimedia Support (Movies)
Composite image comparing JPEG and PNG: notice artifacts in JPEG versus solid PNG background.
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

49. Multimedia Support (PNG vs JPG)
Size x
360x150
162K
Size 4x
720x300
648K
1:20
1:4
1:5
1:30
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

50. Multimedia Support (PNG vs JPG)
Ethernet (60Mb/s)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

51. Multimedia Support (PNG vs JPG)
Wireless (4Mb/s)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

52. PNG/JPG Detection Algorithm
Region>
40,000px ?
-1,0,1
coordinates
PNG Size
counter
YES
Time Stamp
New
Region ?
Create a record
&
Start Checking
Region record
YES NO
Detected ?
Continue
Checking
Use Detected
Format NO
YES
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

53. Sharing a Movie (Media Player)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

54. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Sharing a Movie File
Capturing from the Frame Buffer is expensive.
Instead
Transcode the movie to Theora beforehand
Then stream the theora directly to participants
Java client supports theora playback
Negligible CPU usage during playback on the server
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

55. Sharing a Movie (Our Method)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

56. Sharing a Movie File Movie File (wmv,mpg,divx)
FFMpeg2Theora
Java
Client
Theora Movie
Java
Streaming
Server
Java
Client
UDP/Multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

57. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Challenges
Different client bandwidths/speeds
Late Joiner
The effects of packet loss
Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

58. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

59. Different Client Bandwidths/Speeds
Possible Solutions
Slowest one
Average speed
Fastest one
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

60. Different Client Bandwidths/Speeds
Possible Solutions
Slowest one
Problem: Penalize everybody except the slowest
Average speed
Fastest one
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

61. Different Client Bandwidths/Speeds
Possible Solutions
Slowest one
Problem: Penalize everybody except the slowest
Average speed
Possible solution
Fastest one
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

62. Different Client Bandwidths/Speeds
Possible Solutions
Slowest one
Problem: Penalize everybody except the slowest
Average speed
Possible solution (Can we do better?)
Fastest one
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

63. Different Client Bandwidths/Speeds
Possible Solutions
Slowest one
Problem: Penalize everybody except the slowest
Average speed
Possible solution (Can we do better?)
Fastest one
The best solution
Client bandwidths are fully utilized
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

64. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

65. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

66. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Challenges
Different client bandwidths/speeds
Late Joiner
The effects of packet loss
Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

67. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Late Joiner
Force server to generate full screen update
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

68. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Late Joiner
Force server to generate full screen update
Problems
Misbehaving clients can degrade performance
If Join/Leave rate is high, too much burden on server
Solution
Generate full screen updates if really necessary
Otherwise start the new client from last full screen update
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

69. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

70. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

71. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

72. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

73. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

74. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

75. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

76. Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

77. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Challenges
Different client bandwidths/speeds
Late Joiner
The effects of packet loss
Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

78. The effects of Packet Loss
This problem applies to
Multicast
UDP
The PNG images can be large
Regular desktop can be ~900KB
~600 Ethernet packets
One packet loss wastes all PNG image
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

79. The effects of Packet Loss
Solution
Small PNG images
Around ~1500 bytes
Consist of a few scanlines
Disadvantages
Increased CPU usage (client&server)
Lower compression ratio (%20 lower)
Advantages
One packet loss = no update for a few scanlines
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

80. The effects of Packet Loss
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

81. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Challenges
Different client bandwidths/speeds
Late Joiner
The effects of packet loss
Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

82. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Reliable Multicast
RTP Library stores last N rtp packets
Clients send NACK for lost packets
RTP Library resend the requested packets
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

83. The effects of Packet Loss
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

84. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Overview
Introduction
Demo
Architecture
Challenges
Features
Conclusion
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

85. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Recording
Clients can record the whole/part session
Anybody can play these files locally
These files can be streamed to receivers via streaming server
Streaming server supports multiple receivers
Also late joiners
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

86. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Listening Client
Client waits for incoming connections
It can display windows from multiple user
Can be used for RGB cable replacement
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00

87. Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00
Conclusion
Application sharing allows users to share a single application with multiple participants.
Participants don't need the application.
It is not specific to a single application.
Extra features like recording is added.
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00