1. Using Audio, Video, and Imaging in your Longhorn Applications
Session Code: CLI342
Using Audio, Video, and Imaging in your Longhorn Applications
Bill Schiefelbein
Program Manager
Microsoft Corporation

2. Agenda Background on current media APIs
Pillars and value add in Longhorn
APIs and samples at multiple levels
Extensibility
Infrastructure enhancements
Key Takeaways
Q&A

3. Client Application Model Web & Service Application Model
Tools
Client Application Model
Web & Service Application Model
Data Systems Application Model
Mobile PC & Devices Application Model
Command Line
Compact Framework
Avalon
Windows Forms
ASP.NET / Indigo
Win FS
Yukon
System.Console
Mobile PC Optimized
System.Windows
System.Windows.Forms
System.Web
System.Storage
System.Data.SqlServer
System.Windows.Forms
System.Windows
NT Service
System.ServiceProcess
Presentation
Data
Communication
System.Windows
System.Search
System.Data
System.Messaging
System. Discovery
System.Collaboration
UI Element
Explorer
Media
Annotations
SqlClient
DataSet
System.DirectoryServices
RealTimeEndpoint
Documents
Controls
Animation
SqlTypes
Mapping
Active Directory
TransientDataSession
Monitoring
System.Remoting
Text Element
Dialogs
Controls
SqlXML
ObjectSpaces
SignalingSession
Logging
System.Runtime.Remoting
Uddi
Shapes
SideBar
Control
OdbcClient
ObjectSpace
Media
Notification
Relevance
Shape
Panel
OleDbClient
Query
Activities
Ink
Navigation
Design
OracleClient
Schema
System.Web.Services
System.MessageBus
System.Storage
Web.Service
Transport
Queue
System.Windows.Forms
System.Web.UI
Item
Core
Description
Port
PubSub
Forms
Page
WebControls
Channel
Router
Relationship
Contact
Discovery
Control
Control
Adaptors
Service
Policy
Media
Location
Protocols
Print Dialog
HtmlControls
Design
Peer Group
Audio
Message
Design
MobileControls
Video
Document
System.Net
Images
Event
System.Web
System.Help
System.Speech
HttpWebRequest
NetworkInformation
System.Xml
Personalization
FtpWebListener
Sockets
System.Drawing
Recognition
Synthesis
Caching
Schema
Xpath
SslClientStream
Cache
System.NaturalLanguageServices
Serialization
Query
SessionState
WebClient
Fundamentals
Base & Application Services
Security
Configuration
Deployment/Management
System.Timers
System.Text
System.Collections
System.Windows. TrustManagement
System.Web.Configuration
System.Security
System.Web
System.Globalization
System.Design
Generic
System.MessageBus.Configuration
Administration
System.Serialization
System.IO
System.Web. Security
Authorization
Permissions
System.ComponentModel
System.Configuration
Management
System.Threading
Ports
AccessControl
Policy
System.CodeDom
System.Resources
System.Message Bus.Security
Credentials
Principal
System.Management
System.Runtime
System.Reflection
Cryptography
Token
System.Deployment
Serialization
InteropServices
System.EnterpriseServices
System.Diagnostics
CompilerServices
System.Transactions

4. DirectX Graphics Mini port
Avalon
Document UI
Media
Models
Desktop Services
Desktop Window
Manager
Presentation Object
Manager
Desktop
Composition Engine
Media Services
Framework
Animation and
Composition
Media
Processing
Capture and
Sourcing
Hardware
Rendering
Software Rendering
And Sinks
Base Operating System Services
CLR
Direct X Graphics
GDI/GDI+
Window Manager
Global Audio Engine
Graphics Drivers
DDI
Input Manager
Audio Drivers
DirectX Graphics Mini port
Base Services
Kernel Mode
Kernel
Hardware Abstraction Layer
Make the connection

5. Power And Ease Of A/V & Image Elements

6. Technology Mapping What areas are we discussing
Task
Win32
WinFX
System Windows.Controls
A/V Playback
WMP OCX
System Windows.Controls
Image Viewing
GDI+
System Windows.Media
Image Processing
GDI+
WM Encoder OCX
DirectShow
System Windows Media.Core
A/V Capture/Encoding
DirectShow
DirectSound
DirectShow Editing Services
WM Format SDK
Advanced Media Processing
System Windows Media.Core
Existing applications continue to work in Longhorn
New APIs are the way forward

7. Today’s Multimedia APIs
Multiple SDKs
SDKs have overlap and gaps and do not derive from a common base technology
Difficult to integrate components from each into cohesive solution
Overly Complex Architecture
ISVs are often forced to make big investments for relatively simple solutions
Poor factored access
Hard to get at bits and functionality, large footprint required for simple needs
Where’s the .Net?
Few managed APIs are available for multimedia

8. Media Services Value Add
Powerful, Cohesive platform
Multimedia woven into programming model
Well factored architecture
Advanced content protection support
Distributed multimedia support
Glitch-free, low-latency processing
Higher fidelity audio and enhanced color-space support
Hardware accelerated processing and rendering
Industry leading codecs and HD media processing
Quality and Performance
Extensibility
Audio, Video, and Image codecs
Effects, transforms, and digital signal processing
Support for industry and custom formats and protocols

9. Media Capture & Editing
Levels Of API
UI/Content
Designer
Typical App
Author
Control
Developer
Graphics
Heavy
App Author
Media Capture & Editing
Codecs
& Effects
Developer
Framework
Element
Shapes
Image
Text
Video
Custom Control
Drawing
DrawingContext
Visual API
Media Pipeline
Media
Plugins

10. Framework Elements Attributes Key Elements
Integrates with property, layout, and styling systems
Two means of expression: XAML Markup & Code
Simplest level of programmability, yet enables very compelling experiences
Key Elements
<Audio>
<Video>
<Image>

11. Channel Surfing Demo

12. A Quick Look Under The Hood…
Application
Application
SDK
Application
Visual Tree
Infrastructure
Root
Process Boundary
Image
Canvas
Button
Text
Video
Composition Engine
Media Engine
Drawing Context
Desktop Composition Engine
Media Session
Media Processor
WMV Media Source
WMV Decode
Color Conversion
Video Renderer
Audio Service
WMA Decode
Resample
Audio Renderer
D3D
User
Kernel
Audio Driver
Video Driver

13. Drawing Context/Resources
Attributes
Create pen and brush resources
Lower-level programmability
No hierarchical structure
Some for expert-level developer w/ deep involvement in graphical aspects of app
Bedrock for the Avalon Presentation Framework
Key Classes
AudioData
VideoData
ImageData

14. Pipelines Media Pipeline
Attributes
Apply image encoding and decoding and effects
Configure topology for A/V capture, processing, and encoding
Dynamic configuration with rich eventing
Key Classes
ImageDecoder, ImageEncoder
MediaEngine
MediaClip

15. Convert BMP To Compressed JPG & Apply Effect

16. A/V Pipeline Components
Attributes
Lowest level programmability and extensibility layer
Access to individual components for use in custom pipelines and media processing
Interfaces
ImageSource
MediaSource
MediaSink
MediaTransform

17. Metadata Extraction

18. Extensibility Application Application Image Encoder/Decoder
Visual Tree
Root
Image
Canvas
Button
Text
Video
Image Encoder/Decoder
Image Effects
Media Engine
Drawing Context
ImageDecode, ImageEncoder, ImageEffect
Media Session
Media Processor
Formats
Protocols
Media Source
Media Transform
Media Transform
Media Sink
Formats
Protocols
Media Transform
Media Transform
Media Sink
Encoders/Decoders
A/V Effects and DSPs

19. Key Infrastructure Innovations More than just APIs
Content Protection
“Glitch-free” Services
Redesigned Audio Subsystem
Low latency
A/V Distribution Optimization

20. Redesigned Audio Subsystem
More Stability
Audio engine in user-mode provides stability
Class drivers provide dependable device support
Better Quality
Redesigned for glitch-free operation
Vastly improved audio fidelity and core latency
More Control
Transparent, inserts no unsolicited processing
Audio policy allows end user to define volume control and cross-app behaviors without burdening application developer
Avalon
Kernel Mode
Framework
Models
Document UI
Media
Desktop Services
Manager
Desktop Window
Presentation Object
And Sinks
Software Rendering
Rendering
Hardware
Sourcing
Capture and
Processing
Composition
Animation and
GDI/GDI+
Window Manager
Global Audio Engine
Graphics Drivers
Direct X Graphics
DirectX Graphics Mini port
Audio Drivers
Input Manager
DDI
Base Services
Kernel
Hardware Abstraction Layer
Base Operating System Services
CLR
Composition Engine
Desktop
Media Services

21. Distributed Audio And Video
Scenarios
Remoting of A/V enabled educational programs in schools
Remoting of A/V enabled business applications in the enterprise
Benefits
Drastically less load on host machine and network
High quality experience for end user
Previous Media Player
Avalon Media Services-based player
Terminal Services Host CPU and networking activity

22. *A/V Remoting Through Terminal Services
*This demo does not work in PDC Longhorn distribution

23. Glitch-Free Services
Aggressive criteria goals to provide Consumer Electronics quality
For video, max of 1 Glitch-Induced dropped video frame in 8 hours of playback for full screen video rendering
For audio, max of 1 audio glitch in 8 hours of playback
Utilizes base OS services including
Scheduled file I/O
CPU Reservation
Memory locking (code and heap)
User controllable and application accessible resource management
Ensures quality and consistency when using more than one A/V application
Implements extensive instrumentation
Uses WMI-based Event Tracing for Windows
Modeling where time is spent and when frames and samples are dropped
Hardware efforts
WHQL guidelines and testing to certify drivers as “glitch-free” enabled

24. *Glitch-Free Services
*This demo does not work in PDC Longhorn distribution

25. Key Takeaways Rich multimedia is woven into new programming model
Longhorn enables you to deliver “Consumer Electronics” quality, HD media experiences
New media content protection design in “Longhorn” ensures creation of and access to premium content
Factored access will cater to Web Designers and professional editing developers alike
Numerous extensibility points enable 3rd party innovation and business models

26. Community Resources Get Your Questions Answered!
Newsgroup:
Microsoft.public.windows.developer.winfx.avalon
Client Lounge: middle of the Exhibit Hall
connect with Microsoft client product teams, and PDC 2003 Speakers
Ask The Experts:
Tuesday 7 pm – 9 pm in Hall G,H (RIGHT NOW!)
Longhorn Dev Center:
PDC Weblogs:
Now! Questions? Comments?

27. © 2003-2004 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.