Using Audio, Video, and Imaging in your Longhorn Applications Session Code: CLI342 Using Audio, Video, and Imaging in your Longhorn Applications Bill Schiefelbein Program Manager Microsoft Corporation billsch@microsoft.com

Agenda Background on current media APIs Pillars and value add in Longhorn APIs and samples at multiple levels Extensibility Infrastructure enhancements Key Takeaways Q&A

Client Application Model Web & Service Application Model Tools Client Application Model Web & Service Application Model Data Systems Application Model Mobile PC & Devices Application Model Command Line Compact Framework Avalon Windows Forms ASP.NET / Indigo Win FS Yukon System.Console Mobile PC Optimized System.Windows System.Windows.Forms System.Web System.Storage System.Data.SqlServer System.Windows.Forms System.Windows NT Service System.ServiceProcess Presentation Data Communication System.Windows System.Search System.Data System.Messaging System. Discovery System.Collaboration UI Element Explorer Media Annotations SqlClient DataSet System.DirectoryServices RealTimeEndpoint Documents Controls Animation SqlTypes Mapping Active Directory TransientDataSession Monitoring System.Remoting Text Element Dialogs Controls SqlXML ObjectSpaces SignalingSession Logging System.Runtime.Remoting Uddi Shapes SideBar Control OdbcClient ObjectSpace Media Notification Relevance Shape Panel OleDbClient Query Activities Ink Navigation Design OracleClient Schema System.Web.Services System.MessageBus System.Storage Web.Service Transport Queue System.Windows.Forms System.Web.UI Item Core Description Port PubSub Forms Page WebControls Channel Router Relationship Contact Discovery Control Control Adaptors Service Policy Media Location Protocols Print Dialog HtmlControls Design Peer Group Audio Message Design MobileControls Video Document System.Net Images Event System.Web System.Help System.Speech HttpWebRequest NetworkInformation System.Xml Personalization FtpWebListener Sockets System.Drawing Recognition Synthesis Caching Schema Xpath SslClientStream Cache System.NaturalLanguageServices Serialization Query SessionState WebClient Fundamentals Base & Application Services Security Configuration Deployment/Management System.Timers System.Text System.Collections System.Windows. TrustManagement System.Web.Configuration System.Security System.Web System.Globalization System.Design Generic System.MessageBus.Configuration Administration System.Serialization System.IO System.Web. Security Authorization Permissions System.ComponentModel System.Configuration Management System.Threading Ports AccessControl Policy System.CodeDom System.Resources System.Message Bus.Security Credentials Principal System.Management System.Runtime System.Reflection Cryptography Token System.Deployment Serialization InteropServices System.EnterpriseServices System.Diagnostics CompilerServices System.Transactions

DirectX Graphics Mini port Avalon Document UI Media Models Desktop Services Desktop Window Manager Presentation Object Manager Desktop Composition Engine Media Services Framework Animation and Composition Media Processing Capture and Sourcing Hardware Rendering Software Rendering And Sinks Base Operating System Services CLR Direct X Graphics GDI/GDI+ Window Manager Global Audio Engine Graphics Drivers DDI Input Manager Audio Drivers DirectX Graphics Mini port Base Services Kernel Mode Kernel Hardware Abstraction Layer Make the connection

Power And Ease Of A/V & Image Elements

Technology Mapping What areas are we discussing Task Win32 WinFX System Windows.Controls A/V Playback WMP OCX System Windows.Controls Image Viewing GDI+ System Windows.Media Image Processing GDI+ WM Encoder OCX DirectShow System Windows Media.Core A/V Capture/Encoding DirectShow DirectSound DirectShow Editing Services WM Format SDK Advanced Media Processing System Windows Media.Core Existing applications continue to work in Longhorn New APIs are the way forward

Today’s Multimedia APIs Multiple SDKs SDKs have overlap and gaps and do not derive from a common base technology Difficult to integrate components from each into cohesive solution Overly Complex Architecture ISVs are often forced to make big investments for relatively simple solutions Poor factored access Hard to get at bits and functionality, large footprint required for simple needs Where’s the .Net? Few managed APIs are available for multimedia

Media Services Value Add Powerful, Cohesive platform Multimedia woven into programming model Well factored architecture Advanced content protection support Distributed multimedia support Glitch-free, low-latency processing Higher fidelity audio and enhanced color-space support Hardware accelerated processing and rendering Industry leading codecs and HD media processing Quality and Performance Extensibility Audio, Video, and Image codecs Effects, transforms, and digital signal processing Support for industry and custom formats and protocols

Media Capture & Editing Levels Of API UI/Content Designer Typical App Author Control Developer Graphics Heavy App Author Media Capture & Editing Codecs & Effects Developer Framework Element Shapes Image Text Video Custom Control Drawing DrawingContext Visual API Media Pipeline Media Plugins

Framework Elements Attributes Key Elements Integrates with property, layout, and styling systems Two means of expression: XAML Markup & Code Simplest level of programmability, yet enables very compelling experiences Key Elements <Audio> <Video> <Image>

Channel Surfing Demo

A Quick Look Under The Hood… Application Application SDK Application Visual Tree Infrastructure Root Process Boundary Image Canvas Button Text Video Composition Engine Media Engine Drawing Context Desktop Composition Engine Media Session Media Processor WMV Media Source WMV Decode Color Conversion Video Renderer Audio Service WMA Decode Resample Audio Renderer D3D User Kernel Audio Driver Video Driver

Drawing Context/Resources Attributes Create pen and brush resources Lower-level programmability No hierarchical structure Some for expert-level developer w/ deep involvement in graphical aspects of app Bedrock for the Avalon Presentation Framework Key Classes AudioData VideoData ImageData

Pipelines Media Pipeline Attributes Apply image encoding and decoding and effects Configure topology for A/V capture, processing, and encoding Dynamic configuration with rich eventing Key Classes ImageDecoder, ImageEncoder MediaEngine MediaClip

Convert BMP To Compressed JPG & Apply Effect

A/V Pipeline Components Attributes Lowest level programmability and extensibility layer Access to individual components for use in custom pipelines and media processing Interfaces ImageSource MediaSource MediaSink MediaTransform

Metadata Extraction

Extensibility Application Application Image Encoder/Decoder Visual Tree Root Image Canvas Button Text Video Image Encoder/Decoder Image Effects Media Engine Drawing Context ImageDecode, ImageEncoder, ImageEffect Media Session Media Processor Formats Protocols Media Source Media Transform Media Transform Media Sink Formats Protocols Media Transform Media Transform Media Sink Encoders/Decoders A/V Effects and DSPs

Key Infrastructure Innovations More than just APIs Content Protection “Glitch-free” Services Redesigned Audio Subsystem Low latency A/V Distribution Optimization

Redesigned Audio Subsystem More Stability Audio engine in user-mode provides stability Class drivers provide dependable device support Better Quality Redesigned for glitch-free operation Vastly improved audio fidelity and core latency More Control Transparent, inserts no unsolicited processing Audio policy allows end user to define volume control and cross-app behaviors without burdening application developer Avalon Kernel Mode Framework Models Document UI Media Desktop Services Manager Desktop Window Presentation Object And Sinks Software Rendering Rendering Hardware Sourcing Capture and Processing Composition Animation and GDI/GDI+ Window Manager Global Audio Engine Graphics Drivers Direct X Graphics DirectX Graphics Mini port Audio Drivers Input Manager DDI Base Services Kernel Hardware Abstraction Layer Base Operating System Services CLR Composition Engine Desktop Media Services

Distributed Audio And Video Scenarios Remoting of A/V enabled educational programs in schools Remoting of A/V enabled business applications in the enterprise Benefits Drastically less load on host machine and network High quality experience for end user Previous Media Player Avalon Media Services-based player Terminal Services Host CPU and networking activity

*A/V Remoting Through Terminal Services *This demo does not work in PDC Longhorn distribution

Glitch-Free Services Aggressive criteria goals to provide Consumer Electronics quality For video, max of 1 Glitch-Induced dropped video frame in 8 hours of playback for full screen video rendering For audio, max of 1 audio glitch in 8 hours of playback Utilizes base OS services including Scheduled file I/O CPU Reservation Memory locking (code and heap) User controllable and application accessible resource management Ensures quality and consistency when using more than one A/V application Implements extensive instrumentation Uses WMI-based Event Tracing for Windows Modeling where time is spent and when frames and samples are dropped Hardware efforts WHQL guidelines and testing to certify drivers as “glitch-free” enabled

*Glitch-Free Services *This demo does not work in PDC Longhorn distribution

Key Takeaways Rich multimedia is woven into new programming model Longhorn enables you to deliver “Consumer Electronics” quality, HD media experiences New media content protection design in “Longhorn” ensures creation of and access to premium content Factored access will cater to Web Designers and professional editing developers alike Numerous extensibility points enable 3rd party innovation and business models

Community Resources Get Your Questions Answered! Newsgroup: Microsoft.public.windows.developer.winfx.avalon Client Lounge: middle of the Exhibit Hall connect with Microsoft client product teams, and PDC 2003 Speakers Ask The Experts: Tuesday 7 pm – 9 pm in Hall G,H (RIGHT NOW!) Longhorn Dev Center: http://msdn.microsoft.com/longhorn PDC Weblogs: http://pdcbloggers.net Now! Questions? Comments?

© 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.