11/17/2018 9:30 PM Session Code: ARC 420 Transaction Support Under The Covers: Transactional NTFS, KTM, and WinLS Rajeev Nagar rajeevn@microsoft.com Surendra Verma sverma@microsoft.com Dana Groff dgroff@microsoft.com © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

11/17/2018 9:30 PM Tools Client Application Model Web & Service Application Model Data Systems Application Model Mobile PC & Devices Application Model Command Line Compact Framework Avalon Windows Forms ASP.NET / Indigo Win FS Yukon System.Console Mobile PC Optimized System.Web System.Storage System.Data.SqlServer System.Windows.Forms System.Windows NT Service System.Windows System.Windows.Forms System.ServiceProcess Presentation Data Communication System.Windows System.Search System.Data System.Messaging System. Discovery System.Collaboration UI Element Explorer Media Annotations SqlClient DataSet System.DirectoryServices RealTimeEndpoint Documents Controls Animation SqlTypes Mapping Active Directory TransientDataSession Monitoring System.Remoting Text Element Dialogs Controls SqlXML ObjectSpaces SignalingSession Logging System.Runtime.Remoting Uddi Shapes SideBar Control OdbcClient ObjectSpace Media Shape Notification Panel Relevance OleDbClient Query Activities Ink Navigation Design OracleClient Schema System.Web.Services System.MessageBus Web.Service Transport Queue System.Storage System.Windows.Forms System.Web.UI Port PubSub Item Core Description Forms Page WebControls Discovery Channel Router Relationship Contact Control Control Adaptors Service Policy Media Location Protocols Print Dialog HtmlControls Design Peer Group Audio Message Design MobileControls Video Document System.Net Images Event System.Web System.Help System.Speech HttpWebRequest NetworkInformation System.Xml Personalization FtpWebListener Sockets System.Drawing Recognition Synthesis Caching Schema Xpath SslClientStream Cache System.NaturalLanguageServices Serialization Query SessionState WebClient Fundamentals Base & Application Services Security Configuration Deployment/Management System.Timers System.Text System.Collections System.Windows. TrustManagement System.Web.Configuration System.Globalization System.Design System.Security System.Web Generic System.MessageBus.Configuration Administration System.Serialization System.IO System.Web. Security Authorization Permissions System.ComponentModel System.Configuration Management System.Threading Ports AccessControl Policy System.CodeDom System.Resources System.Message Bus.Security Credentials Principal System.Management System.Runtime System.Reflection Cryptography Token System.Deployment Serialization InteropServices System.EnterpriseServices System.Diagnostics CompilerServices System.Transactions © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Internet Connection Firewall 11/17/2018 9:30 PM Presentation Data Communication Communication Avalon Avalon Windows Forms Windows Forms ASP.NET WinFS WinFS ADO.NET Indigo Indigo Collaboration Document UI Media Object T/SQL XML Service Models Services Schemas Desktop Window Manager Controls Interop Engine Connectivity Messaging Services Desktop Services Adaptive UI Engine People Group ObjectSpaces People and Groups Synchronization (WinFS, Win32..) Channels (Datagram, Reliable, Peer, …) Policy Engine Page/Site Composition Calendar Media Queuing Eventing Routing Presentation Object Manager Desktop Composition Engine Application Services Document … DataSet Channel Security Collaboration History Media Services InfoAgent (PreferenceRules..) Data Model Transport Channels (IPC, HTTP, TCP…) Animation and Composition Media Processing Capture and Sourcing Designer Services Personalization and Profiling Services Items SQL XML Message Encoder Real-Time Activities System Services FileSystem Services (MetaDataHandlers..) Relationships Providers Communications Manager (Port) Transaction Signaling Framework Hardware Rendering Software Rendering and Sinks Controls Membership and Security Services Extensions Federation Base Operating System Services CLR Base Class Libraries Network Class Library Application Deployment Engine (Click-Once) Memory Manager Code Execution Loader Security Serialization Hosting Layer Network Services Demand Activation and Protocol Health GDI/GDI+ Window Manager Global Audio Engine Direct 3D Graphics Transactions Identity & Security System Storage PNRP Native WiFi SIP Management Services (Event Logs, Tracing, Probes, Auto Update, Admin) TCP Listener UDP Listener IPC Listener Graphics drivers Lightweight Transactions Transaction Coordinator Virtual Shadow Copy Service File Replication Service Virtual Disk Service Internet Connection Firewall DDI Input Manager Audio Drivers DirectX Graphics Mini port Kernel Transaction Manager Logging Service Redirectors Distributed File System Filter Manager Cache Manager Protocols Filter Engine TCP, UDP IPV4, IPV6 IPSEC QOS HTTP Listener Transactional NTFS Plug and Play Memory Manager Power Manager Config Manager Process Manager Security Reference Monitor LPC Facility Universal Data Format FAT 16/32 IO Manager Device Drivers Kernel SCSI/FC 802.3 802.11 .. Kernel Mode Hardware Abstraction Layer © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Kernel Infrastructure for Tx 11/17/2018 9:30 PM Kernel Infrastructure for Tx TxF KTM System.IO System.Transactions WinLS © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Transactional NTFS (TxF) 11/17/2018 9:30 PM Transactional NTFS (TxF) Surendra Verma Software Design Engineer (Lead) Core File Services sverma@microsoft.com © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Transactional NTFS (TxF) 11/17/2018 9:30 PM Transactional NTFS (TxF) Web site publishing © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Functionality Transaction Support for all NTFS file operations 11/17/2018 9:30 PM Functionality Transaction Support for all NTFS file operations Full Atomicity, Consistency, Isolation, Durability Allow an arbitrary number of file system operations as an atomic unit Reads, writes, file creations, deletions, renames, etc. Memory mapping Network file access via SMB © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Usage Scenarios For TxF 11/17/2018 9:30 PM Usage Scenarios For TxF Safe and efficient in-place update of documents Transactional multi-file update and network transfer Files in NTFS, related data in another transactional store such as SQL WinFS Filestream (File Backed) © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Example Usage – Compliance, Archive & Retrieval 11/17/2018 9:30 PM Example Usage – Compliance, Archive & Retrieval Metadata index (SQL) Dequeue message Add metadata to index Store message in NTFS Tx Exchange Server Browser MSMQ Client NTFS Query index Add “read” record to database Add “read” record to message in NTFS © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Transactional NTFS (TxF) 11/17/2018 9:30 PM Transactional NTFS (TxF) Transactional File I/O © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

11/17/2018 9:30 PM Isolation And Locking Transactions don’t see changes made by other transactions Isolation for non-transacted readers File is the unit of locking Readers don’t block © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Logging Modes Simple Mode Full Mode 11/17/2018 9:30 PM Logging Modes Simple Mode Only before-images of changes logged Minimal logging for common cases Create a new file and write to it Delete or rename a file Extend a file Fully truncate or overwrite a file Full Mode Full change history over time in log Very Useful, comes at a cost © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Build Powerful Solutions 11/17/2018 9:30 PM Build Powerful Solutions Disaster recovery Specify and backup a set of files Set Logging Mode to Full Continuously Archive log Restore the set of files and archived log from backup Roll-forward using the log to a chosen point in time © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Resource Units (RU) Subset of a volume: System-managed Resource Unit 11/17/2018 9:30 PM Resource Units (RU) Subset of a volume: You create it for a directory tree Keep data and transactional meta-data in one unit Unit moves as a whole You control start, stop and policies System-managed Resource Unit Includes files not in any other RU Self managed – logging mode, … © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

How To Program To It System.IO, System.Transaction, Win32 11/17/2018 9:30 PM How To Program To It System.IO, System.Transaction, Win32 Start Transaction Associate with current context Do multiple file operations Commit/Rollback Transaction Try Win32 APIs in Hands On Labs © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Kernel (mode) Transaction Manager 11/17/2018 9:30 PM Kernel (mode) Transaction Manager Rajeev Nagar Lead Program Manager Core File Services rajeevn@microsoft.com © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Technology Overview Kernel infrastructure for transaction support 11/17/2018 9:30 PM Technology Overview Kernel infrastructure for transaction support Services for both kernel and user mode components Superior performance and scale Well integrated with Distributed Transaction Services – supplies the state machine core for distributed transactions © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Distributed Transaction Flow 11/17/2018 9:30 PM Distributed Transaction Flow 4 Distribute Tx 4a App CRM CRM WinFX App App 5 SQL Update WinFX SQL Server Create Tx File I/O (Tx) 2 1 TxF (RM) KTM 3 Enlist KTM TxF Machine 1 Machine 2 © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

KTM Features Include Full (fast) 2-phase commit on single machine 11/17/2018 9:30 PM KTM Features Include Full (fast) 2-phase commit on single machine Virtual clocks & save-points Roll-forward recovery Multiple log support Highly available (log failover) Integrated with protocol layers Kernel, Win32 and System.Transaction Interfaces © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Windows Logging Service 11/17/2018 9:30 PM Windows Logging Service Dana Groff Program Manager Core File Services dgroff@microsoft.com © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Usage Scenarios When you need Examples Sequential Record Oriented Data 11/17/2018 9:30 PM Usage Scenarios When you need Sequential Record Oriented Data Grouped Records Persistence High performance Examples Transactional Stores Change Records for Replication Event Logging © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

1 Kernel Transition per flush 11/17/2018 9:30 PM Data Flow Log File Application Application Marshal Records Marshal Records Encoded  Unique Identifier (LSN)  Batched Single I/O Flush Queue 1 Kernel Transition per flush © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Windows Logging Service (WinLS) 11/17/2018 9:30 PM Windows Logging Service (WinLS) Create And Write To A Log © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Log:/full/path/name/to/baseLog::StreamName 11/17/2018 9:30 PM Log Records in Streams Log:/full/path/name/to/baseLog::StreamName One or many streams in a log No need to change to code, just change the logname! © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Windows Logging Service (WinLS) 11/17/2018 9:30 PM Windows Logging Service (WinLS) Easy Streams © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

11/17/2018 9:30 PM Recycling Space Restart Area A R C H I V E Archiving without interrupting reading and writing to the log Restart Area : dedicated area to write restart / recovery data. © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Windows Logging Service (WinLS) 11/17/2018 9:30 PM Windows Logging Service (WinLS) Recycling Log Space © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Key Features To Remember 11/17/2018 9:30 PM Key Features To Remember Fast, dependable, and optimized I/O Multiple streams without added coding Services for transactional logging APIs: Managed Code (not available for PDC) Win32 Kernel © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Community Resources Get Your Questions Answered! 11/17/2018 9:30 PM Community Resources Get Your Questions Answered! HOL HOL-404 Transactional File System Meet the team Thursday 9 am to 11 am Send Us Email 309 Foyer connect with PDC 2003 Architecture and Infrastructure Speakers and their product team members Newsgroups: microsoft.public.windows  .developer.winfx.fundamentals © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

© 2003-2004 Microsoft Corporation. All rights reserved. 11/17/2018 9:30 PM © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary. © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

11/17/2018 9:30 PM © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.