1. Under the Hood: Storage and Advanced Application Development Brian Dewey DAT406 Group Program Manager Microsoft Corporation

2. 2 Advanced Storage Applications PlatformSolution System Protection Points Transactions Windows Backup System Restore “Previous versions” Trustworthy Symbolic links NFSCross-Platform Remote Differential Compression DFS PC-PC Sync Offline files Distributed

3. 3 What Is RDC? An algorithm to copy just the differences between two files over a network A library and SDK for Windows that let you add differential copy to your application An awesome way to reduce bandwidth used by distributed applications

4. 4 RDC In Action

5. 5 Overview Of The Core Algorithm The files are divided into variable length chunks based on their contents Fingerprint function computed based on a trailing window of the file’s contents (H3 or Rabin) Modifications affect only small number of chunks Hashes (MD4) are computed for each chunk, on both server and client Strong signature for entire file deals with collisions The server transmits the list of strong hashes to the client For large files, the algorithm is applied recursively at this point The client assembles file by using existing chunks from old file and requesting missing chunks Original file Updated file MD4 21 … MD4 25 Fetch new chunks 3, 4 Updated file The quick fox jumped over the lazy brown dog. the fox jumping over him. “The brown dog was” “so lazy that he …” copy ClientServer MD4 21 MD4 22 MD4 23 MD4 24 MD4 25 MD4 11 MD4 12 MD4 13 MD4 14 Request file The quick fox jumped over the lazy brown dog. The dog was so lazy that he didn’t notice the fox jumping over him. The quick fox jumped over the lazy brown dog. The brown dog was the fox jumping over him. so lazy that he didn’t notice The quick fox jumped over the lazy brown dog. The brown dog was the fox jumping over him. so lazy that he didn’t notice [use recursion]

6. 6 RDC Sample ServerServerClientClient RdcSdkTestClient.exe RdcSdkTestServer.dll Msrdc.dll Msrdc.dllDCOM Source Seed Target Simplest application of RDC – no advanced tricks Will be in Beta 2 SDK – not today’s Contact rdctalk@microsoft.com to get started early rdctalk@microsoft.com

7. 7 Overview Of The Core Algorithm The files are divided into variable length chunks based on their contents Fingerprint function computed based on a trailing window of the file’s contents (H3 or Rabin) Modifications affect only small number of chunks Hashes (MD4) are computed for each chunk, on both server and client The server transmits the list of strong hashes to the client The client assembles file by using existing chunks from old file and requesting missing chunks Original file Updated file MD4 21 … MD4 25 Fetch new chunks 3, 4 Updated file The quick fox jumped over the lazy brown dog. the fox jumping over him. “The brown dog was” “so lazy that he …” copy ClientServer MD4 21 MD4 22 MD4 23 MD4 24 MD4 25 MD4 11 MD4 12 MD4 13 MD4 14 Request file The quick fox jumped over the lazy brown dog. The dog was so lazy that he didn’t notice the fox jumping over him. The quick fox jumped over the lazy brown dog. The brown dog was the fox jumping over him. so lazy that he didn’t notice The quick fox jumped over the lazy brown dog. The brown dog was the fox jumping over him. so lazy that he didn’t notice [use recursion]

8. 8 Tips For Using RDC Target file verification using a strong hash Do regular dictionary compression on traffic Cache signatures Multiple seed files Use recursion Batch needs Transfer multiple files in parallel Tune, tune, tune

9. 9 Symbolic Links Symbolic links let one file/directory transparently redirect to another file/directory by name Example: C:\public\link can refer to: C:\Users\BillG\Documents – “absolute” link..\..\Users\BillG\Documents – “relative” link \\machine\Users\BillG\Documents\\machine\Users\BillG\Documents -- “remote” link \\machine\Users\BillG\Documents

10. 10 Symbolic Links In Action

11. 11 C:\Users\BillG\Documents\link\file.txt C:\Users\SteveB\Documents\Shared\file.txt Under the Hood: Symbolic Links link -> C:\Users\SteveB\Documents\Shared (Stored as NTFS reparse point) C:\Users\BillG\Documents\link

12. 12 C:\Users\BillG\Documents\..\..\public\documents\file.txt C:\Users\public\documents\file.txt Under the Hood: Symbolic Links C:\Users\BillG\Documents\link\file.txt link ->..\..\public\documents link

13. 13 Symbolic Link Tips Know what functions work on the link and what work on the target General rule: “Data” operations work on the target, “metadata” on the link Data: Open, copy, modify Metadata: Delete, rename Currently documented on MSDN Use relative symbolic links when you copy subtrees Remember: Links evaluated on the client Be very careful with name parsing \\machine\public\directory\file.txt\\machine\public\directory\file.txt might not be a file on \\machine... \\machine \\machine\public\directory\file.txt \\machine

14. 14 System Protection Points Extension of “shadow copy” technology from Windows Server 2003 Shadow copies are copies of all data on disk frozen at a point in time Copy-on-write ensures minimal physical disk space consumption Single disk shadow copy with multiple uses System Restore “Safe documents” “Safe system” Single document recovery

15. 15 System Protection Points In Action

16. 16 System Protection Point Tips Shadow copies may be reclaimed at any time If you find an old version of data that is important, you must copy it to the “live” volume if you want to keep it! Shadow copies are read-only

17. 17 Transactions In Longhorn Simple way to add data reliability to your application Can transact updates to file system and registry Transactions can be coordinated with databases and with other machines File system transactions work remotely

18. 18 Transactions In Action

19. 19 Transactions Under The Hood

20. 20 Transactional NTFS Tips Transactions lock the entire file Keep transaction lifespan short Transactions are all-or-nothing Consider intermediate checkpoints for operations like bulk file copies You can’t rollback once you commit

21. 21 Where To Learn More Transactions: FUN320 (Thursday, 5:15) Sync Center: DAT317 (Thursday, 10:00) RDC: rdctalk@microsoft.com

22. 22 Call to Action Platform Feature Availability Distributed Remote Differential Compression Beta 2 Cross-Platform Symbolic links Today Trustworthy System Protection Points TransactionsTodayToday Fill out your evaluation!

23. © 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.