3. Stories from the trenches - Partitioning as a design pattern About the FLIS project Data warehouse heroes and architecture Partitioning in many disguises

4. The FLIS Project

8. Access to your own data? ~ 72 mill. kr. to get data for 4 years (10 mill. euro)

11. Bill Ralph Dan

22. par·ti·tion (pär-tshn)n. 1.a. The act or process of dividing something into parts. b. The state of being so divided. 2.a. Something that divides or separates, as a wall dividing one room or cubicle from another. b. A wall, septum, or other separating membrane in an organism. 3. A part or section into which something has been divided. … Source: http://www.thefreedictionary.com/partitioning

24. Dividing something into parts Files Database Table State of being so divided ETL developers vs. Customer Developers vs (evil?) DBA’s Backups (what is a full backup anyway)

25. Xml, csv, xls, fixed format 1 file Data from 1 or more tables Data from 1 or more municipality 30 different file naming schemes

26. Csv 1 file 1 table Data from just 1 municipality 1 file naming scheme

27. Files (complex) File-dsa mappings (complex) Dsa tables Files (complex) Preprocessor (complex) File-dsa mappings (simple) Dsa tables

28. 1 data warehouse layer = 1 database Scaling IO-pattern for a data flow

29. 1 database 4 LUNS, RAID 10 4 datafiles

30. Filegroups PRIMARY 160MB BIG ONE A lot of data

31. Table partitioning (2005 EE feature) Pruning Divide the data warehouse into 100 parts 1/100 the size Switching Separation of readers and writers Fast

32. DSA Partition by (municipality_id, month_year) EDW and data marts Partition by municipality_id

34. CREATE TABLE [dbo].[partition_test]( [Kommune_id] [varchar](11) NULL, [Kommunenummer] [nvarchar](4000) NULL, [Distrikt_kode] [nvarchar](4000) NULL, [Distrikt_type] [nvarchar](4000) NULL, [Distrikt_tekst] [nvarchar](4000) NULL, [Cpr_Dist_Tekst_TS] [nvarchar](4000) NULL ) ON [partition_test_pt_sc]([Kommune_id]) GO

36. Affinity masking Not really possible in Oracle – even on Windows

39. MappingID (int) DestinationColumnFK (int) SourceColumnFK (int) Mapping ColumnID (int) ColumnName (varchar) Column DWLayerID (int) DWLayerName (varchar) DWLayer TransformationID (int) FlowFK (int) Transformation ColumnTypeFK (int) ColumnTypeID (int) ColumnTypeName (varchar) ColumnType (source data) TableFK (int) TableTypeID (int) TableTypeName (varchar) TableType TransformationSchemaID (int) TransformationTypeID (int) TransformationSchema StandardColumnID (int) StandardColumnName (varchar) StandardColumn DWLayerFK (int) Logic (varchar) TransformationTypeID (int) TransformationTypeName (varchar) TransformationType TransformationSchemaName (varchar) FlowID (int) Flow FlowName (varchar) FlowTypeFK (int) FlowTypeID (int) FlowType FlowTypeName (varchar) TransformationFK (int) TransformationSchemaFK (int) Logic (varchar) DataTypeFK (int) TableID (int) TableName (varchar) Table DWLayerFK (int) TableTypeFK (int) Order??? DataTypeID (int) DataTypeName (varchar) DataType

40. Developers vs. (evil) DBA’s Meta data on table definitions Script ddl from meta data Hide partitioning in ddl Partition Scheme Change File group design

41. Backups: DBA’s vs. Backup administrator Partitioning helps us divide data into Hot (C)old (and therefore read only) Only backup hot partitions

42. Restoring: DBA vs. SQL server  PRIMARY filegroup 150 MB Default filegroup Big Restore database only primary filegroup => online

43. DW vs OLTP and OLAP DW server ETL processing OLAP databases/cubes Reporting server Sharepoint databases (OLTP) restoring OLAP databases/cubes

46. Page compression results table Factor (size)Num rowsnum reads cpu time (ms) elapsed time (ms) dg118913237231739 dg12178264743311517 dg11089132037812817551 dg1_page_compr18913214315748 dg1_page_compr2178264285621512 dg1_page_compr1089132014274217596 codevartypevarval DA000DGALT00K00 DA000DGCAT06M38A DA000DGPROP26X01 DA001DGALT00K00 DA001DGCAT06M38A DA001DGPROP26X01 DA009DGALT00K00 DA009DGCAT06M38A DA009DGPROP26X01

47. or

48. Evaluation Create a Text message on your phone and send it to 1919 with the content: DB301 5 5 5 I liked it a lot Session Code Kennie Performance (1 to 5) Match of technical Level (1 to 5) Relevance (1 to 5) Comments (optional) Evaluation Scale: 1 = Very bad 2 = Bad 3 = Relevant 4 = Good 5 = Very Good! Questions: Speaker Performance Relevance according to your work Match of technical level according to published level Comments

49. © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.