ALM and practical guidance for VSTS Database Projects Jens K. Suessmeyer Developer hearted, relational minded Visual Studio ALM Core Ranger

Session objectives Get to know the database project advantages over common database development approaches Get to know the relevance of Application lifecycle management for a database Map the stages of an ALM process to database projects Get to know the importance of a formalized process to develop, maintain, test and deploy databases Get to know what is in the Visual Studio ALM Ranger Guidance for Database projects for you 2

about:me Jens K. Süßmeyer, Consultant from Microsoft Services Germany Main focus areas.Net Development Database development Consulting SQL Server 2008 MCM Visual Studio ALM Core Ranger 3

Question Did you ever had inconsistencies in your database development that you think could have been avoidable ?

General understanding of lifecycle Management 5 What is ALM ? (

Database Lifecycle DatabaseProjectDatabaseProject BuildBuildDeploymentDeployment TestTest DevelopmentDevelopment Service Creation (Requirements definition / Design / Implementation) Service management (Deployment / Operation / Optimizing)

The evolution of development tools 7 Tools for team development (

Version Control Challenge CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) class AuctionApplication ( int id; void MethodA(); ) class AuctionApplication ( int id; void MethodA(); )DatabaseDatabase class AuctionApplication ( int id; void MethodA(); void MethodB(); ) class AuctionApplication ( int id; void MethodA(); void MethodB(); ) class AuctionApplication ( int id; string cacheTitle; void MethodA(); void MethodB(); ) class AuctionApplication ( int id; string cacheTitle; void MethodA(); void MethodB(); ) Revision HistoryAppApp V 1 V 2V 3 ALTER TABLE dbo.Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) ALTER TABLE dbo.Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) ALTER TABLE dbo.Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) ALTER TABLE dbo.Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name)

Hand crafted versioning -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END

Hand crafted versioning -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END V1 => V2 -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END V2 => V3 -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END V3 => V4 -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END V1 => V3 -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END V2 => V4 -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END -- version 1 Add table dbo.Auction IF OBJECT_ID (N'dbo.Auction', N'U') IS NULL BEGIN CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) END -- version 2 Add PK Au_PK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_PK' AND type = 'PK') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_PK PRIMARY KEY (id) END -- version 3 Add UC Au_SK IF NOT EXISTS (SELECT * FROM sys.key_constraints WHERE name = 'Au_SK' AND type = ‘UQ') BEGIN ALTER TABLE Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name) END V1 => V4

Hand crafted versioning Instead of having to guess the order by names, scripts would need to be omnipotent and not version agnostics in first place. Produces a high effort in maintaining scripts and persisting the knowledge of the order

The enemy of a copy & paste deployment - Data Unlike deployments of applications / assemblies in common, the database deployment will need to preserve data Incremental deployment needs to provide a way to migrate data without loosing information or violate referential integrity leaving the database in an inconsistent state Data scripts will need to honor the dependencies, constraints, dependent objects

Deployment – Visual Studio Database Projects CREATE TABLE dbo.Auction ( id INT NOT NULL PRIMARY KEY, name VARCHAR(25) NOT NULL UNIQUE, start DATETIME NULL, len INT NULL ) CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) CREATE TABLE dbo.Auction ( id INT NOT NULL, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) CREATE TABLE dbo.Auction ( id INT NOT NULL PRIMARY KEY, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) CREATE TABLE dbo.Auction ( id INT NOT NULL PRIMARY KEY, name VARCHAR(25) NOT NULL, start DATETIME NULL, len INT NULL ) Revision HistoryV 1 V 2V 3 CREATE TABLE dbo.Auction ( id INT NOT NULL PRIMARY KEY, name VARCHAR(25) NOT NULL UNIQUE, start DATETIME NULL, len INT NULL ) CREATE TABLE dbo.Auction ( id INT NOT NULL PRIMARY KEY, name VARCHAR(25) NOT NULL UNIQUE, start DATETIME NULL, len INT NULL ) ALTER TABLE dbo.Auction WITH CHECK ADD CONSTRAINT Au_SK UNIQUE (name)

The advantages of Database projects The from the database uncoupled schema model contains all the necessary information to create / update a database and represents the state “as-should-be”. The schema motion scripts are created on deployment time, NOT development time, letting the developer focus on his development, NOT deployment goals. Referential constraints etc. Are automatically honored. Maintaining a conceptual models instead of scripts let you do easily use all the features of the ALM platform, like merging Branching

Database development with various tools 15 Benutzer Testing Database Tester ProductionalDatabase SQL Administrator Design Database Source Control System SQL Developer

Visual Studio ALM Rangers Mission Accelerate adoption of Visual Studio with out-of-band solutions for missing features or guidance Joint Venture Co-funded and co-lead by Services and PG Bijan Javidi/PG and Jeff Jurvis/Services Community readiness Gather field experience Empower the community Collaboration ~130 Extended Rangers ~46 External Rangers 10 Core Rangers Find us here … MSDN Blog ALM Rangers IP on Campus Want to join? us on

Visual Studio Database Projects guidance Rangers guidance vs. MSDN MSDN explains features not scenarios This guidance is scenario based Shows ideas and experiences based on real world engagements

The guidance materialization We provide best practices and guidance in each sub topics in a Guidance document Corresponding HOLs

Application lifecycle sub topics

Development DatabaseProjectDatabaseProject BuildBuildDeploymentDeployment TestTest DevelopmentDevelopment Service Creation (Requirements definition / Design / Implementation) Service management (Deployment / Operation / Optimizing)

Development Why is sometimes 0…

Development Internal database validation decreases the chance of getting unsuccessful builds or inconsistent projects artifacts by modeling and validating the schema on the fly Developers can be provided with templates to adhere to certain guidelines and with static code analysis rules to ensure coding standards Developers can work without having any dependency on a database prior to deployment.

Development / Database guidance The database guidance document shows how to manage projects streamline development when to use / not to use certain features like different project types (Server, Composite / Partial) how to apply known concepts of existing guidance documents like Branching & Merging to the database world be aware of limitations during development and project management and how to get around them

DEMO Working with database projects

Build DatabaseProjectDatabaseProject BuildBuildDeploymentDeployment TestTest DevelopmentDevelopment Service Creation (Requirements definition / Design / Implementation) Service management (Deployment / Operation / Optimizing)

Build Why a successful build is sometimes not useful at all …

Build Like any other projects Database projects can be included in your build definition within TFS or any external build processes. The guidance will in addition help you to: Continuous integration with VSDB Building and Deploying outside Team Build Inclusion of automated tests / test lists to be included within the build Provide you with the option to deploy databases automatically with the build Build and Deployment packaging using WIX

DEMO Managing a (W)indows (I)nstaller (X)ML Project for Database Projects

Test DatabaseProjectDatabaseProject BuildBuildDeploymentDeployment TestTest DevelopmentDevelopment Service Creation (Requirements definition / Design / Implementation) Service management (Deployment / Operation / Optimizing)

Test Why testing should be important…

Test The cost of lousy tests…

Test Creation of tests was put into a designer to enable database developers and tester to create their Unit tests without having to learn any coding language Unit Tests can run on sample data as well as random or real data to ensure a stable quality Unit Tests can be included in the automated build to ensure continuous checks of your database artifacts automate processes.

Test / Creating useful data Simple data generators Strings (char, varchar, nvarchar…) Numbers (smallint, int, bigint, float…) Binary (varbinary, image…) Date and Time GUID and Bit Complex generators Foreign Key Regular Expression / Unique Expression Generator Data Bound / Sequential Data Bound FileBound Generator XMLGenerator You can cover the full complexity of data types here to make your code robust !

Test / Creating useful data Extend it to your need and extend the framework by simply writing your data generator. Publishing works with copy & paste to the Visual Studio machine. Include interface that you might already use by accessing internals data sources, etc.

Test / Database guidance The database guidance document shows how to create simple tests and integrate them in the project create a multi-step process to build / deploy / load databases with random or live data and test against that Integrate your tests within your automated build definitions

DEMO Integrated creation of Unit tests

Test Extend it to your need and extend the framework by simply writing your own condition. Publishing works with copy & paste to the Visual Studio machine

Deployment DatabaseProjectDatabaseProject BuildBuildDeploymentDeployment TestTest DevelopmentDevelopment Service Creation (Requirements definition / Design / Implementation) Service management (Deployment / Operation / Optimizing)

Deployment Without knowing your deployment environment you might be surprised of the results…

Deployment The transistion to deployment should much more than just throwing the build artifacts over the fence. There is much more valueable information buried in here: Finding bugs and deployment issues early Feasability of upgrade scenarios / discover major shostoppers Durations of database upgrades Needed resources for database upgrades (like log space, usage of database parameters, etc.) Building confidence about the delivered quality

Deployment In a perfect world, most of the deployment checks can be done on the „assumed“ database enviroment as the source control is the source of truth.

Deployment Database administrator doubts “The database upgrade will drop tables and created index and will make my database slow.” “I doubt that the VSDBCMD process really does all the magic behind the scenes and honors all changes I have done in the database” “I will not run any command line tool against the database before I haven´t seen the actual.sql code executed beforehand.”

Deployment Database developer doubts “We did all the upgrade testing against the database, but who ensures that the assumed schema on the real database still exists?” “The database administrators created several objects in addition which we cannot guarantee to be preserved over the deployment.” “The duration of the upgrade process is unpredictable as the database administrator might have created several indexes which need to be preserved.”

Deployment Finding model drifts (extract from Guidance document) Make sure that you tested the application database with the right version you once have been given WITHOUT having access to the database (Trusted DBA version) Make sure that you tested the application database with the right version you once have been given having access to the database (Trusted developer version) Review the deployment script prior executing them in the database (Suspicious DBA version)

DEMO Finding model drifts

Summary Visual Studio Database Projects can ease development, build, test and deployment process by providing the right tools. VSTS Database Projects cover the operative part of ALM for Database projects. Many formalized processes with little effort will help you to ensure quality across the database projects The Database Guidance document can help you to find out how you can get around limitations, implement best practices and handle special situations. 46

Stay up to date with MSDN Belux Register for our newsletters and stay up to date: Technical updates Event announcements and registration Top downloads Follow our blog Join us on Facebook LinkedIn: Download MSDN/TechNet Desktop Gadget

TechDays 2011 On-Demand Watch this session on-demand via Channel9 Download to your favorite MP3 or video player Get access to slides and recommended resources by the speakers

Additional Ressources My blog: Barclay Hill (PM of Visual Studio Data Tools): Visual Studio Database Projects Guidance: Visual Studio Rangers:

Questions ? ? ? ?

THANK YOU