1. Data Flow and SDLC HUSH via Firm Solutions, Inc. Virginia Mushkatblat copyright

2. Architecture : definition From the Greek ἀρχιτέκτων – arkhitekton,ἀρχιτέκτων ἀρχι-ἀρχι- "chief" τέκτωντέκτων "builder, carpenter, mason“ The science of designing: the ideas and the constraints balancing act Protecting from flood yet pretty and convenient : stilts prevent from water damage Roman Dome: majestic and supporting the sound, yet quake proof

3. Architecture: definition Applying the term “Architecture” to software systems is a metaphor High level structure Discipline of creating architecture Documentation of this high level structure The important stuff—whatever that is By Martin Fowler in "Design - Who needs an architect” Things that are hard to change Always in context of environment

4. Architectural Analysis: functional requirements : what the system does runtime non-functional requirements: how well the system will perform development-time non-functional requirements: business requirements and environmental contexts of a system :legal, social, financial, competitive, and technology concerns reliability operability Performance efficiency security Compatibility maintainabilitytransferability

5. Different Companies – Different Needs Start Ups New Development. No data to start with. Developed Organizations New Development. Data Already Exists. New data and existing Data Feeds. Continuous Development. A lot of Data in the System. Maintenance Stage. Migrations.

6. SDLC –Part Of Architecture WATERFALL: analyse/design/code/test/deplo y/maintain AGILE: ask/code/integrate/demo/iterat e/deploy Fred Brooks, Mythical Man-Months concept of the process as part of architecture human elements of software engineering

7. New Development When we start from Scratch, there is nothing and we can initially treat everything as if databases were code.

8. New Development

9. To Infinity And Beyond : in Production Big day being behind, we are in production  Lots of transactions  Database size reaches GB, TB, PT – think Amazon, we all want our business be there  We scale in various ways, yet the CRUD logic is the same  Master data matures and gets into DB via GUI

10. Production and Source Control: “no ‘change’ left behind act” We address two code bases with  ROOT. Production code base, the version of the “Truth”. In development it is also called BASELINE. Code is out there, in the trenches and so is DATA.  BRANCH. The future of our product, in the branch, all the changes that will flow to production eventually. How data will go?

11. Problem child of the builds: DATA A couple of definitions, first: Extreme Programming (XP) improve software quality and responsiveness to changing customer requirements. Continuous integration (CI) the practice of merging all developer workspaces with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP).extreme programming How does data fit the process?

12. DATA Concepts Master Data It is standardized and managed ( formally or not) across enterprise Often has external source from other governing body It is relatively un-changeable. Change is in the context of user based transaction Examples: Roles, Countries, Rates. Rates can be volatile, yet have to be uniform across enterprise in close to real-time Transactional Data User introduces transactions

13. Problem child of the builds: DATA Master data and number of test cases grows Builds take time, either continuous or daily Inserts, updates, deletes are physical phenomena: they take time to write to disk. Not everyone is rich to get SSDs or Fusion IO for development environments Image circa Microsoft

14. Solution Strategies Constraints: Operational requirements for  data availability  data consistency  performance  data integrity Development Requirements for:  development time  skill sets Environmental  Monetary (space and processors)  Political (we just do not want to use third party) Architectural patterns:  Backup/Load  ETL Solutions (different kinds)

15. Backup/Load: truncate data; refactoring USE [master] GO ALTER DATABASE [TestSQLSatRefactor] SET SINGLE_USER WITH ROLLBACK IMMEDIATE GO RESTORE DATABASE [TestSQLSatRefactor] FROM DISK = N'C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL \Backup\TestSQLSat.bak' WITH MOVE 'TestSQLSat' TO 'C:\Program Files\Microsoft SQL Server \MSSQL10.MSSQLSERVER\MSSQL\DATA\TestSQLSatRefactor.mdf', MOVE 'TestSQLSat_log' TO 'C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER \MSSQL\DATA\TestSQLSatRefactor_log.ldf', FILE = 1, NOUNLOAD, REPLACE, STATS = 10; GO

16. Perils of environments: Data homogeneity across environments. How close should environments be in terms of data? The sandbox and integration environments hold the least amount of data. A rule of thumb: data set sufficient enough for developing functional requirements. Pros: speeds up development, cons: can’t accommodate all the test cases and needs constant data set assessments. The QA/Staging should hold complete data set to allow for UAT and for performance and regression testing. Break Fix environment holds data set and schema as close to production as possible to allow for speedy production issues resolutions. Physical Constraints: goes without saying. NO disk space means no disk space. Environment SLAs: is development around the clock with international development team, 24/7 or only happens in one place with 8 hours development day/time? Rate of refreshes: depends on whether we do continuous deployments or scheduled releases Data retention: how much and often transactional data gets purged? Schema management: schema/data in source control? Are deployments automated including data? Are there specific structures that support metadata?

17. Development When Data Exists. Backup / Load When: Referential Integrity of your DB is custom, not documented testing sub-setting is essential You have space You have time to do Change management and corresponding data transforms Static Data Masking : GLBA, HIPAA, PSS/DSA

18. Yet Another Way: ETL When: Staging space is a consideration You need instant deltas of data You want masking be part of your ETL: No significant upfront investment No learning curve Part of development toolbox

19. Dreaded Maintenance: Break/Fix

20. QA/Staging/Break Fix Environments !!!! BE AWARE!!!! Backup/Load takes time. Count on it. Refactoring takes time. Count on it. ETL takes time. Count on it. Masking has its own architectures. Chose the one appropriate.

21. PRACTICAL SECTION: TOOLS AND EXAMPLES Let’s say we have TFS. We create a Database Project:

22. TFS: continued moving to solution explorer Adding a project What is next?

23. TFS: continued Just one way of many, let’s create a DB Add Objects: Three tables Views Procedures

24. TFS: continued we can import existing database from the sandbox:

25. TFS: continued It created the objects that currently exist in the database:

26. Database: data to test Let’s say we created some master data and transactional data to test: ----------------------------------- --- Insert Master Data ------------ ----------------------------------- INSERT INTO [dbo].[role]([role]) VALUES ('Presenter') INSERT INTO [dbo].[role]([role]) VALUES ('Sponsor') INSERT INTO [dbo].[role]([role]) VALUES ('Participant') select * from [dbo].[role]

27. Database: data to test ----------------------------------- --- Insert Test Cases ------------ ----------------------------------- INSERT INTO [dbo].[person] ([name],[email],[gender],[date_of_birth],[place_id]) VALUES ('Virginia M','genya@firmsolutionsinc.com','f',null,null) INSERT INTO [dbo].[person] ([name],[email],[gender],[date_of_birth],[place_id]) VALUES ('Lynn Langit','lynn@lynnlangit.com','f',null,null) INSERT INTO [dbo].[person] ([name],[email],[gender],[date_of_birth],[place_id]) VALUES ('Andrew Karcher ','andrew@akarcher.com','m',null,null) INSERT INTO [dbo].[person] ([name],[email],[gender],[date_of_birth],[place_id]) VALUES ('Josh S ','Josh@js.com','m',null,null) --Virginia INSERT INTO [dbo].[person_role]([person_id],[role_id]) VALUES (1,1) INSERT INTO [dbo].[person_role]([person_id],[role_id]) VALUES (1,2) INSERT INTO [dbo].[person_role]([person_id],[role_id]) VALUES (1,3) -- Lynn INSERT INTO [dbo].[person_role]([person_id],[role_id]) VALUES (2,1) INSERT INTO [dbo].[person_role]([person_id],[role_id]) VALUES (2,3) --Andrew INSERT INTO [dbo].[person_role]([person_id],[role_id]) VALUES (3,2) INSERT INTO [dbo].[person_role]([person_id],[role_id]) VALUES (3,3) --Josh INSERT INTO [dbo].[person_role]([person_id],[role_id]) VALUES (4,3)

28. TFS: How we deal with data? If I choose in the best traditions of building a sandbox management to drop and recreate my still small database, data is going to be lost:

29. TFS: How do we deal with data? Introducing Change: After the change is deployed, the database loses data:

30. The Wonderful Post Deployment Script We can retain the data if we put it into the post-deployment script Here is how it looks:

31. The Wonderful Post Deployment Script The Test Cases:

32. Deploy Again After the deployment, the data stayed:

33. TFS: let QA do their job Sometimes, we do not want to deploy transactional data to the QA environment. We let QA test independently. We can put environmental variables in the script:

34. Small teams -introducing Data Tier Applications. DAC. Working with Data Tier Via Visual Studio Significantly improved interface and functionality

35. DAC –continued. It warns of Data Issues

36. DAC –continued. It prepares deployment script

37. DAC – continued. It automates scripts and deployments

39. DAC – continued. Data is in Scripts

40. DAC – continued. Data?

41. Data Flow with ETL Masking Example

42. Used Tools and Other Tools That Help What I used: SQL Server, SSMS, SSIS, Data Quality Services, TFS, VS, Visio If you do not have VS and TFS: Red Gate: SQL Compare, SQL Data Compare,SQL Data Generator, SQL Source Control Embarcadero: E/R Studio, DB Change Manager HUSH-HUSH Masking components for ETL architectures from FSI