1. Introducing the Limitless DBA Copy Data Management- DBA Style
Kellyn Pot’Vin-Gorman
Introducing the Limitless DBA Copy Data Management- DBA Style

2. Kellyn Pot’Vin-Gorman
Technical Intelligence Manager for the Office of CTO, Delphix
Multi-platform DBA, (Oracle, MSSQL, MySQL, Sybase, PostgreSQL…..)
Oracle ACE Director, (Alumni), Oak Table
APEX Women in Technology Award, CTA
STEM education with Raspberry Pi and Python
Liaison for Denver SQL Server User Group
Rocky Mountain Oracle Training Days Conference Director and Board Director
Author, blogger, (

3. Copy Data Management: Virtualized sanity for the DBA Realist.
Provisioning: Patching, refreshing and if you ask me one more time!
Cloud: Cloudy with a chance of failures.
Security: Yo Developer- Is that the SA password taped to your monitor??

4. The Life of a DBA Provision Databases
Refresh and provide data to reporting, testing and development
Secure database environments
Optimize data access
Collaborate to solve business challenges
Talk about the future of the DBA with DevOps-
Learn other database platforms
Learn Shell, other than Powershell, learn Python and automation tools for DevOps

5. What is Copy Data Management, (CDM)
The management of all non-production databases.
Broad Term- Physical and virtual clones
Managed or unmanaged
Command line or User Interface, (or both)
Administrative, Infrastructure, security
ETL, subsets of data, as well as physical and virtual clones, backup, replication.
Where is all that data going?
DBA 1.0/2.0? Does it translate?
Manage all those copies.

6. Why Use Virtualization for CDM?
Storage costs
Thin-provisioning storage avoidance
Data transfer costs
Far less data transferred during provisioning/refresh operations
As opposed to the volume of data transferred using traditional cloning techniques
Simplifies Provisioning vs. archaic processes to copy data
80-90% storage savings from traditional migration methods.
Data In flight can be significant cost for many cloud vendors
Network is the new bottleneck. You can avoid that with less copies- one golden copy- we call it the “validated sync environment”

7. Copies: 90%+ Repeated Data
Spin a VIRTUAL database up a patch test, without having to remove a current development or test one.
PRODUCTION
Database/App Tier
1 TB
Storage Pool for Delphix
DEV
PATCH TEST QA
TEST
▶ ▶ ▶
Virtualize and Deployed▶ ▶ ▶
Read From Production
0.6 TB

8. Example Environment- 1TB

9. Data Virtualization: From Prod to Virtual
Validated Sync Environment
Backups over SMB
Compressed and
De-duplicated
SCSI/SSL
Source Environment

10. Data Virtualization: From Prod to Virtual
Validated Sync Environment
SCSI/SSL
Create as many VDBs as needed!
Backups over SMB
SCSI/SSL
Any Storage
Source Environment

11. Data Virtualization: Space Savings
10 TB?
1 TB
Validated Sync Target
Delphix Virtualization Engine
Dozen Virtual Database Copies, (VDBs)

12. Data Virtualization: How is it possible?
Delphix Virtualization Engine
Any Storage
Only Storage is
Written Blocks
Virtual Database

13. This is Data Version Control
Using any storage and only fraction of space
Syncs with native or third-party SQL Server backups
Can maintain two weeks of data changes
Managed just like any SQL Server database
Users can instantly provision a read/write virtual copy of a database
Can be used for replication, mirroring, and change data capture (CDC).

14. Administration Ease

15. Bookmarking and Branching
This is the interface for Developers and testers- they can bookmark before important tasks or rewind to any point in the process. They can bookmark and branch for full development/testing needs.
Release 1.2
Release 1.3
Branch 1.3

16. Epiphany e·piph·a·ny əˈpifənē/ noun
a (1) : a usually sudden manifestation or perception of the essential nature or meaning of something (2) : an intuitive grasp of reality through something (as an event) usually simple and striking (3) : an illuminating discovery, realization, or disclosure
b : a revealing scene or moment

17. Patching- Got the T-shirt
How often does Microsoft send patches?
Do we start picking priorities about what we apply depending on environment access, resources and such?

18. How Do We Build Out? How Long?
Finance
Days? HR
Weeks?
DBA has to commandeer a database for patch testing.
This has to be performed for EACH environment, 100’s or 1000’s of databases!
Most are not synchronized with production, different outcomes when released to production.
Bugs occurring in one, not another!
CRM
Months?

19. Now You Have To
Maintain Everything!

20. Patching and Upgrading Databases
Prod
UAT
Test
Dev
Each patch, would need to be applied to a development database, requiring outage to development teams and then tested before applying to test, UAT and then onto production.
This has to be performed to EACH environment, every SQL Server, each quarter.
Patching is
Finance
A Major Undertaking HR
CRM

21. Risks/Challenges Of This Approach
Downtime for valuable resources.
DBAs working afterhours
Each database must have it done and…
The tedious task must be performed over and over again.
Little opportunity for advanced learning.
Each database may experience different bugs.

22. Environment Virtualization, DB Style
Spin up a new VIRTUAL DB and SQL bin files and apply patch to it. V D
CRM V V
Finance V V HR V

23. After Testing, Apply to Production
No need to keep the extra VDBs post patch to prod. V D
CRM V V
Finance V V HR V

24. Environment Virtualization, DB Style
CRM
The Compressed Copies in the Delphix Engine are Upgraded!
Finance
Testing upgrades and patches can be greatly simplified using the portability and ease-of-use of Delphix Virtual Databases (VDBs). Here are two approaches that can be used, depending on the upgrade or patch.
Link the production database with the Delphix Server.
Provision a VDB at the existing patch level.
Patch the existing SQL Server bin files against the live VDB.
or 
Create the new SQL Server bin file directory and switch the VDB.
Rollback VDB or Refresh from production.
Repeat 3 or 4 until confident.
Once the process has been tested and confirmed, it can be rolled out with confidence into production. HR

25. Environment Virtualization, DB Style
In minutes, refresh and complete! V D
CRM
Finance V V HR

26. That’s It.

27. Patching and Upgrading with Virtualization
I didn’t have to take away a valuable resource’s database environment to test the patches.
I didn’t have to apply the patches to subsequent environments, as they are virtualized copies of the source, simply requiring a refresh from production, post final patch.
I save significant time that commonly has to be allocated to quarterly and annual maintenance for patching.
I apply the patch twice- once to test, once to production. I only need to refresh my environments after I’m done.
For releases, this can be “containerized”, simplifying release and if required, rollback.

28. The quickest way to complete a task is not having to do it.
And there’s the epiphany…
The quickest way to complete a task is not having to do it.

29. What if I’m Retro- Command Line?
The name of the VDB you want to create
The group in which to create the VDB
The Oracle database name
The Oracle database unique name
The Oracle database instance number
The Oracle database instance name
The source dSource or VDB from which you wish to provision
The SCN or timestamp of the point you want to provision from, (commands can be run to get the list of snapshots or timeflow ranges.)
The CLI, (Command Line Interface) for CDM should be robust and able to incorporate into DevOps

30. Choose Source Snapshot: >snapshot list database=<dbname>
timeflow ”<dbname>" timeflowRanges;
commit;
Create A Virtual Database:
>database provision;
delphix database provision > defaults
delphix database provision defaults > set container=<Vdbname>
delphix database provision defaults > commit;

31. What a CLI Scripted Refresh Looks Like
> database
> select <VDB name>
> refresh
> set timeflowPointParameters.type= (one of TimeflowPointBookmark, TimeflowPointBookmarkTag, TimeflowPointLocation, TimeflowPointSemantic, TimeflowPointTimestamp as appropriate)
> set timeflowPointParameters.location= (the location, timestamp, or bookmark you wish to refresh to)
> set timeflowPointParameters.timeflow= (the timeflow associated with location)
> commit
Still, very simple vs. what we write every day to support a simple create table or create database…
This can all be called via a Powershell script…

32. Use the Right Tool for the Job

33. When Do I USE the CLI or Graphical UI?
Monstrous deployments, (20+) benefit from a scripted, CLI option.
Single or several VDB deployment.
When investigating deep level issues, using the CLI can make details easier to search through.
When looking for “just the facts”, then a graphical UI is beneficial for high level error messages.
Self-service has great benefits and make it simple for those without deep level skills to take advantage of virtualized environments for development and testing.
The CLI helps with DevOps automation and orchestration.

34. Cloud Projects Requirements Initial Testing Project Timeline Migrate
Optimize
Migrate
Initial Testing
Requirements
Project Timeline
Do any of you see the problem with the high level project steps?
We commonly leave optimizing the environment until after we’ve migrated to the cloud.

35. How Are Companies Migrating to the Cloud?
Just copy data and applications into the cloud…
Straightforward approach
Inefficient, non-incremental for large environments
Open-source “bcp” uses encryption, compresses, and multi-threads
Archaic processes recommended by vendors
Start with backups to IaaS storage, then populate re-hosted applications by restoring from those backups
Cloud backups are easy, known technology
Bulk copy protocol or other archaic processes

36. Cost Estimates for Azure
Storage Capacity LRS
First 1 TB / Month $0.024 per GB
Next 49 TB (1 to 50 TB) / Month $ per GB
Next 450 TB (50 to 500 TB) / Month $ per GB
Next 500 TB (500 to 1,000 TB) / Month $ per GB
ZRS
$0.03 per GB
$ per GB
$0.029 per GB
$ per GB
Amazon Pricing : Sep 2016

37. Backups cloud service: from on-premises to the Cloud
Cloud Storage
Database server
Standard backup and recovery methods
Replication
Cloning, SSIS Packages to push data to Azure
Continual feed to keep up to date or refresh on regular basis, via archaic tools- bcp, log shipping or paid replication tools.

38. Migration Complete…Not so Fast…
What if you only want dev and test in the cloud.
Data is migrated, but this doesn’t count for ongoing data loads, application connectivity across the network.
To refresh will take considerable time to perform with traditional tools or cloning methods.
Rarely a consideration for the difference in cost structure for processing large amounts of data from on-premise to the cloud.
How many of you have moved dev and test to the cloud? How many moved cloud or moved it first??
If you moved it, would you consider keeping processing the same?
How can the cost structure impact you?

39. What’s the right configuration??
How Microsoft Sees a Migration to Azure
What’s the right configuration??
What all has to be moved? What issues are you going to run into?
Optimize first? Why?

40. Cloud Performance
And if you choose wrong or use more resources than expected, you can experience severe performance issues.
What resources are you really using? DBAs know, but do the developers and other stakeholders in the cloud migration project?

41. “Simple” Database Migration
Database must be MSSQL 2005 or higher, (easy)
Ensure that the database is compatible with Azure SQL DB, (correct any incompatible functions, etc.)
Must have identified all performance issues that will be impacted beforehand.
Ensure there is as little physical distance between the cloud data center and bacpac files to be used for migration.
Disable management jobs that will hinder migration processing.
Drop any objects or historical data that can impact migration time and can be performed post migration.
This is for Azure migrations- the requirements

42. If IIS and Using the Azure Cloud Migration Asst.
Performs assessment of IIS environment and creates report of what can be migrated.
Creates report to document what will require moving and any issues identified.
Creates any websites and associated databases in preparation for the migration.
If non-compatible products are in use, (which is in case of 90% of environments) then those must be addressed manually.
The tool is free to start, but will cost as project proceeds in duration.
Create this report- what can be migrated and what can’t?
90% of environments won’t fulfill the requirements and won’t migrate with the cloud migration asst.

43. From on-premises to the Cloud, then restore- Assumption
Cloud Storage
Database server
Traditional Migration
Database server
Once final tests are done- you are testing.
Perform final migration, final sync to prod and downtime to switch from on-prem to cloud.

44. Migrating from on-prem to IaaS with Virtualization
Much Improved Option-
Data virtualization
Easy, secure, revolutionary
Optimized for faster agile development and testing
Optimized for new costing structures in IaaS
With a software appliance, no concerns about hardware or software outside of virtualization lock-in.

45. Data Virtualization: on-premise Environment
Delphix Virtualization Engine
2 TB storage
Backups via SMB
SCCI/SSL
Source DB server
SQL Server
1 TB
Target DB server
SQL Server
No database storage

46. Data virtualization: from on-premises into the Cloud
Delphix Virtualization Engine
2 TB storage
Source DB server
SQL Server
2TB storage
Target DB server
SQL Server
No database storage

47. How Does Data Virtualization Enhance this?
Optimized for the cloud in the first place…not after! Different cost structures Much smaller storage footprint, much less data-transfer
By going to a single source, loading to a single source and maintaining a single source, a smaller footprint is attained.
Cost savings in the way of less storage required results in even bigger savings.
= $$$$

48. Different cost structures
Traditional copy data management techniques Developed without concerns about infrastructure chargeback This corresponds to higher cost. IaaS vendors monitor storage and data transfers Help meet SLAs, garner profits It’s not just the data that exists in the end, so transformations can equal big money for cloud vendors.
Different cloud manufacturers have different pricing structures- verify what you are being charged for and make sure those costs aren’t in contrast with your environment.
Many avoid RDS on Amazon- we don’t support it. For our Oracle customers, too limited.

49. Know Thy Enemy…
Tune SQL and Apps to perform efficiently as possible- natural life of database is growth, (in processes, resources, etc.) before migrating.
The less network latency, the better- network tracing to eliminate database blame is important.
Many of the same tools and data provides value- DMVs provide data internally to SQL Server.
Look at management tools such as Cloudmonix, (formerly AzureWatch) AppDynamics, Dynatrace, Zabbix or Logic Monitor.
Before you start, tune SQL instaead of after.
Use network tools like Nagios network analyzer or. Solarwinds Network Performance Monitor, (NPM)
Your performance data can assist you in identifying huge IO, CPU and remote resource work that should be minimized beforehand.

50. Secondary Considerations
For Non-Production Systems… Change the way you’ve always performed tasks. Performing common tasks the same way as previously might end up costing more. Secure Data All IaaS alternatives promote encryption for data in-flight and for data at-rest, but encryption may not be the right answer…
Data in flight can cost you and data processing that was normal on-prem, may need to be redesigned post cloud migration.
Inspect pricing small print carefully and know you’re final decision on *what* choice in cloud and type of service will determine.
Encryption is important for production.
SQL 2016 dynamic data masking isn’t production ready- three steps and I had ‘un-masked
data!

51. Confidential data
All IaaS solutions provide encryption in-flight and encryption at-rest But encryption doesn’t protect data as much as it needs to be . Europe already requires data masking, not just data encryption for any confindential data, (GDPR):

52. Confidential data
Encryption is reversible data obfuscation, which is very different from masking data.
Data masking is non-reversible.
It solves the issue at the data level.
Is authentication and authorization in non-production in compliance with security goals?
All organizations will soon need to review if critical data in non-production environments be accessible to developers, testers and users.
Or does it shift the problem toward authentication and authorization?

53. Why Masking is Part of the Answer
Masking personally-identifiable, (PII, HIPPA, PCI, etc.) information renders it useless from a security standpoint Resolves both the technical and personal responsibility issue. The data can be masked before it moves to non-production, removing unnecessary risk.

54. Confidential data
Production
Non-Production
Exposure

55. Data virtualization: transformation by masking
SQL Server Validated Sync Environment
Delphix Masking Engine
Delphix Virtualization Engine
2TB storage
SQL Server Target
No database storage

56. Data virtualization: Masking and then to the Cloud
Delphix Virtualization Engine
2 TB storage
SQL Server Target
Storage Only for Masked Data
Delphix Virtualization Engine
2TB storage
Delphix Masking Engine
SQL Server Validated Sync Environment
SQL Server Target
Storage Only for Masked Data

57. Confidential data Production Non-Production Exposure Encryption
Masking
Solution

58. On-Prem, Masked, then Replicated to IaaS
Delphix Virtualization Engine
4 TB storage
SQL Server Validated Sync Environment
SQL Server Target
Storage Only for Masked Data
Delphix Virtualization Engine
4 TB storage
SQL Server Target
Storage Only for Masked Data
Delphix Masking Engine

59. Virtualization Makes Copy Data Management Simple
Patching and Upgrading can be done with less effort and resources.
Cloud migrations are more successful when virtualized and planned accordingly.
Security with masking and encryption is best

60. Want to try it out, download the Delphix AWS Trial- Available for Oracle, MSSQL Soon!
Twittter: @DBAKevlar
Linked in:
Blog:

61. References and Tips
Delphix with SQL Server- the Basics Delphix Upgrade Workflow: Upgrading the Dsource after an Upgrade: Delphix in the Cloud

62. Thank you!