GO BUILD, YOUNG DEV! AUTOMATING AX BUILDS WITH TFS BRIAN HOF, MARSHALLTOWN

INTRODUCTION “You see, in this world there’s two kinds of people, my friend: Those with loaded guns, and those who dig. You dig.” - the man with no name

Who is MARSHALLTOWN?  Manufacturer of quality construction tools  Size: Relatively small  IT Investment: Relatively large  Information Engineering 6 full-time in-house developers Maintaining and enhancing AX Building add-on tools using primarily.NET  AX Implementation:  Currently running AX 2009  Migrating to AX 2012 R3 (CU8)

Who am I?  Bachelor of Science in Computer Engineering Iowa State University, 2003  Joined MARSHALLTOWN in 2014  AX 2009 / 2012 R3 Still learning!  C# / ASP.NET (MVC) Still re-(un-)learning web development!  Former gigs:  Emerson Process Management, Fisher Controls C#, C++, VB VSS, TFS  IBM Java

Automated builds: why?  Consistency  Limit build environment Same steps, same hardware, same software every time  Reliability  Limit time between builds Frequent build enables frequent verification  Collaboration  Limit changes between builds Why did it break? Who can fix it?  Result: Agility  I build (and verify), therefore I release

SETTING …draw!

Ecosystem: AX TFS  Individual, isolated developer systems:  AOS instance  AX client  Visual Studio  SQL server (optional)  Shared version control (Team Foundation Server):  Model code stored as.XPO,.ALD files  AX Builds in this environment are complicated  More reason to automate!

Ecosystem: AX TFS  TFS integrates:  Version control  Team Foundation Build Build automation tools & mechanisms  Application Lifecycle Management (ALM) Out-of-box process templates: Scrum, Agile, CMMI Work item tracking Features, bugs, etc Collaborative Team Web Access portal Manage development cycles & work items Review code Monitor builds

Ecosystem: AX TFS

Automated AX builds: how?  Roll your own?  PowerShell scripts  Referenced in whitepaper: Change Management and TFS Integration for Multi- Developer Projects (white paper) Change Management and TFS Integration for Multi- Developer Projects (white paper)  Team Foundation Build  Pioneered by Joris de Gruyter (now at Microsoft)  Caveat: TFS 2015 has been released! Support introduced in AX 2012 R2/R3 CU9 Team Foundation Build has been overhauled Legacy XAML builds still supported

Team Foundation Build  Built on Windows Workflow Foundation (WF)  A workflow resembles a flowchart Accepts input parameters Declares (scoped) variables Each step is an Activity (System.Activities.CodeActivity) Accept input arguments (System.Activities.InArgument ) Perform logic (Execute method) Return output arguments (System.Activities.OutArgument ) Visual Studio WYSIWYG editor Stored as.XAML  Activity toolbox: WF out-of-box: Control flow: Sequence, Parallel, If, While Primitives: Assign, InvokeMethod Error Handling: TryCatch, Throw TFS out-of-box: Build & Source Control operations Custom: Code your own with any.NET language CodeCrib.AX.TFS – Joris de Gruyter’s AX activity library

Team Foundation Build  Concepts:  Build process template WF workflow (a.k.a. XAML build) implementing a build process  Build definition Configures a specific build within a TFS team project, specifying: Triggers Build workspace Build process template Input parameters Drop location Retention policy  Build controller Initiates and monitors builds WF execution begins on the controller  Build agent Executes builds Software/licensing needed to build lives here Controller selects agent using tags WF execution resumes & completes on an agent See Run on agent activity block

Team Foundation Build  Example:

THE GOOD What did we end up with?

Disclaimers  We acknowledge:  No such thing as a completed AX implementation Continuous, ongoing, collaborative project management is complex Overhead of source control, automation is justified  We favor:  K.I.S.S. methodology Our customizations are Single layer Single model Deviate at your own risk TFS may have issues with any AOT elements customized across multiple models  Test and Development environments with real business data Refreshed often from production Dev/Test environments – less frequently Sandbox environment – nightly Enables us to quickly respond to issues

Our Process  Build phase  Generate model file (.axmodel) Uncompiled AOT  Generate model store file (.axmodelstore) Compiled AOT  Deployment phase  Restore database (optional)  Import model store file  Execute post-deployment tasks

Build Phase  Generate model file (.axmodel)  Generate combined XPO from all XPOs in TFS  Using Temporary DB (more on this later) Clean file system Delete XppIL, cache, VSAssemblies, etc Repeated throughout build to avoid various problems Uninstall all models Synchronize data dictionary Import labels (ALDs from TFS) Clean file system Import combined XPO Export model file

Build Phase  Generate model store file (.axmodel)  Using Primary DB (more on this later) Install the model file Clean file system Deploy.NET references Import labels (ALDs from TFS) Import any VS projects from TFS (currently none) Compile Using either AX client or AxBuild.exe (more on this later) Full CIL generation Synchronize data dictionary Export primary model file for archival Export model store file Publish artifacts to network drop location

Deployment Phase  Restore database (optional)  Restore SQL backup of business and model database (more on this later)  Execute post-restore SQL script  Clean file system  Import model store file  Initialize temporary schema in model database  Import model store file to temporary schema  Apply temporary schema  Drop temporary schema  Clean file system  Synchronize data dictionary

Deployment Phase  Execute post-deployment tasks  Often business data must be modified to work with new code, or other tasks must be completed  Build invokes our custom AOT class: MT_PostDeployTasks Executes its own methods alphabetically, once-and-only-once Task completion status is logged to a table

Deployment Phase  Notes:  Deployment phase can be run independently Deploys most recently built model store file Enables progressive deployment of a build to multiple environments, pending verification  Multiple targets may be specified Steps repeated on all targets

Demonstration  Show & tell:  Build definition  Build process template  Activity library  Build results in VS and Team Web Access

THE BAD (& THE UGLY) What did we learn (the hard way)?

Business vs. Model Data  AX 2012 “Database”  Two separate SQL databases: Business Data Model Data  Element IDs Tie model database to business database Every AOT element has one Class, method, database table, field, etc Generated when first added to AOT Manual creation XPO import Model file import  Data from either cannot be haphazardly interchanged

Business vs. Model Data  What does Microsoft tell us?  Models, Layers, and the Model Store [AX 2012] Models, Layers, and the Model Store [AX 2012] XPO filesModel filesModel store files Installation toolMorphXAXUtil.exe or Windows PowerShell cmdlets The files can be uninstalled.NoYesNo The files can be signed.NoYesNo Microsoft Dynamics AX element IDs are preserved. Yes, if the elements already exist in the target model store Yes, all element IDs of the source model store are preserved. Compilation is required after installation. Yes No CIL generation is required after installation. Yes No ScenarioRecommended installation method Distributing a solution to customersModel files Deploying a solution in a development or test environmentModel files or XPO files Deploying a solution to a production environmentModel store files

Business vs. Model Data: Scenario  Alice and Xander begin with the same data, V1  Alice modifies AOT, synchronizes data dictionary, yielding Va  Deletes element: D1  Adds elements [IDs]: A1 [1], A2 [2]  Xander modifies AOT, synchronizes data dictionary, yielding Vx  Adds elements [IDs]: X1 [1], X2 [2], X3 [3]  At this point they can no longer swap business databases!  OK…but how can they exchange model data?

Business vs. Model Data: Scenario  Exchange.XPOs?  Alice has: A1 [1], A2 [2], X1 [4], X2 [5], X3 [3]  Xander has: D1 A1 [4], A2 [5], X1 [1], X2 [2], X3 [3]

Business vs. Model Data: Scenario  Exchange model files? (.axmodel)  Alice has: X1 [?], X2 [?], X3 [?]  Xander has: A1 [?], A2 [?]

Business vs. Model Data: Scenario  Synchronize via TFS?  Alice has: A1 [1], A2 [2], X1 [4], X2 [5], X3 [3]  Xander has: A1 [4], A2 [5], X1 [1], X2 [2], X3 [3]

Business vs. Model Data: Scenario  Synchronize via TFS… then Exchange model store file? (.axmodelstore)  Alice and Xander synchronize with TFS  Alice creates model store file, Xander imports They now have the same elements and IDs Xander’s business data no longer matches his model data 

Business vs. Model Data  What’s the answer?  Developers synchronize to TFS Their element IDs won’t match Doesn’t matter, if they don’t need to share business data  Build and Deployment environments: Must share common business data origin Must deploy model data exclusively from build system Must design build process to maintain element ID integrity between builds Build against Temporary DB and Primary DB (next slide)

Temporary vs. Primary DB  Maintaining element ID integrity between builds  Temporary DB AOT is reset for each build Uninstall all models before importing XPOs Ensures deleted elements are removed Used solely to generate a model file Element IDs are unimportant  Primary DB AOT must never be reset Code changes must be accumulative Model file from Temporary DB is installed over existing AOT Deleted elements are removed Pre-existing elements & their IDs are preserved New elements are added with new auto-generated IDs Updated AOT is compiled Model store file exported & deployed to matching targets

Client Compile vs. AXBuild.exe  AXBuild.exe  Multi-threaded compile – shaves off hours  Difficult to detect success Log: AxCompileAll.html Part actual errors Part false alarms Complicated retry loop Errors in log? Import into AX client Recompile problem objects Repeat N times Refer to: Provides code to automate the retry process  Unreliable Others have reported apparently successful builds which do not actually work  Compile in AX client  Single-threaded – slow  Easy to detect success  Reliable

Performance Notes VMWare Hyper-V Process Total7:247:477:455:025:30HH:MM BUILD Synch 11:211:251:351:051:13 Import XPO1:161:191:171:061:14 Compile2:372:462:391:181:20 CIL0:270:250:270:18 Synch 20:240:210:230:080:10 Create model store0:13 0:070:11 DEPLOY Import model store0:09 0:14 Synch 30:220:230:190:210:18  vSphere vs. Hyper-V w/SSHD

CONCLUSION

 Environment considerations and Microsoft-recommended practices:  Change Management and TFS Integration for Multi-Developer Projects (white paper) Change Management and TFS Integration for Multi-Developer Projects (white paper) Build and deploy scripts for Microsoft Dynamics AX 2012 (PowerShell)  Deploying customizations across Microsoft Dynamics AX 2012 environments (white paper) Deploying customizations across Microsoft Dynamics AX 2012 environments (white paper)  Building with TFS (materials by Joris de Gruyter):  How We Manage Development - Automated Builds How We Manage Development - Automated Builds  Easy Automated Builds: Part 1 Part 2 Part 3 Sample build templates CodeCrib.AX.TFS (custom activity library) Class documentation CodePlex open source repository References

DISCUSSION “I want you to stand up there and put your head in that noose.” - the man with no name