1. John Lloyd, Edgewater Fullscope
What, Why, and how of dixf
John Lloyd, Edgewater Fullscope

2. Data import export framework - overview

3. Data import export framework
Data import/export process
Source to Staging
Staging to Target
Remove Staging data
The Microsoft Dynamics AX 2012 Data Import/Export Framework is an extension that helps you export data and import it into Microsoft Dynamics AX. Data is imported from a source to a DIEF staging table in AX. Then it is copied to the target table(s). After migration is completed, you can delete the staging data. For AX 2012 R2, DIEF is available in CU7. For AX 2012 R3, DIEF is available in the installer under components. You install DIEF on the client, AOS, and SSIS computers.

4. Data import export framework
Data export/import architecture
Source entity
Staging entity
Target entity
Source to staging map
Staging to target map
This is a diagram showing the architecture of the Data Import/Export Framework. There is a source to staging map that converts the source data into the staging data and a staging to target map that converts the staging data to the standard AX target data.

5. DIEF – Setup Setup Common Periodic
Processing group
[Form\DMFDefinitionGroup]
Create a custom entity for data import/export
[Form\DMFCustomWizard]
Compare entity data between companies
[Form\DMFDefinitionGroup; DMFCompare::CompareEntity]
Copy entity data between companies
[Form\DMFDefinitionGroup ; DMFCompare::CopyEntity]
Periodic
Staging cleanup
[Classes\DMFStagingCleanup]
Setup
Target entities
[Form\DMFEntity]
Source data formats
[Form\DMFDataSource]
Data import/export framework parameters
[Form\DMFParameters]
Setup of roles for processing group
[Form\DMFDefinationGroupAccess]
The DIEF module in Dynamics AX 2012 has Common, Periodic, and Setup submenu. The Setup submenu has the Parameters, Source data formats, and Target entities. The Common submenu has the Processing groups. There are other menu items for comparing, copying, and purging the DIEF data.

6. DIEF – Setup - parameters
The AX 2012 DIEF has a working directory that must be set up with read write permissions for the AOS service user. You specify this working directory in the DIEF Parameters and you can test the service account access.

7. DIEF – Setup – data source formats
Source types
File AX
ODBC
File format
Delimited
Fixed width
XML
Excel
Row/column delimiter
{CR}{LF}
{CR}
{LF}
Semicolon {;}
Colon {:}
Comma {,}
Tab {t}
Vertical bar {|}
You define the format of your source data in Data source formats . You define the source type (file,ODBC,AX). If the format is file, you define the file format (delimited, fixed width, xml, Excel), row, and column identifiers.

8. DIEF – Setup – data source formats
Dimensions and name sequence
Dimension format
Name sequence
If you are importing financial dimensions, you need to setup a sequence for the dimension string. You can also setup the name sequence.

9. DIEF – Setup – target entities
Out of the box target entities
Target mappings
Entity structure
Target fields
AX 2012 has a standard set of target entities that can be used right away. There is also an entity wizard that you can use to create entities for tables that do not have a standard entity or tables that are custom.

10. DIEF – Setup – target entity mapping
Mapping visualization
Functions
Function parameters and return values
The staging data is copied and converted to the target data via the Target mapping. There is a visualization view where you can see the mappings directly into the target table fields or into and from generate methods. Mandatory fields have a red asterisk.

11. DIEF – Setup – target entity mapping
Mapping detail
Mapping type ( = or () )
Function
Sequence
Return parameter sequence
In the mapping details view, you can see the mapping type ( =,() ), the generate functions, the sequence of the functions, and the sequence of the return parameters.

12. DIEF – Setup – target entity Structure
Parent data source
Level
AssetTable
AssetLocation
AssetBook
AssetBookTable
Target entities may be comprised of multiple tables and have a structure. You can see the target fields of each table in the structure from the Entity structure form.

13. DIEF – Setup – target entity fields
Table name
Table field
You can see the target fields for the entire target from the Target entity form.

14. DIEF – Common Setup Common Periodic
Processing group
[Form\DMFDefinitionGroup]
Create a custom entity for data import/export
[Form\DMFCustomWizard]
Compare entity data between companies
[Form\DMFDefinitionGroup; DMFCompare::CompareEntity]
Copy entity data between companies
[Form\DMFDefinitionGroup ; DMFCompare::CopyEntity]
Periodic
Staging cleanup
[Classes\DMFStagingCleanup]
Setup
Target entities
[Form\DMFEntity]
Source data formats
[Form\DMFDataSource]
Data import/export framework parameters
[Form\DMFParameters]
Setup of roles for processing group
[Form\DMFDefinationGroupAccess]
The DIEF module in Dynamics AX 2012 has Common, Periodic, and Setup submenu. The Setup submenu has the Parameters, Source data formats, and Target entities. The Common submenu has the Processing groups. There are other menu items for comparing, copying, and purging the DIEF data.

15. DIEF – Common – Processing groups
Combine target entities
Source to staging mapping
Execute get staging data; copy data to Target; and view execution history, staging data, and error logging
The DIEF Processing groups form groups different, related entities to be processed together. For instance, you have the customer then the customer address entity. This is where you setup you source entity, target entity, data source format, and source to staging map. You can copy the source data into the staging table and from the staging table to the target table(s). You can view the staging data, the execution history for each job (execution), it’s statuses, record counts, start/end times, and error messages.

16. DIEF – entities for Processing groups
Target entity
Source data format
Sequence
Run business logic in insert or update method
Run business validations
Sample file path (SQL for ODBC)
View staging preview
Generate source file
Generate source mapping
Modify source mapping
Validate
View target mapping
Preview source file
Entity attributes
Preview error details
From the Entities button, you can view the Entities for processing group form. This is where you add entities to your processing group, specify the source data format used for that entity, order the execution of the entities, and specify whether you want to execute the code in the insert/update and validate methods on the target table. You also specify the file or SQL (ODBC). You can generate your source file layout, generate/edit your source to staging mapping, and validate/preview your mapping.

17. DIEF – Map source to staging
Mapping visualization
Drag and drop
Mandatory fields with a red asterisk
The source to staging mapping is similar to the staging to target mapping except that there are no functions.

18. DIEF – Map source to staging
Mapping details
Grid view
Auto-generate
Query criteria
Auto default
Default value
Conversion
In the Mapping details view, you can add/delete mapping lines, set staging fields to be auto-generated like customer account numbers or sales order numbers, query criteria if you want to auto-assign a number when a field values changes (ex. Auto-assign a sales order number when the source sales order number in the file changes), set default values (ex. Currency=USD), and set conversions. Conversions map values in the source to values that are in AX (ex. Vendor groups).

19. DIEF – preview source file
Test source to staging mapping
Find errors
Uses SSIS package
Once you have mapped your source data, you can test it via the Preview source file button. It is fast and you can visually check your data before you import it. You can also find errors in your mapping or the source data.

20. DIEF – entity attributes
Sequence of fields in the file
Field name – column header
Mandatory flag
Field type
Field size
Up/down buttons
The Entity attributes is the sequence of the fields in the file, their type and size, and if they are mandatory or not.

21. DIEF – execution history
Staging and target status
Staging and target details
Number of records created/updated
Start and end times
View staging data
Error log
Once you have import data into the DIEF staging table, the Execution history button will be active and you can view information about import job. You can see the run statuses, run times, record counts, and error logs. You can also view the staging data. If you have errors, you can correct the AX data and rerun the staging records that have errored.

22. DIEF – view staging data
Select records to copy to target
Export to Excel
Target button
You can select records in the staging data and only process those records to target. If the data was successfully copied to the target, you can view the target data (the menu item is specified in the wizard for entities you create.)

23. DIEF – view Error log View error log Count of records inserted/updated
Error messages for data issues
The Error log will display the Staging error log form. It has a grid of jobs by stage and you can view the Infolog that was generated for that stage.

24. Data import export framework – creating templates and demo files

25. DIEF – adding A processing group
Processing groups
Create a processing group
Add entities
Generate source file
Generate sample file
Save sample file
Once you have added your entities to your processing group, you can generate a source file. In the Generate source file wizard, there is a Generate sample file button that can be used to generate a sample file with column headings. You can use this file as a template for your data.

26. DIEF – demo files Demo files
C:\Program Files (x86)\Microsoft Dynamics AX\60\Client\DemoFiles\
Delimited
XML
Generate source mapping
In the AX client folder, there is a Demo file folder that has Delimited and XML sample files. You can use these sample files to test your DIEF setup.

27. Data import export framework – creating entities

28. DIEF – Entity wizard Custom entity wizard Select a table
Select code generation parameters
Table
Query
Class
Menu item
Specify fields in the target table
Creates a private project with generate/modified components
BaseEnum\DMFEntityType
DataDictionary\Tables\DMFEntity
In the DIEF Common sub-menu, there is a Create custom entity for data import/export menu item. This starts the Custom entity wizard. You begin by selecting the table for which you would like to create an entity. The next form will automatically be populated with a table, query, and class for the entity. You can select a menu item that will bring up the target table data. In the next form you can specify which fields you would like to populate in the target table. When the wizard finishes, it will open a private project with the generated objects plus the DMFEntityType enum and DMFEntity table.

29. DIEF – entity Updated objects BaseEnum\DMFEntityType
DataDictionary\Tables\DMFEntity.defaultEntityEx()
The entity table is added to the enumerate DMFEntityType and the DMFEntity table defaultEntityEx() is modified to default the entity query, table, and class. This is used by the Target entity form.

30. DIEF – entity target Target entity
BaseEnum\DMFEntityType in the Entity drop down
DataDictionary\Tables\DMFEntity.defaultEntityEx() auto-populates form
The target entity can be created by selecting the new DMFEntityType enumerate in the drop down and entering a name in the Entity field.

31. DIEF – generate methods
Staging to target map
Method sequence
Return parameter sequence
Target table reference
Generate method
The entity wizard creates the generate methods with a TODO in them. You must manually add the generate code to complete them. There are a number of existing generate methods and helper classes that can be leveraged to create your generate method. In the staging to target map, you can see the generate methods mapping. The mapping type is ‘()’ and the function name is specified. The staging fields that are used as parameters by the generate method and the target fields that are populated by them. In the generate method code, you see the staging fields being used and the return of the value to be populated in the target table. You also see a sequence and a return parameter sequence column with zero or integer values.

32. DIEF – generate methods
The staging table will have a table group with the generate method name + ‘”_” + Sequence (from the target entity details). The fields in the fields group will have the order that is in the return parameter sequence. The getReturnFields method in the entity class maps the output from the generate method to the target field.

33. DIEF – generate methods
Generate method examples
Generate*Link
GenerateDefaultDimension
GenerateInventDim
GenerateParty
GeneratePostalAddress
GenerateElectronicLocation
Generate*Worker
GenerateLedgerDimension
GenerateDeliveryAddress
GenerateVendExternalItem
DMF helper classes
DMFDimensionHelper
DMFLogisticsAddressHelper
DMFParty
There are a number of generate methods that are frequently used. The Generate*Link (ex. GenerateJournalLink, GenerateSalesTableLink) generate the header for entities that are lines for a sales order, inventory journal, general ledger journal, etc. The GenerateDefaultDimension method generates the default financial dimensions, the GenerateLedgerDimension method generates the ledger dimension and the GenerateInventDim method generates the InventDimId for the entity. There is a DMFDimensionHelper class to help generate default and ledger dimensions. There are other standard AX classes and views like the DirPartyPostalAddressView, which helps to create postal addresses.

34. DIEF – entity methods Generate method examples
Derived from DMFBaseEntity
insertUpdate
postGetStagingData
addStagingLink
postTargetProcess
Where the methods are called from
Classes\DMFGenerateSSISPackage.generateStagingData() [341]
Classes\DMFEntityWriter.write() [572]
Generate methods
Classes\DMFEntityWriter.write() [500]
Classes\DMFEntityWriter.write() [202]
DMFEntity classes are derived from the DMFBaseEntity class and you can override some of the methods so that you can filter or convert data using more complex logic. The insertUpdate method is where the target data is created or updated in AX (The DMFBaseEntity insertUpdate method does the actual insert or doInsert depending on if you have Run business logic in insert or update method field checked or not). You can overwrite this method if want to convert or filter the data before it is created or updated. The postGetStagingData method can be overwritten if you want to convert or filter data after the SSIS package has been executed and the staging table populated. The postGetStagingData method is executed during the Get staging data process and the rest are executed in the Copy data to target process.
As we can see, the DIEF Framework is a toolset that has a set of the most commonly used entities, tools that you can use to extend the framework and create new entities, and methods that can be overridden to provide access to the data at different parts of the process. However, there are some limitations (ex. minimal transformation that can be done when copy from source to staging) that we can overcome.

35. Data import export framework – conversion entity

36. DIEF – conversion entity
Standard DIEF Conversion
The conversion data set may be very large or frequently changing
Difficult to maintain in the standard DIEF Source to staging map conversion table/form
The conversion data may be used by multiple entities
Conversion Entity
The staging data is never copied to a target
Manually create the DIEF table and class
Conversion entity table fields
Definition group
ExecutionId
IsSelected
TransferStatus
ConversionFromField
ConversionToField
In DIEF to convert a value in a field to another value that is in AX, you add those pairs of values to the Define conversion values form. If the dataset is very large, populating and updating this data my be difficult and time-consuming. If this data is used in multiple places, the data must keyed into different processing group entities each time it is used. If this data could be imported like the migrated data into the staging table like the other data, it could save time, eliminate mistakes and redundancy. For example if you migrate items but there are the same items already in your instance of AX, you may want to map those to items to your items and not migrate them. Later you migrate purchase orders with those items and again you want to map the same items to the same existing items in AX. Using standard AX DIEF, you would add those item pairs to the conversion values in the source data mapping for the items and the purchase orders. You may also want to initiate inventory balances with an inventory journal. Then you would need to create the same conversion on that entity’s source map. The solution is to create a conversion entity so that this data can be maintained in one place and used by any of the other entities.

37. DIEF – conversion entity class
Extends DMFEntityBase
The entity variable is set to the entity table
The target entity is created to define the staging table and class
The conversion DMF class extends DMFEntityBase and the entity variable is set to the conversion staging table. Although the staging to target process is not executed, the target entity must be created to so that DIEF can reference the entity’s DMF class in the Get staging data process. You may also add the value to the enumerate DMFEntityType and the DMFEntity table method defaultEntityEx() similar to the way the wizard does.

38. DIEF – postgetstagingdata
The postGetStagingData method
It is called after the execution of the SSIS package
The DMFDefinitionGroupExecution table is the parameter.
The staging data for the current execution can be iterated through and complex conversions executed (ex. set a field based on the contents of one or more other fields)
In the main entity postGetStagingData, the staging table has been populated via an SSIS package with the source data. The definition group execution table record is the input parameter and the execution id can be used to iterate thru the set of staging table records for the current execution. More complex conversion, conversions using conversion entities, and data conversion/filtering can be done in this method. In this example, projects are being migrated and only level 2 sub-projects are being created in AX with an exiting project as the parent project. The mapping is the parent project of the migrated subproject and the existing AX parent project (also the sales manager is being set). Also in this example, the data had projects from levels 1, 2, and 3 but only the level 2 project was created in AX. The costs for the project were rolled up from level 3 to level 2 and the parent was not used. So, these staging records were deleted. You may want to “mark” the records for deletion and filter on “mark” in the insertUpdate method.

39. Data import export framework – processes

40. DIEF – source to staging
Get staging data
\Classes\DMFStagingWriter\execute
\Classes\DMFGenerateSSISPackage\generateStagingData
The two main processes of DIEF are the executed from the Processing group form. The Get staging data button executes the copy from source to staging process. The DMFStaging writer class execute method is executed and calls class DMFGenerateSSISPackage method generateStagingData which executes the SSIS package and imports the data into the DIEF staging table. This is executed by the DMF SSIS helper service that is a Windows service on the SSIS tier. The executable for the service is in C:\Program Files\Microsoft Dynamics AX\60\DataImportExportFramework. You cannot modify this code.

41. DIEF – source to staging
SSIS Package
.DSTX file
SQL Server Business Intelligence Development Studio
You can view the SSIS package file, .DSTX, in the working directory of DIEF using SQL Server Business Intelligence Development Studio. You can see the source file layout and the Copy component expressions.

42. DIEF – staging to target
Copy data to Target
\Classes\DMFEntityWriter\write
setTargetBuffer method
Generate methods
insertUpdate method
The Processing group form Copy data to Target button executes the DMFEntityWriter class write method which executes most of the DMF entity class methods like the generate methods and the insertUpdate method. The method DMFEntityWriter.write is the main method and does the insert/update of the target table(s) and calls the entity class methods.

43. DIEF – staging to target
Copy data to Target
\Classes\DMFEntityWriter\write
Line 152 main loop
Line 215 target hierarchy loop
Line 500 generate method call
Line 572 insertUpdate method call
The Processing group form Copy data to Target button executes the DMFEntityWriter class write method which executes most of the DMF entity class methods like the generate methods and the insertUpdate method. The method DMFEntityWriter.write is the main method and does the insert/update of the target table(s) and calls the entity class methods.

44. Final reminders
CPE Credit Code: 53C2 Complete Surveys

45. Speaker contact info
John Lloyd Edgewater Fullscope