Esri UC2013. Technical Workshop. Technical Workshop 2013 Esri International User Conference July 8–12, 2013 | San Diego, California Data Interoperability An Introduction Bruce Harold – Esri Dale Lutz – Safe Software

Esri UC2013. Technical Workshop. Why Data Interoperability? Background

Esri UC2013. Technical Workshop. What is the Data Interoperability Extension? ESRI & SAFE Co-Development based on FME Format SupportFormat Support –Adds support for 200+ data formats –Plus Your Custom Formats –Direct use in ArcGIS Desktop Quick Data TranslationQuick Data Translation –Data Interoperability Tools Toolbox –Convert between data formats Spatial ETL (Extract, Transform, Load)Spatial ETL (Extract, Transform, Load) –Schema Mapping (may be Dynamic) –Transformation of Geometry –Logic Controls Data Flow – Per Feature

Esri UC2013. Technical Workshop. ArcGIS Data Interop Common Tasks Data Integration - Bring multiple formats together Data Migration - Process of moving data between systems Data Cleanup - Clean up errors in the data and streamline data preparation Data Distribution - Distribute data to different systems Data Merge - Merge changes from edited data Data Validation - Geometry and Attributes ETL

Esri UC2013. Technical Workshop. Data Interoperability Differentiators Not Just About Schema Manipulation Change Detection 3D PDF Writing XML Documents as Features or Attributes JSON Documents as Features or Attributes Protocol Support (HTTP, RSS…) Web and Compressed Inputs Parallel Processing Vertical Coordinate Systems BLOB Attributes

Esri UC2013. Technical Workshop. Basics: OGC Support What you get just by installing

Esri UC2013. Technical Workshop. OGC GML Simple Feature Format Support Support for GML-SF in ArcGIS Desktop GML – SF – simple features WFS – optionally non-cached for auto refresh WFS – cache timeout Note: Installation of the extension is required Note: Installation of the extension is required

Esri UC2013. Technical Workshop. Bruce Harold OGC Formats: GML & WFS Demo Data Interoperability - An Introduction

Esri UC2013. Technical Workshop. Demo Summary: GML & WFS Comes Free - But you have to install - Simple Features Only Better with the extension licensed - Non-simple parameters supported Direct Read of GML, consume WFS Important in many jurisdictions

Esri UC2013. Technical Workshop. Using Formats Direct Read of any format

Esri UC2013. Technical Workshop. Direct Read of Formats Formats recognized by Data Interop. - File-based, by extension, like.tab,.mid,.skp,.accdb - Visible in Catalog window browse tree - May be extended with Custom Formats Directly usable in ArcGIS - Save time converting data Consume external data - ArcPad AXF - Other application formats - Web feeds

Esri UC2013. Technical Workshop. Easy Format Selection Browse or Search

Esri UC2013. Technical Workshop. When to Browse and when to Connect Browse & Direct Read uses format defaults Many formats have optional parameters - Select subsets, set conditions, passwords, filters… Connections give you access to format parameters Connections let you set Coordinate Systems

Esri UC2013. Technical Workshop. Interoperability Connections Persist between application sessions Convenient to access in browse tree Retain your settings Off-disk data - Web Feeds - Google Spreadsheets Share with colleagues -.FDL file created - C:\Users\ \AppData\Roaming\Safe Software\Interoperability

Esri UC2013. Technical Workshop. Bruce Harold Direct Read Demo Data Interoperability - An Introduction

Esri UC2013. Technical Workshop. Demo Summary: Direct Read Direct Read Summary - Directly read, inspect and browse within ArcGIS - Works like native ESRI formats (no GDB behavior) - Access directly in Desktop Apps, Geoprocessing - No development or customization required

Esri UC2013. Technical Workshop. Quick Translation Quick Import & Quick Export GP Tools

Esri UC2013. Technical Workshop. Quick Data Import/Export Enabling Extension Provides ‘Quick’ Tools Quick Export - To any format Quick Import - To staging Geodatabase Not Configurable - Is this OK?

Esri UC2013. Technical Workshop. Automation and Quick Translations Use Quick Tools in ModelBuilder - Get any format into a temporary File GDB - Handle Multiple Formats In/Out

Esri UC2013. Technical Workshop. Bruce Harold Quick Tools Demo Data Interoperability - An Introduction

Esri UC2013. Technical Workshop. Demo Summary: Quick Import/Export “Quick” GP Tools - Useful in ModelBuilder - Also Scriptable in Python - Easy interaction with others - Departments - Contractors - 3 rd Party Software Users

Esri UC2013. Technical Workshop. Leveraging Formats Custom Formats

Esri UC2013. Technical Workshop. What if you need your own format? Save time by “canning” a custom format workflow Commonly handled text-based sources - CSV - XML Build format recognition workflow once Re-use the persisted workflow any time Lets you develop a file format for sharing

Esri UC2013. Technical Workshop. Bruce Harold Custom Formats Demo Data Interoperability - An Introduction

Esri UC2013. Technical Workshop. Demo Summary: Custom Formats Based on an existing format May be shared -.FDS file in profile directory for each format - C:\Users\ \My Documents\FME\Formats Editable later Get the custom Quake format by downloading the slides!

Esri UC2013. Technical Workshop. The Power of Transformation Spatial ETL

Esri UC2013. Technical Workshop. The Full Power of Spatial ETL Formats are only part of the story Often the format does not change * - But the schema does Transformation is key - Many categories of transformer - Feature by Feature or Dataset by Dataset - Geometry or Attribute transformations Many Inputs Many Outputs is OK *like shapefile to shapefile

Esri UC2013. Technical Workshop. Workbench Application Graphically Design Data Flows READ TRANSFORM WRITE

Esri UC2013. Technical Workshop. It’s a Pipeline Readers send features into the workspace - Geometry – X:Y:Z:M - Attributes – with a schema Transformers “do something’ - May operate “per feature” and pass it immediately - May work on sets or all features together - Some offer parallel processing Writers send features to the destination DATA

Esri UC2013. Technical Workshop. Some Transformers Are All You Need This example derives Adds, Deletes and NoChanges in one step - Esri Community Maps contributor data is processed this way

Esri UC2013. Technical Workshop. The Power of Transformation Using Spatial ETL 100’s of Transformers - Categorized Browse Search Popup in Workbench Navigate to in Workbench Don’t see what you need? - Ask for it!

Esri UC2013. Technical Workshop. Authoring Spatial ETL Tools Readers, Transformers, Visualizers, Writers Add Spatial ETL tools into a ToolBox - Right Click context menu Wizard to stub out your Workbench Refine the workspace Publish Parameters - Inputs are automatic Run the workspace - In Workbench - As a GP Tool

Esri UC2013. Technical Workshop. Bruce Harold ETL Demo Data Interoperability - An Introduction

Esri UC2013. Technical Workshop. Demo Summary: Spatial ETL Complements Core Functionality Analogous to ModelBuilder, except… - Feature by Feature granularity - Feature geometry manipulable - Non-Esri & Web data sources are OK

Esri UC2013. Technical Workshop. Wrapping Up More on use cases…

Esri UC2013. Technical Workshop. A Word on XML XML is a Document, not a Format XMLTemplater - Easily use an XML file to stub out XML - Populate attributes with XQuery expressions XQuery Extract, Update, Explode {fme:get-attribute("feederName")} {fme:get-attribute("feederNo")} {fme:get-attribute("eaLoc")}

Esri UC2013. Technical Workshop. Organizational Interaction Your ELA can include Data Interoperability as an option State Federal Regional / County ETL National GIS

Esri UC2013. Technical Workshop. ArcGIS Server and Data Interoperability 64bit Non-Native formats in Map Services Publish ETL as GP services Esri WFS services more flexible - Non-cached streaming option - Setting in Interop Connection

Esri UC2013. Technical Workshop. New in 10.2 See the Help too Based on FME 2013 SP1 + - Improved performance - Improved FME Workbench user experience New Formats – highlights only - Salesforce - OData - RIEGL Laser Scan - SpatiaLite* - ZIP Files New or Updated Transformers - Many condensed (e.g. AttributeCreator)

Esri UC2013. Technical Workshop. Resources: Learn More How do I try it? - Request software evaluation (60 days): - ns/datainteroperability/evaluate ns/datainteroperability/evaluate How do I learn it? - New course (another coming): - =catalog.webCourseDetail&courseid= =catalog.webCourseDetail&courseid=2571 Who do I talk to? - Visit the Analysis Island in the Esri Showcase or the SAFE Software booth (#1608) in the Expo.

Esri UC2013. Technical Workshop. Please fill out the session evaluation Wednesday Offering ID: 1343 Thursday Offering ID: 1430 Online – Paper – pick up and put in drop box Thank you…

Esri UC2013. Technical Workshop.Data Interoperability - An Introduction