Sep , 2006 v FME Worldwide User Conference – Vancouver FME and ArcSDE / Geodatabase Mark Stoakes, Safe Software, Vancouver, BC

FME & ArcSDE /Geodatabase 2 Agenda  Supported ESRI databases  Differences between ArcSDE and Geodatabase  Writing data into ArcSDE and Enterprise Geodatabases  Reading data from ArcSDE and Enterprise Geodatabases  Raster Support  ArcSDEQuerier  What’s coming in FME 2007

FME & ArcSDE /Geodatabase 3 Supported ESRI Databases  ArcSDE A spatial data gateway providing spatial database functionality on top of relational databases (e.g. Oracle, SQL Server).  Multi-User Geodatabase (Enterprise) Enterprise Geodatabase is a layer of functionality and structure built on top of ArcSDE.  Individual Geodatabases  Personal Geodatabase  Personal geodatabase stored in a Microsoft Access database.  File-Based Geodatabase  ArcGIS 9.2 geodatabase stored as a set of files in a directory tree.

FME & ArcSDE /Geodatabase 4 Enterprise Geodatabase and ArcSDE Relational Database ArcSDE Enterprise Geodatabase C API COM API Server Side Client Side File System Client Side

FME & ArcSDE /Geodatabase 5 Personal Geodatabase Microsoft Access Personal Geodatabase COM API Client Side File System Client Side

FME & ArcSDE /Geodatabase 6 Enterprise Geodatabase and ArcSDE File Based Geodatabase COM API Client Side File System Client Side

FME & ArcSDE /Geodatabase 7 Definitions  Layer Spatial column within an ArcSDE table  Feature Class Geodatabase table with geometry (layer)  Object Class or Table Database table with no geometry  Feature Dataset A grouping of spatially related feature classes with the same spatial reference (domain and coordinate system)  Raster Map Geodatabase table with a single raster mosaic  Raster Catalog Geodatabase table with separate rasters in each record

FME & ArcSDE /Geodatabase 8 ArcSDE vs. Geodatabase FME ArcSDE reader / writerFME Geodatabase reader / writer Improved performance using SDE C-API Less performance using Geodb ArcObjects API Loads directly into ArcSDE layers – which can be registered with Geodb Loads directly into feature classes and also feature datasets No support for Geodb specific entities Supports annotation, domains, subtypes, networks (read only), paths & arcs, dimensions and feature-linked annotations Supports versioning

FME & ArcSDE /Geodatabase 9 ArcSDE vs Geodatabase FME ArcSDE reader / writerFME Geodatabase reader / writer Support for low and high precision geometry Only default precision created. Can read and write both low and high precision existing tables C API doesn’t provide access to arcs in SDE Supports the FME rich geometry model – i.e. paths No license issuesRequires ArcGIS license and ArcGIS install on the client machine Raster supportRasters not yet supported

FME & ArcSDE /Geodatabase 10 Why File-Based Geodatabase? Access GeodatabaseFile Based Geodatabase Only supported on WindowsSupported on many different platforms Limited to 2 Gigabytes in sizeCan easily be terabytes in size Support for limited set of geodatabase primitives Support for larger set of geodatabase primitives Available in all versions of ArcGISAvailable in ArcGIS 9.2 and later

FME & ArcSDE /Geodatabase 11 Spatial Index Organization  Consists of grids  Grid 1 and 2 optional  If defined, Grid 1 contains features which require > 4 Grid 0 cells  A feature is only stored in one Grid

FME & ArcSDE /Geodatabase 12 Writing ArcSDE / Geodb  Spatial Domain  The most common reason for data load errors  setting:data resolution / precision data extents (origin) coordinate system  Use ArcCatalog to determine the best domain parameters

FME & ArcSDE /Geodatabase 13 Writing ArcSDE / Geodb Step 1  Determine Spatial Domain for each table  Coordinate Reference System (e.g. LL84)  Smallest coordinate in x,y of entire area to be loaded  Largest coordinate in x,y of entire area to be loaded Step 2  Calculate the scale factor  Larger of (xExtent) / 2 31 or (yExtent) / Scale is the reciprocal of this number Step 3  Specify -1 (ArcSDE) as the grid size so that you can load the data in quickly

FME & ArcSDE /Geodatabase 14 Writing ArcSDE / Geodb Within ArcCatalog:  Double click the feature class  Select the 'Fields' tab on the Feature Class Properties dialog  Select the Geometry field  In the Field Properties table at the bottom of the dialog, press the ellipsis button (corresponding to the Spatial Reference entry)  Select the 'X/Y Domain' tab on the resulting Spatial Reference Properties dialog Determining the Spatial Domain

FME & ArcSDE /Geodatabase 15 Writing ArcSDE / Geodb  Versions  Supported by both ArcSDE and Geodatabase  Writer can perform RECONCILE and POST operations  Feature Datasets (Geodb only)  All feature classes in a feature dataset must have the same spatial domain  FME writer spatial domain is used when writing to a feature dataset  Subtypes & Domains (Geodb only)  Load subtype codes or descriptions use geodb_subtype_name to load subtype descriptions  Create coded and range domains

FME & ArcSDE /Geodatabase 16 Writing ArcSDE / Geodb Spatial Domain Feature Dataset Forces the writer spatial domain parameters to be used Feature Dataset & Spatial Domain

FME & ArcSDE /Geodatabase 17 Writing ArcSDE / Geodb  Domain/Subtype creation in the feature type definition Parameter TAB  Template files will simplify creation of subtypes & domains Subtypes & Domains (Geodb only)

FME & ArcSDE /Geodatabase 18 Writing ArcSDE / Geodb  Annotation (Geodb only)  Feature-linked annotation  Annotation scale & units  Relationship Classes (Geodb only)  FME currently cannot create a relationship class – but can write to them  Building relationships requires workspace configuration to determine the OBJECTID of the child tables  Data Validation  Catches features rejected by ArcSDE / Geodb and the underlying database  ArcSDE has rejected feature pipeline – cleans & resubmits features  Geodb can send failed features to FFS file

FME & ArcSDE /Geodatabase 19 Writing ArcSDE / Geodb.  Every geometry that is loaded is validated  Slower loading and some geometry is rejected  Can perform queries with confidence Good Geometries Bad Geometries! self intersecting Geometry Consistency

FME & ArcSDE /Geodatabase 20 Writing ArcSDE / Geodb  Things are not always so simple  Coordinates in low precision are stored in database as 31 bit integers  Self-intersections can be introduced by the ArcSDE itself!  High precision  Headaches go away or are reduced! Geometry Consistency

FME & ArcSDE /Geodatabase 21 Writing ArcSDE / Geodb ArcSDE 1. FME tries to write feature 2. Invalid feature sent to ffs or rejected feature pipeline 3. Pipeline is applied to try and fix feature 4. FME tries to write feature one last time ArcSDE Geodb – writes failed features to FFS file How does FME Help?

FME & ArcSDE /Geodatabase 22 Writing ArcSDE / Geodb  Continue on bad data  ArcSDE writer ignores the bad data and continues the translation  Roll back then continue  Rolls back to the previous commit and then continues the translation  Ignore failed features  Geodb writer ignores the bad data and continues the translation more validation options…

FME & ArcSDE /Geodatabase 23 Rejected Feature Pipeline Transformers  SelfIntersector  Check to see if there are any self intersections  CoordinateRounder  Round the feature to coordinates to approximate the scale factor  GeometryFilter  Check to see if there are geometry types not valid for table  MRFCleaner  New extra cost transformer that can be used to automatically clean data

FME & ArcSDE /Geodatabase 24 Rejected Feature Pipeline Transformers  Others  Any others people have used? Any transformer can be used in the rejected feature pipeline.

FME & ArcSDE /Geodatabase 25 Writing ArcSDE / Geodb  Paths  Geometries containing lines & arcs as a single entity  Now supported by FME rich geometry  Measures  Use the geodb_measures or sde30_measures attributes sde30_measures = 20,35,65,105,145  Use the MeasureGenerator transformer  Metadata  FME does not read / write Metadata, but…  FME can process metadata XML once it is exported from ArcGIS

FME & ArcSDE /Geodatabase 26 Writing ArcSDE / Geodb  Transactions & Commits  ArcSDE transactions can be controlled at the writer level or per feature  Writer parameters such as:  Transaction Type (geodb only)  Transaction Interval  Feature attribute (ArcSDE only):  fme_db_transaction = COMMIT_BEFORE | COMMIT_AFTER | ROLLBACK_AFTER | IGNORE  allows groups of related features to be committed or rolled back

FME & ArcSDE /Geodatabase 27 Writing ArcSDE / Geodb  Inserts, Updates & Deletes  Can be controlled at the writer level or per feature  Writer parameter:  Writer Mode = INSERT | UPDATE | DELETE (geodb)  restricts the writer to a single mode  Feature attribute (ArcSDE only):  fme_db_operation = INSERT | UPDATE | DELETE  allows each feature to control it’s destiny!  Updates and Deletes easier with the OBJECTID  Journal Tables (JTX):  track the changes (add, modify or delete) to allow incremental updates  FME can read the JTX XML (GMLFeatureReplacer) and extract feature changes

FME & ArcSDE /Geodatabase 28 Writing ArcSDE / Geodb  Tips on Performance  Use the ArcSDE writer if possible  Do not build the spatial index when creating new layers  Use load_only mode - set the grid size to -1  Build the spatial index after the data load END_TCL  Use Transactions if possible. Versioning is slower  Increasing the transaction interval / size  Set SIMPLIFY_GEOM to NO  Writing annotation features is slow  Use two writers – Geodb for annotation, ArcSDE for other features  Avoid user-defined fields on annotations, if possible

FME & ArcSDE /Geodatabase 29 Other Writing Tips  Use Import Feature Type definitions  If writing to existing ArcSDE this saves time  Put FME on the same machine as the ArcSDE Server  Network latency is significant when loading  Use High Precision ArcSDE Tables  This can save you headaches down the road  Much easier to get data into

FME & ArcSDE /Geodatabase 30 Reading ArcSDE / Geodb  Versions  supported by both ArcSDE and Geodatabase readers  can either read from existing version or create new version  more on this later  Spatial Query  allows a spatial predicate (INTERSECTS, TOUCHES, etc) and search envelope  WHERE clause  to be expanded in 9.2 for a full SELECT statement  Domains & Subtypes  resolve domains & subtype descriptions, i.e.: STATUS & STATUS_resolved

FME & ArcSDE /Geodatabase 31 Reading ArcSDE / Geodb  Networks  returns geometry elements and element ID, from / to junction IDs and junction geometry  Annotation & Labels (geodb only)  supports feature linked annotation  Metadata  not supported, but… FME can process metadata XML once it is exported from ArcGIS  Measures  loads the geodb/sde30_measures attributes

FME & ArcSDE /Geodatabase 32 Working with Raster Data  Reading from ArcSDE Raster  Reads mosaics (Raster Maps) and Catalogs  Raster maps can be queried using spatial extents  Writing to ArcSDE Raster  Writes both Raster Maps & Raster Catalog  Creates the FOOTPRINT (bounding box vector)  Control color compression, raster compression and pyramiding  Resolution set by the first raster loaded (as per SDE convention)  FME Universal Viewer can view rasters

FME & ArcSDE /Geodatabase 33 Rasters & FME Universal Viewer

FME & ArcSDE /Geodatabase 34 Working with Raster Data  Raster formats and more… Focus is on loading raster to database and database to raster extract JPEG2000MrSID ESRI ArcSDEOracle 10g GeoRaster DTED / CDED DEMERDAS GeoTIFFPNG / GIF ASRPHDF 4 (Hyperion & ASTER) ADRG / CADRGCADRG KMZESRI ASCII Grid

FME & ArcSDE /Geodatabase 35 ArcSDEQuerier  ArcSDEQuerier is an ArcSDE reader /writer embedded in a transformer  Enables user to access ArcSDE data in the middle of a workspace data flow  ArcSDEQuerier allows:  QUERIES | UPDATES | DELETES  supports spatial & attribute queries  Applications  more complex ArcSDE spatial queries involving multiple input features  transaction table support – JTX tables  multiple attribute only (WHERE) queries using several input features

FME & ArcSDE /Geodatabase 36 ArcSDEQuerier ArcSDE ArcSDEQuerier FME Feature Data Stream The ArcSDEQuerier enables data to be retrieved from the ArcSDE anywhere in a workspace. 1. Query Features are routed to ArcSDEQuerier 2. Data retrieved from ArcSDE 3. Retrieved data is output to rest of FME data flow

FME & ArcSDE /Geodatabase 37 Other Issues  The Geodatabase reader/writer is only available on Windows platforms  And requires an ArcGIS license be available  Will be available on other platforms in ArcGIS 9.2  The ArcSDE reader/writer is available on Windows and UNIX platforms  And requires no additional licenses  The ArcSDE reader/writer is faster because it avoids an entire layer of software  But has somewhat reduced functionality as a result  Many clients doing large scale data migrations use the ArcSDE writer for speed and portability  Running FME for loading on the same machine as ArcSDE Server makes a noticeable difference

FME & ArcSDE /Geodatabase 38 Working with Versions ArcSDE Base Version Child Version ArcSDE Reader 1.Create child version 2. Read from child version Data Update 3. Data is then updated either by FME directly or FME writes data to other system for update

FME & ArcSDE /Geodatabase 39 Working with Versions ArcSDE Base Version Child Version ArcSDE Writer Data Update 5. Write to child version 4. After update by FME or other app data is given to ArcSDE Writer 6. Reconcile and Post 7. If no conflicts then child version removed

FME & ArcSDE /Geodatabase 40 Other Safe Support for ArcSDE/Geodb  Geomedia edit ability  We have an ArcSDE editing capability enabling Geomedia users to edit ArcSDE  Uses version support of reader and writer  Fits nicely in Arc environment

FME & ArcSDE /Geodatabase 41 What’s in the Pipeline …  File Geodatabase  File Geodatabase is supported in ArcGIS 9.2  FME Geodatabase Writer Template File  Pass writer a reference geodatabase file and all settings will be used  Domains, relationships, feature datasets, feature classes, etc.  Rich Geometry Support  Rich geometry support will be enhanced  Writing Networks to Geodatabase

FME & ArcSDE /Geodatabase 42 What’s in the Pipeline …  High Precision Support  Full support for both low and high precision ArcSDE and Geodatabase files  History/Archiving Support  Probably not for FME 2007  ArcGIS Server  Use geodatabase reader/writer to connect to and read maps provided by ArcGIS Servers  Rasters  Improved raster support & functionality

FME & ArcSDE /Geodatabase 43 ArcSDE/Geodb and Server FME Spatial ETL Server ArcSDE/Geodb

FME & ArcSDE /Geodatabase 44 Summary  ArcSDE and Geodatabase provide very comprehensive database capabilities  A moving target with new features with each ArcGIS release from ESRI  Safe Software continues to focus on improving the usability of these formats  Let us know what you need when working with these formats  Even if it is just what should be easier!