National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Discovery of Relationships between 2D Engineering Drawings and 3D CAD Models Presented by: Peter Bajcsy -Research Scientist at NCSA -Associate Director of I-CHASS, I3 Institute -Adjunct Assistant Professor, CS & ECE UIUC

Acknowledgement This research was partially supported by a National Archive and Records Administration (NARA) supplement to NSF PACI cooperative agreement CA #SCI , ONR TRECC, and NCSA Industrial Partners. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the National Archive and Records Administration, or the U.S. government. Contributions by: Peter Bajcsy, Kenton McHenry, Rob Kooper, Michal Ondrejcek, Jason Kastner and Luigi Marini Imaginations unbound

Outline Introduction Previous Related Work to Relationship Discovery A Discovery of Relationships Among 2D and 3D Digital File Collections (file2learn) Summary

Introduction

Discovering Relationships: Introduction Problem statement: How should one establish relationships among electronic records coming From disparate sources Before and after processing From the same source at multiple time instances? Note: Sources = digital information providers Reports from agencies, complex measurement instruments, simulations by computers, twitter, … Imaginations unbound

Discovering Relationships: Introduction File content changes over time and space Motivation: need for relating new files to existing files Applications: perform change detection, search, appraisal, social networking, … Additional cost of redundant bytes Motivation: need to minimize the cost Applications: content repository management, processing, preservation, … Imaginations unbound

Discovering Relationships: Challenges How to ingest content coming in many file formats and many data representations? What files are of my interest? How to describe file content? How to extract descriptors of file content? How to represent and store file content descriptors? What methods and metrics to use for comparing descriptors of two files? How to support end applications with comparisons/relationships? Imaginations unbound Automation and Scalability

Previous Related Work to Relationship Discovery Open source software Web services for communities

A Stack of Prototype Solutions Conversion Engine (Polyglot) Conversion Software Registry (CSR) Content Management (Medici) 3D-2D Relationship Discovery (File2Learn) Closed Source Software Software Reuse (AutoScript) Electronic Records Hardware Document Appraisal (Doc2Learn) Image Provenance Evaluation (IP2Learn) Computational Scalability Testbed Image Understanding (Im2Learn) Geospatial Image Understanding (GeoLearn, SP2Learn) Comparison (Versus) Distributed Computation (Cyberintegrator) 3D File Comparison (Model Browser)

Conversion Software Registry (CSR) Conversion Engine (Polyglot) How to ingest content coming in many file formats and many data representations?

Conversion Software Registry

File Format Conversion Engine: Polyglot

What files are of my interest? Multimedia content management (Medici)

3D Comparison Example (ModelBrowser) Software: Adobe 3D Reviewer Original File: WRL Converted Files: STP, STL, IGS, U3D Comparison Method: Light Fields [Chen, 2003] compares silhouettes from various viewing angles around the objects heart.wrl heart.stp heart.stl Conclusion: Information loss(WRL  STP)=Information loss (WRL  STL)>0

Comparison of Complex Files Content Based Comparison of Files in the Same Format: Given hundreds of versions of the ‘same’ Adobe PDF file, which file version(s) are similar?

Multiple Object Comparisons (Doc2Learn) Adobe PDF documents ~ {text, images, vector graphics, ….}

What methods and metrics to use for comparing descriptors of two files?

Comparison Framework: Versus

Information Loss Evaluation: Computational Requirements Files: one file in STP file format Software: Adobe 3D Reviewer, Cyberware PlyTool Comparison Method: Light Fields [Chen, 2003] Number of closed paths: 10 (28 individual conversions) Phase I: Find Phase II: Execute Phase III: Compare

A Discovery of Relationships Among 2D and 3D Digital File Collections

Relationships Among 2D and 3D Data Types Example Data: Torpedo Weapon Retriever existing 2D image drawings and N>22 3D CAD models Problem: How to establish relationships among the 3D CAD models and 2D image drawings during a product lifecycle? Imaginations unbound Hypothetical Distribution of 3D CAD models for TWR 841

Methodology File Identification Information Extraction from File System File Content Information Organization Taxonomy (classification) Ontology (relationships) Information Representation, Integration and Storage XML RDF Relationship Discovery

File Identification and File System Analyses File Identification What is the file format? Is the file format well formed? Approach: Used DROID built on top of the PRONOM File Registry with additional NCSA support of 3D file identification Metadata extraction about a file system Where is the file located? What is the file size, time stamp, etc.? Approach: Use any file system information extraction software, such as Aperture (cross platform, open source, active development), Google desktop, OS specific solutions (e.g., Apple Spotlight, Linux, MS Search)

Imaginations unbound OCR Content Analyses: Automation ? File Descriptors Relationship Discovery Part name, Author, Software, Date, …

Content Analyses: Optical Character Recognition (OCR) of 2D Drawings Reference Block Title Block MMC Block (Marinette Marine Corporation)

‘Standard’ Title Blocks: Organization and Ontology Examples of title blocks used on drawings prepared by Naval Construction Battalion and Naval Construction Regiment TEMPLATES

Title Block: Ontology and Metadata Representation Ontology for sub-fields: A – Record of preparation ( ), B – Drawing title ( ), C – Preparing Activity, F – Code identification number ( ), G – Drawing size ( ), H – Drawing number ( ), J – Scale ( ), K – Specification number ( ), L – Sheet number ( ). Resource Description Framework (RDF): Metadata representation: subject – predicate - object

MMC and Reference Blocks: Organization MMC Blocks Inconsistencies The list varies in length The notation is not standardized

Summary of OCR Based Analyses Manually encoded block coordinates for 784 files in PNG (converted from originally LZW compressed TIFF files) Automated OCR and executed OCR on 700 title blocks, 150 reference blocks, dozen of revisions and lists of materials about 200 additional areas with the drawing numbers (MMC DWG. NO.). Performance benchmarks: Full OCR of TB, MMC and RF for about 50 image files (105 blocks) took about 6 hours on a quad core machine

Content Based Extraction from STEP Files 3D CAD models in STEP file format are searched for any ASCII strings matching English dictionary and following STEP metadata specification. STEP METADATA SPECIFICATIONEXPECTED STEP METADATAPARSED STEP METADATA FILE_DESCRIPTION( /* description */ (''), /* implementation_level */ '2;1'); FILE_NAME( /* name */ '', /* time_stamp */'', /* author */ (''), /* organization */ (''), /* preprocessor_version */ ' ', /* originating_system */ '', /* authorization */ ' '); FILE_DESCRIPTION((''), /* implementation_level */ '2;1'); FILE_NAME( '120 TORPEDO WEAPONS RETRIEVER, TRANSVERSE BULKHEADS BELOW, MAIN DECK', ‘ ', ('LDOBSON'), ('NAVAL SEA SYSTEMS COMMAND'), ' ', 'IDA-STEP', ' '); FILE_DESCRIPTION((''), '2;1'); FILE_NAME( 'D:\\NARA\\Archieve_data_samples\\BHD_FR12\\ U2110_BHD12_2007_05_09.stp', ' T13:45:37', ('rakowpj'), (''), 'Autodesk Inventor 11', ''); Example Metadata for TWR841 ship deck

Exploratory Framework – User Interface Overview Files Filter for Files Preview of Selected Data Graph of Relationships Between Selected Files

Exploratory Framework – User Interface Overview Table of Relationships Between Selected Files Additional Import/Export and Preference Options

Exploratory Framework: Modes of Operations Detection of discrepancies/anomalies in file descriptors OCR results View 2D drawings and OCR results, and then edit OCR descriptors 3D Model View 3D model and content based extraction, and then edit descriptors Comparison of pairs of files Pairs of 2D drawings Pairs of 3D models Pairs of (2D drawing, 3D model) Establish file relationships Insert logical links to relate a pair of files

Detection of Anomalies in OCR Results

Comparison of Files Color encoding: Predicates and values match Predicates match Predicate occurs only in one file

Establish File Relationships

Establish File Relationships: Logical Link

Summary In general, finding relationships is still an open problem Automation and computational scalability are the keys to keeping files current, to dealing with the quantities, and to finding information Preservation solutions Forward looking solutions to preservation are based on standards Contemporary solutions to preservation are based on understanding historical electronic records

Software Summary File2Learn software – the work is still in progress The technologies are documented at Most of the software is available for downloading at Feedback is very welcome Questions: Peter Bajcsy –