1. ICS-FORTH September 21, 2010 CRM Digital A Digital Provenance Ontology Center for Cultural Informatics, Institute of Computer Science Foundation for Research and Technology - Hellas Martin Doerr Berlin, Germany September 25, 2011 TPDL 2011

2. ICS-FORTH September 21, 2010 CRM Digital  Requirements  Competitors  CRM and Provenance  Data example  About Provenance-based reasoning  Conclusions Outline

3. ICS-FORTH September 21, 2010 CRM Digital Scientific data are empirical or synthetic. Scientific data cannot be understood without knowledge about the meaning of the data and the ways and circumstances of their creation We use Metadata to assess  meaning (view, experimental setup, instrument settings),  relevance (depicted things, their status, their conditions),  quality (calibration, tolerances, errors, “artifacts”),  possibilities of Improvement and Reprocessing. From generation to use, permanent storage, reuse (life-cycle) Digital Provenance Metadata Requirements

4. ICS-FORTH September 21, 2010 CRM Digital Acquisition: Reliable registration of the process and context conditions  The experimental setup and environment (geometry, light sources, tools, obstacles, sources of noise/reflections etc.)  Capture device type, identity (individual behavior!)  Hierarchical model: Inherit metadata common to series of “shots”  The identity of the measured or depicted object import identifiers, metadata identity of location – GPS data import? Requirements

5. ICS-FORTH September 21, 2010 CRM Digital Processing: Reliable registration of parameters  Workflow logs, reliable identification of outputs with inputs input files (URIs!) output files (URIs!), formats, warning and error reports. S/W identifiers and parameters, manual adjustments! process types for reasoning  Reliable linking with captured data Use and Reuse: parts, wholes and annotation:  Composition of final products, information packages (SIP, DIP, AIP) composition of aggregates, selection of versions or parts for permanent storage, reuse or transfer between labs, to and from Digital Libraries.  Migration to other formats (compatibility and obsoletion)  Authenticity, rights Requirements

6. ICS-FORTH September 21, 2010 CRM Digital There is no provenance data standard format.  Too many application-oriented, partial, overspecialized solutions.  Several stand-alone models, overgeneralizations  No integrated ontology of activity context Competitors:  “Open Provenance Model”,”Provenance Vocabulary”,”Provenir”,”Premise” no notion of acquisition (measurement, observation), place Confuse agentive role with substance of actors, machines, S/W, context No notion of temporal indeterminacy  W3C Provenance WG precondition: No use of a larger reference ontology => a dogmatic reinvention of the wheel….”antimodularity of ontologies?” Competitors

7. ICS-FORTH September 21, 2010 CRM Digital Competitors “Provenance Vocabulary”

8. ICS-FORTH September 21, 2010 CRM Digital Competitors: “Provenir”

9. ICS-FORTH September 21, 2010 CRM Digital The Idea:  First conceived by Stephen Stead for CHI, San Francisco 2007  Scientific data and metadata are historical records!  Scientific observation and machine-supported processing is initiated, on behalf of and controlled by human activity in physical space-time, not in cyber-space!  Things, data, people, times and places are causally related by events.  Other relations are either deductions from events or found by observation events. CRM Digital: Specialize the CIDOC CRM (ISO21127)!  Will allow for rich integrated reasoning (Christ – Ascension – Ivory panel) Innovations:  The Digital Measurement Event transfers from physical to digital world.  Machines “act” due to human initiative and responsibility. Humans use machines. No non-human actors! CRM and Provenance

10. ICS-FORTH September 21, 2010 CRM Digital S t operator 1 st Computer coherence volume of mesh-creation coherence volume of acquisition Museum museumobject It-Lab 2 nd Computer coherence volume of rendering scan-data 3D model 3D Model Creation as Meetings scanner mesh-data

11. ICS-FORTH September 21, 2010 CRM Digital CRM Digital 2.5 http://www.ics.forth.gr/isl/rdfs/3D-COFORM_CRMdig_v2.5.rdfs

12. ICS-FORTH September 21, 2010 CRM Digital D2 Digitization Process E11 Modification E7 Activity E65 Creation E16 Measurement D11 Digital Measurement Event D7 Digital Machine Event D10 Software Execution D3 Formal Derivation D12 Data Transfer Event D27 Calibration Process CRM Digital 2.5 : Digital Events http://www.ics.forth.gr/isl/rdfs/3D-COFORM_CRMdig_v2.5.rdfs

13. ICS-FORTH September 21, 2010 CRM Digital D1 Digital Object E54 Dimension D9 Data Object E73 Information Object E70 Thing D8 Digital Device E22 Man-Made Object E84 Information Carrier D13 Digital Information Carrier D35 AreaD14 Software CRM Digital 2.5: Digital Things

14. ICS-FORTH September 21, 2010 CRM Digital D1 Digital Object E24 Physical Man-Made Thing P31 has modified (was modified by) P94 has created (was created by) E54 Dimension P40 observed dimension (was observed in) E18 Physical Thing E1 CRM Entity E16 Measurement E65 Creation P39 measured (was measured by) E11 Modification E28 Conceptual Object L1 digitized (was digitized by) L20 has created (was created by) D2 Digitization Process D11 Digital Measurement Event D9 Data Object D13 Digital Information Carrier L19 stores (is stored on) Digitization = feature transfer from physical to digital CRM Digital 2.5: Digitization

15. ICS-FORTH September 21, 2010 CRM Digital D7 Digital Machine Event D8 Digital Device D1 Digital Object L10 had input (was input of) D1 Digital Object L11 had output (was output of) L12 happened on device (was device for) D10 Software Execution D3 Formal Derivation L2 used as source (was source for) D1 Digital Object L13 used parameters (parameters for) D1 Digital Object D13 Digital Information Carrier L18 has modified (was modified by) D1 Digital Object L21 used as derivation source (was derivation source for) D1 Digital Object L22 created derivative (was derivative created by) CRM Digital 2.5: Software Execution Formal Derivation = feature transfer from digital to digital E55 Type P2 has type (is type of)

16. ICS-FORTH September 21, 2010 CRM Digital D7 Digital Machine Event D8 Digital Device D1 Digital Object L11 had output (was output of) L12 happened on device (was device for) D12 Data Transfer Event L10 had input (was input of) D1 Digital Object L14 transferred (was transferred by) D8 Digital Device L16 has receiver (was receiver for) L15 has sender (was sender for) D13 Digital Information Carrier L18 has modified (was modified by) CRM Digital 2.5: Data Transfer Event Unreliable transfer

17. ICS-FORTH September 21, 2010 CRM Digital Applications European IP CASPAR  European Space Agency: satellite data  IRCAM: Digital media performances  FORTH: Art Object Digitization  FORTH/ Metaware: Integrating Digital Rights with Provenance model. European IP 3D-COFORM  3D model acquisition by camera, manual or by camera array. Up to 20.000 files per object.  3D model acquisition by laser scan.  Mesh processing, rendering  Synthetic models and scene compositions.  Provenance-based reasoning  Scalable repositories, representative amounts of data.

18. ICS-FORTH September 21, 2010 CRM Digital 3D Acquisition Example: 3D Reconstruction from Photographs – The Gipsmuseum Campaign Sven Havemann, CGV, TU Graz June 30, 2009 worst case for metadata capture: a complex manual process

19. ICS-FORTH September 21, 2010 CRM Digital Object Acquisition Event OAE1 Sequence Event SE1 Capturing Event CapE1 Data Acquisition Event DAE1 Digital Documentation Event DDE2 Calibration Event CE2 Digital Documentation Event DDE1 Calibration Event CE3 Calibration Event CE1 D2 Digitization Process instantiation example has part Sequence Event SE1 Object Acquisition Event OAE1 Capturing Event CapE1 Capturing Event CapE1 Capturing Event CapE1 Capturing Event CapE1 Sequence Event SE1 Acquisition Workflow Hierarchy

20. ICS-FORTH September 21, 2010 CRM Digital Modelling the Acquisition Process (AP) Register:  Who, when, where.  equipment identifiers, equipment models, firmware  Setup geometry and conditions. Assumptions: worst case, a completely manual process!  Set of objects captured under common conditions.  Each object captured by a sequence of “shots”  Metadata are stored by “historical order” (like workflow logs) step-by-step as executed, not as planned! concatenated by referring to identifiers of previously existing or created entities and initialized events. = robust against exceptions in the planned workflow  Avoid redundancy of information Hold common information as high as possible in a hierarchy of nested activities

21. ICS-FORTH September 21, 2010 CRM Digital 3D Acquisition 21 Workflow 3D scanning – NextEngine 3D model creation – Meshlab Rapid prototyping Testing glue, stabilizers, colours Print final replica Colour final replica Example: The Kazafani Boat Found in 1963, during a salvage excavation in the now Turkish occupied part of Cyprus (inaccessible and destroyed site). Tomb from the 12th century B.C. Unique object, hand made pottery 40x20.5x23 cm – canoe boat shape Permanently exhibited at the Nicosia Museum

22. ICS-FORTH September 21, 2010 CRM Digital Data Acquisition Event - Schema Person: uuid:aeac5200-0138-11e0-a976-0800200c9a66 (E21 Person) P131 is identified by : D21 Person Name L51 has first name: Martin (Literal  E62 String) L52 has last name: Doerr (Literal  E62 String) P107 is current or former member: http://www.ics.forth.gr/ (E40 Legal Body) L62 in the role of: http://www.3d-coform.eu/RoleType/researcher (E55 Type) Persons (“operators”)

23. ICS-FORTH September 21, 2010 CRM Digital Data Acquisition Event - Schema Legal Body: http://starc.cyi.ac.cy/ (E40 Legal Body) L4 has preferred label: STARC-The Cyprus Institute, Nicosia, Cyprus (Literal  E62 String) no address P74 has current or former residence: http://www.geonames.org/146268/ (E53 Place) L4 has preferred label: Nicosia (Literal  E62 String) P3 has note: Cyprus (Literal  E62 String) exact address and the city where it is located P74 has current or former residence: uuid: dbae7cd0-e371-11e0-9572-0800200c9a66 (E53 Place) L4 has preferred label: 15 Kypranoros Street (Literal  E62 String) P89 falls within: http://www.geonames.org/146268/ (E53 Place) P3 has note: 15 Kypranoros Street, Nicosia 1061, Cyprus (Literal  E62 String) just the address without details for city P74 has current or former residence: uuid: dbae7cd0-e371-11e0-9572-0800200c9a66 (E53 Place) L4 has preferred label: 15 Kypranoros Street (Literal  E62 String) P3 has note: 15 Kypranoros Street, Nicosia 1061, Cyprus (Literal  E62 String) Legal Bodies & Places

24. ICS-FORTH September 21, 2010 CRM Digital Data Acquisition Event: uuid:354c91e0-b3fa-11de-98c6-0002a5d5c30a (D2 Digitization Process) L4 has preferred label: 2010 Laser scanning in Arch. Museum of Nicosia (Literal  E62 String) P2 has type: http://www.3d-coform.eu/EventType/laser_scanning (E55 Type) P2 has type: http://www.3d-coform.eu/EventType/data_acquisition (E55 Type) P3 has note: “evening sun shines through the west window” (Literal  E62 String) SUPER-EVENTS: P9 forms part of: uuid:07f05f40-b415-11de-9d48-0002a5d5c30c (E7 Activity) (Project) WHEN: L31 has starting date-time: 2010-05-28T08:00:00Z (xs:dateTime  E61 Time Primitive) L32 has ending date-time: 2010-06-02T18:00:00Z (xs:dateTime  E61 Time Primitive) WHERE: P7 took place at: http://starc.cyi.ac.cy/#Place/ArchMuseumNicosia/ConservationLab (E53 Place) WHO: L29 has responsible organisation: http://starc.cyi.ac.cy/ (E40 Legal Body) L30 has operator: uuid:aeac5200-0138-11e0-a976-0800200c9a66 (E21 Person) Data Acquisition Event

25. ICS-FORTH September 21, 2010 CRM Digital Data Acquisition Event: uuid:354c91e0-b3fa-11de-98c6-0002a5d5c30a (D2 Digitization Process) WITH WHAT (camera): L12 happened on device: http://www.nextengine.com/.../E4035623490 (D8 Digital Device) L59 has serial number: E4035623490 (Literal  E62 String) L4 has preferred label: “Next Engine Desktop 3D scanner” (Literal  E62 String)(=Model) P2 has type: http://www.3d-coform.eu/DeviceType/laser_scanner (E55 Type) L33 has maker: http://www.nextengine.com/ (E39 Actor) P3 has note: Next Engine Desktop 3D scanner, Multi stripe laser (Literal  E62 String) L23 used software or firmware: http://www.nextengine.com/.../Scan_Studio (D14 Software) WITH WHAT (additional devices): P16 used specific object: http://b2b.sony.com/.../SONVPLFE4035623490 (E22 Man Made Object) L59 has serial number: 32526158 (Literal  E62 String) L4 has preferred label: SONY PLFE 40 Projector (Literal  E62 String)(= Model) P2 has type: http://www.getty.edu/research/tools/vocabularies/aat/300022665 (E55 Type) L33 has maker: http://www.nikon.com/ (E39 Actor) P16 used specific object: http://www.cgv.tugraz.at/structure_slide_T45a (E22 Man Made Object) Data Acquisition Event

26. ICS-FORTH September 21, 2010 CRM Digital Object Acquisition Event: uuid:07f05f40-b415-11de-9d48-0002a5d5c30b (D2 Digitization Process) L4 has preferred label: 2010 Laser scanning of Kazafani Boat 249.377 in Archaeological Museum of Nicosia (Literal  E62 String) P2 has type: http://www.3d-coform.eu/EventType/object_acquisition (E55 Type) L10 had input: uuid:3d066a90-9cb1-11e0-aa82-0800200c9a66 (D9 Data Object) (calibration file) L10 had input: uuid:46963500-9cce-11e0-aa82-0800200c9a66 (D9 Data Object) (configuration file) SUPER-EVENTS: P9 forms part of: uuid:354c91e0-b3fa-11de-98c6-0002a5d5c30a (D2 Digitization Process) (Data Acquisition Event) Object Acquisition Event

27. ICS-FORTH September 21, 2010 CRM Digital Object Acquisition Event: uuid:07f05f40-b415-11de-9d48-0002a5d5c30b (D2 Digitization Process) WHAT (acquired object): L1 digitized: uuid:e4761f00-0ce7-11e0-81e0-0800200c9a66 (E22 Man-Made Object) P1 is identified by: http://www.mcw.gov.cy/mcw/DA/DA.nsf/Objects/249.377 (E42 Identifier) (all “known” URIs) L4 has preferred label: Kazafani Boat, vase, 249.377 (Literal  E62 String) L53 is not uniquely identified by: Kazafani Boat (Literal  E62 String) L53 is not uniquely identified by: Bronze Age model of a boat (Literal  E62 String) L55 has inventory no: 249.377 (Literal  E62 String) P2 has type: http://www.getty.edu/research/tools/vocabularies/aat/300132254 (E55 Type) (vase) P3 has note: “Deep hollow hull with in-curving flat-topped gunwale ….” (Literal  E62 String) P50 has current keeper: uuid:6f2972e6-ad9e-4a72-930d-263f01e75d8c (E40 Legal Body) (Archaeological Museum of Nicosia) Object Acquisition Event

28. ICS-FORTH September 21, 2010 CRM Digital Calibration Event: uuid:07f05f40-b415-11de-9d48-0002a5d5c30c (D2 Digitization Process) P2 has type: http://www.3d-coform.eu/EventType/calibration (E55 Type) L1 digitized: http://cg.cs.uni-bonn.de/#Calibration/Bariumsulfate/Block10 (E18 Physical Thing) L4 has preferred label: block of bariumsulfate (10x10x1cm) (Literal  E62 String) P2 has type: http://www.3d-coform.eu/InformationObjectType/MultiviewdomeCalibrationData (E55 Type) (color chart, ruler, greyscale) WHEN: L31 has starting date-time: 2009-07-02T16:04:34Z (xs:dateTime  E61 Time Primitive) L32 has ending date-time: 2009-07-02T16:04:34Z (xs:dateTime  E61 Time Primitive) OUTPUT: L20 has created: uuid:07f05f40-b415-11de-9d48-0002a5d5c31c (D9 Data Object) Calibration Event

29. ICS-FORTH September 21, 2010 CRM Digital Capturing Event: uuid:07f05f40-b415-11de-9d48-0002a5d5c30n (D2 Digitization Process) L4 has preferred label: Capture 1_0 for Boat (Literal  E62 String) P2 has type: http://www.3d-coform.eu/EventType/capture (E55 Type) SUPER-EVENTS: P9 forms part of: uuid:07f05f40-b415-11de-9d48-0002a5d5c30b (D2 Digitization Process) (Object Acquisition Event) WHEN: L31 has starting date-time: 2009-07-02T16:07:54Z (xs:dateTime  E61 Time Primitive) L32 has ending date-time: 2009-07-02T16:07:54Z (xs:dateTime  E61 Time Primitive) Capturing Event

30. ICS-FORTH September 21, 2010 CRM Digital Capturing Event: uuid:07f05f40-b415-11de-9d48-0002a5d5c30n (D2 Digitization Process) (cont’d) OUTPUT: L20 has created: uuid:07f05f40-b415-11de-9d48-0002a5d5c31g (D9 Data Object) (image file, zip file...) L4 has preferred label: 1_0.ply (Literal  E62 String) P2 has type: http://www.3d-coform.eu/ObjectType/mesh (E55 Type) P2 has type: http://www.3d-coform.eu/MimeType/ply (E55 Type) P43 has dimension:(E54 Dimension) P2 has type: http://www.3d-coform.eu/DimensionType/mesh_vertices (E55 Type) P90 has value: 394302 (xs:integer  E60 Number) P91 has unit: http://www.3d-coform.eu/UnitType/vertices (E58 Measurement Unit) P43 has dimension:(E54 Dimension) P2 has type: http://www.3d-coform.eu/DimensionType/mesh_faces (E55 Type) P90 has value: 782543 (xs:integer  E60 Number) P91 has unit: http://www.3d-coform.eu/UnitType/faces (E58 Measurement Unit) Capturing Event

31. ICS-FORTH September 21, 2010 CRM Digital ARC 3D component Images Used images + + Depth map for each used image + metadata  Perform the 3D reconstruction of an artefact from images retrieved from the RI  For an input sequence of images, ARC 3D produces a calibration matrix and a depth map for each image identified as usable for the reconstruction.  This output data is then ingested into the RI so that it can be retrieved and loaded into MeshLab to perform the final reconstruction step (integration of the depth maps). ARC 3D web service component

32. ICS-FORTH September 21, 2010 CRM Digital Process Event: uuid:2f7d22db-1d89-11e0-ac64-0800200c9a66 (D3 Formal Derivation) L4 has preferred label: Processing of Ivory Panel raw data with Arc3D (Literal  E62 String) P2 has type: http://www.3d-coform.eu/EventType/process_event (E55 Type) P2 has type: http://www.3d-coform.eu/EventType/modeling_process (E55 Type) WHO: L29 has responsible organisation: http://www.vam.ac.uk/ (E40 Legal Body) L30 has operator: uuid:2f7d22d3-1d89-11e0-ac64-0800200c9a66 (E21 Person) WHEN: L31 has starting date-time: 2010-08-07T08:00:00Z (xs:dateTime  E61 Time Primitive) L32 has ending date-time: 2010-12-02T10:00:00Z (xs:dateTime  E61 Time Primitive) WHERE: P7 took place at: http://www.vam.ac.uk/#Place/ConservationLaboratory (D23 Room) ARC 3D Process Event

33. ICS-FORTH September 21, 2010 CRM Digital Process Event: uuid:354c91e0-b3fa-11de-98c6-0002a5d5c50c (D3 Formal Derivation) (cont’d) WITH WHAT (Software): L2 used as source: http://www.esat.kuleuven.be/psi/visics/ARC3D (D14 Software) L4 has preferred label: ARC3D (Literal  E62 String) P2 has type: http://www.esat.kuleuven.be/psi/visics/ARC3D/Version_1.0.0 (E55 Type) P2 has type: http://www.3d-coform.eu/SoftwareType/processing_software (E55 Type) L33 has maker: http://www.esat.kuleuven.be/psi/visics (E39 Actor) L4 has preferred label: KULeuven PSI VISICS (Literal  E62 String) ARC 3D Process Event

34. ICS-FORTH September 21, 2010 CRM Digital Process Event: uuid:354c91e0-b3fa-11de-98c6-0002a5d5c50c (D3 Formal Derivation) (cont’d) WHAT (Input): L21 used as derivation source:uuid:2f7d22d2-1d89-11e0-ac64-0800200c9a66 (D9 Data Object) L4 has preferred label: A.15-1955-dome-out.zip (Literal  E62 String) WHAT (Derivative output): L22 created derivative: uuid:2f7d22dc-1d89-11e0-ac64-0800200c9a66 (D9 Data Object) L4 has preferred label: Arc3D-A.15-1955_dmy.v3d (Literal  E62 String) P2 has type: http://www.3d-coform.eu/#ObjectType/mesh (E55 Type) P2 has type: http://www.3d-coform.eu/#mimetype/v3d (E55 Type) (calibration files, depth map files, CUN file and respective images) ARC 3D Process Event

35. ICS-FORTH September 21, 2010 CRM Digital params devices, device models meshs params Acquistion metadata subevents Acquistion metadata subevents params raw data objects 2 nd Acquistion who, when, where, what, using what Reasoning: A Coherent Semantic Net object metadata features, history Acquistion metadata subevents Acquistion metadata subevents Acquistion metadata subevents Acquistion metadata subevents Processing metadata who, when, how, what, using what Processing metadata who, when, how, what, using what Processing metadata who, when, how, what, using what models meshs models Acquistion metadata who, when, where, what, using what raw data objects raw data objects raw data objects raw data objects raw data objects raw data objects software, algorithms software, algorithms …+ who - when - where

36. ICS-FORTH September 21, 2010 CRM Digital Good Reasons for Reasoning (3D-COFORM) The integrated semantic network of provenance metadata allows for supporting data consistency, interpretation, reuse, preservation Management:  garbage collection of all reproducible intermediate results (classify software!)  export packages: collect all acquisition data and parameters used for one model. Preservation:  monitor obsoletion of all processing tools, format viewers necessary to interprete or reprocess certain data. Property propagation to subevents and derivatives (economy & consistency):  For instance, “Which object represents my mesh?” result in long query paths: “Jesus Christ”. forms part of “Ascension” is carried by: “Ivory Panel”. was digitized by: “MiniDomeEvent2011934”.has created: “A.15-1955model v1.zip”.used as derivation source:…….

37. ICS-FORTH September 21, 2010 CRM Digital 3D-COFORM: Concatenated Metadata 1IvoryPanel_O bjAcqEvent.rdf 2IvoryPanel_ DocEvent.rdf forms_part_of A.15-1955-dome- out.rdf A.15-1955-dome-out.zip has_created 2009CA5306_0.rdf 2009CR4851_0.rdf … 2009CA5306_0.tif … 2009CR4851_0.tif has_created 4Ivory_Arc3DPro cEvent.rdf used_as_derivation_source Arc3D-A.15-1955_dmy.v3d created_derivative 5Ivory_MeshLa bProcEvent.rdf used_as_derivation_source 2009CA5307v Coloured.ply created_derivative digitized 3IvoPan_LegacyData.rdf Digitization_Process Formal_Derivation Sub-events Data_Object Legend Man_Made_Object 3IvoryPanel_N E_ObjAcqEvent.rdf digitized 4IvoryPanel_NE_ DetSeqEvent.rdf A.15-1955 corner scans.rdf A.15-1955 NXTENG whole model v1.rdf A.15-1955 NXTENG whole model v1.zip A.15-1955 corner scans.zip forms_part_of 7Ivory_NE_Mesh LabProcEvent.rdf A.15-1955 Degree scans.zip A15-1955 Retouched.zip created_derivative 8Ivory_NE_Mesh LabProcEvent.rdf 9Ivory_NE_Mesh LabProcEvent.rdf used_as_derivation_source A.15-1955_nxtng_5_degrees_ complete_bjbrown.ply created_derivative 6Ivory_NE_Mesh LabProcEvent.rdf A.15-1955 Master.zip used_as_derivation_source forms_part_of has_created used_as_derivation_source created_derivative forms_part_of has_created forms_part_of

38. ICS-FORTH September 21, 2010 CRM Digital Conclusions CRMdig provides a good high-level model for empirical provenance of digital data, open for further specialization, integrated with arbitrary context representations CRM-CRMDig outperforms competitors in expressive power and integration potential. Future work: Theory of property propagation to subevents and derivatives Needed: Theories of feature conservation by kinds of derivation processes. Links: http://www.ics.forth.gr/isl/rdfs/3D-COFORM_CRMdig_v2.5.rdfs