1 How to move test data from existing files into a satellite database Jacek Wojcieszuk Jacek Wojcieszuk Warsaw University of Technology

2 problem description Problems –Test data is usually collected via Labview software –Format of a raw Labview output is not compatible with the database structure so direct transfer of the data is not possible – transformation needed –There is currently many data collected and stored in simple ASCII files –This ASCII files have very often different and uncompatible formats. –To achieve goals of the database system for the Alice detector production, all test data should be stored in databases (including data currently stored in text files) Solutions –Uniform format of the Labview software output (to simplify transfer of data that will be collected in the future). The format should be: readible for a human easy to understand easy to validate not too complicated to process Choice: XML –Software inserting the unified Labview output into the database.

3 Short introduction to XML General information –eXtensible Markup Language –XML is the W3C standard –Similar to HTML –Usually used to describe logical structure of text documents –XML documents can be easily created manually, read, validated, and processed by computer programs –XML has many successful applications in almost all fields of computer science Main elements of XML –XML Documents consist of: elements (e.g. ) Attributes (e.g. ) additional and less important at this point mechanizms like: processing instructions and namespaces Document structure –An XML document has a hierarchical structure XML element can contain any number of attributes, subelements and/or plain text If element element contains other elements or text data it have to consists of beginning and ending tags Attributes can contain only plain text There have to be one and only one outer element (main element) which contains all other elements and/or plain text Any text Any text

4 Proposed XML Structure – exemplary document <data_labview version="1.16" database_structure_version="1.23"> timestamp name and version of the file generator IReS, Strasbourg <column name="USER_CODE">HAL25_TEST_UC <lookup name="TYPE_CODE" source_table="COMPONENT_TYPES" connect_by="NAME">SDDPASCAL 0 <lookup name="MANUFACTURER_CODE" source_table="MANUFACTURERS" connect_by="NAME">TORINO <lookup name="DETECTOR_CODE" source_table="DETECTORS" connect_by="NAME">SDD <lookup name="DATABASE_CODE" source_table="DATABASES" connect_by="NAME">adad 5 <column name="DESCRIPTION">something :21:54+00 <lookup name="COMPONENT_ID" source_table="COMPONENTS" connect_by="USER_CODE">HAL25_TEST_UC <lookup name="PROCESS_CODE" source_table="DEFINITION_OF_PROCESSES" connect_by="NAME">DIECHECK <lookup name="TYPE_CODE" source_table="COMPONENT_TYPES" connect_by="NAME">SDDPASCAL :00: <lookup name="MANUFACTURER_CODE" source_table="MANUFACTURERS" connect_by="NAME">manAWE any desc John Rohn Jimmy abc 1

5 Proposed XML Structure – elements and attributes desc. - root element of the document attributes: –version - version of this file –database_structure_version - version of the database structure subelements: – - this element includes general information about the file subelements: – - when the file was generated – - name of the program which generated the file – - name of the institute which generated the file - this element includes data of components attributes: –name - defines name of the main table in the database and has to be set to "COMPONENTS” subelements: – - this element represents one row of COMPONENTS table subelements: – - this element represents data from tables which are subordinate for "COMPONENTS" table attributes: –name – defines a name of the table in the database where data contained inside the element should go. subelements: –

6 Proposed XML Structure – elements and attributes desc. - this element represents one row of the database table of the name specified as the value of the attribute ‘name’ of the element subelements: – - this element indicates that data which, will be inserted into column, have to be selected from meta data table. Value of this element will be used as the value for column defined by connect_by attribute attributes: –name - this attribute defines name of the column, the value should be inserted in –source_table - this attribute defines name of the table, which we will select data from –connect_by - this attribute defines the name of the column, which will be used in "where" clause. - this element includes value of column which name is defined by name attribute attributes: –name – specifies a name of the column where the value ot the element should go - this element defines tables with process results subelements: – - values table row wrapper attributes: –nr – number of the row subelements: –tc - single value wrapper attributes: –cn – number of the column

7 Step by step – the header and main component’s data Data to transform XML structure version = 1.16 Database structure version = 1.22 Data collected in IRes, Strasbourg Component’s user_code = „Hal25_EXAMPLE” Component’s type name = „SDDAMBRA” Component’s is_virtual attribute = 0 Component’s manufacturer’s name = „WUT” Component’s detector’s name = „SDD” Database’s name = „TORINO” Component’s serial_number = 6 Component’s description = „description” Component’s production_date = „ ” Result of the transformation <data_labview version="1.16" database_structure_version="1.23"> :21:54+00 Generated by hand IReS, Strasbourg HAL25_EXAMPLE <lookup name="TYPE_CODE” source_table="COMPONENT_TYPES" connect_by="NAME">SDDAMBRA 0 <lookup name="MANUFACTURER_CODE" source_table="MANUFACTURERS" connect_by="NAME">WUT <lookup name="DETECTOR_CODE" source_table="DETECTORS" connect_by="NAME">SDD <lookup name="DATABASE_CODE" source_table="DATABASES" connect_by="NAME">TORINO 6 bla bla bla Components table structure: component_id : not null type_code : not null is_skeleton : not null default 0 is_virtual : not null default 0 manufacturer_code : not null detector_code : not null database_code : user_code : serial_number : production_date : description :

8 Step by step - parameters Data to transform Parameter1: Component’s user_code =„ HAL25_TEST_UC” Parameter’s name = „WAFERNO” Component’s type name = „SDDAMBRA” Parameter’s valid_from attribute = „ :21:54+00” Parameter’s value = 34 Parameter2: Component’s user_code =„ HAL25_TEST_UC” Parameter’s name = „Location” Component’s type name = „SDDAMBRA” Parameter’s valid_from attribute = „ :21:54+00” Parameter’s valid_to attribute = „ :21:54+00” Parameter’s value = „Int_lab” Result of the transformation <lookup name="COMPONENT_ID" source_table="COMPONENTS" connect_by="USER_CODE">HAL25_TEST_UC <lookup name="PARAMETER_CODE" source_table="DEFINITION_OF_PARAMETERS" connect_by="NAME">WAFERNO <lookup name="TYPE_CODE" source_table="COMPONENT_TYPES" connect_by="NAME">SDDAMBRA :21: <lookup name="COMPONENT_ID" source_table="COMPONENTS" connect_by="USER_CODE">HAL25_TEST_UC <lookup name="PARAMETER_CODE” source_table="DEFINITION_OF_PARAMETERS" connect_by="NAME">Location <lookup name="TYPE_CODE" source_table="COMPONENT_TYPES" connect_by="NAME">SDDAMBRA :21: :21:54+00 Int_lab

9 Step by step - processes Data to transform Tested component’s user_code = „HAL25_TEST_UC” Process name = „ANYNAME” Component’s type name = „SDDAMBRA” Process_date = „ :00:00+00” Process values = {12, 23}, {4352, 343} Manufacturer’s name = „CERN” Process description = „any desc” Process processed_by attribute = „John Rohn” Process inspected_by attribute = „Jimmy Abc” Process is_valid attribute = 1 Result of the transformation <lookup name="COMPONENT_ID" source_table="COMPONENTS" connect_by="USER_CODE">HAL25_TEST_UC <lookup name="PROCESS_CODE” source_table="DEFINITION_OF_PROCESSES" connect_by="NAME">ANYNAME <lookup name="TYPE_CODE" source_table="COMPONENT_TYPES" connect_by="NAME">SDDAMBRA :00: <lookup name="MANUFACTURER_CODE" source_table="MANUFACTURERS" connect_by="NAME">CERN any desc John Rohn Jimmy Abc 1