1. Conditions System Update and Discussion
Jeremy McCormick, SLAC
HPS Software Group

2. General Information
Sub-system groups will need to load detector conditions for the Engineering Run into the database, with the correct run numbers, and verify that the information is correct.
The exact steps of this process can depend on the type of condition.
There should be procedures in place for doing this easily, including scripts that automate the process as much as possible, especially for conditions that will be inserted for every run.
Some conditions can be generated by external scripts that produce text files, which can then be loaded via the command line tool.
Others conditions records could be inserted directly from a Driver, if there is a calibration algorithm within LCSim.
Certain types of conditions are likely to only be loaded once for the Engineering Run (e.g. ECAL channel mapping) whereas others could be updated as much as once per run (e.g. pedestal & noise calibrations).
The JLAB database contains the master copy of all detector conditions.
All updates of conditions should first go into the JLAB database.
Reconstruction production runs on the batch system at JLAB will use this database.
The database is only accessible from jlab.org hosts so…
It will be periodically cloned to SLAC for external access.

3. How does it work?
Each type of condition has an associated table in the database that contains records representing the conditions data.
The table contains sets of grouped rows that I generally refer to as “collections” or “conditions sets”.
Each of these tables has a collection_id column, which is used to group sets of records together into collections.
There is a central conditions table that assigns a collection to a run number range.
When the conditions manager is notified of a new run number, it queries the database to find matching conditions records based on their start and end run numbers. The conditions sets referenced by these matching records are then cached in the manager.
The conditions themselves are modeled as collections of Java objects that extend ConditionsObject, one per database record, and they are directly accessible through the conditions manager instance by type and table name.
The objects are returned as a Java Collection, and these containers can have specialized behavior depending on the type of condition (for instance to quickly look up objects by an ID or to sort them).
There are many working examples now in HPS Java of accessing conditions through the manager, including a number of tests in the conditions module.
The conditions manager has extra setup code to make sure that the conditions are properly associated with the detector elements such as crystals and sensors to make accessing them easy from user code.
These are called “detector setup” classes and are executed automatically.
The conditions manager itself now handles the setup of “Test Run” or “Engineering Run” conditions types and configuration, where they differ, based upon run numbers.

4. Conditions Command Line Tool
There is a basic command line tool for performing operations on the conditions system.
printing out conditions sets
loading conditions data into tables from text files
adding conditions records to the conditions table
It can be run from the standard HPS Java distribution jar.
[1076 $] java -cp ./distribution/target/hps-distribution-3.1-SNAPSHOT-bin.jar org.hps.conditions.cli.CommandLineTool -h
usage: CommandLineTool
Commands:
load
add
print
-h Print help and exit
-p Set the connection properties file
-v Enable verbose terminal output
-x Set the conditions database XML configuration file

5. Command Line Tool Examples
Get general help
java -cp hps-distribution-bin.jar org.hps.conditions.cli.CommandLineTool –h
Get sub-command help
java -cp hps-distribution-bin.jar org.hps.conditions.cli.CommandLineTool print -h
Print a conditions set
java -cp hps-distribution-bin.jar org.hps.conditions.cli.CommandLineTool print -r t ecal_channels
Load text data into a table
java -cp hps-distribution-bin.jar org.hps.conditions.cli.CommandLineTool -p connection.prop load -t ecal_calibrations -f ecal_calibrations.txt
Add a conditions record for a collection to give it a run number range
java -cp ./distribution/target/hps-distribution-3.1-SNAPSHOT-bin.jar org.hps.conditions.cli.CommandLineTool –p connection.prop add -c 1 -r e t ecal_calibrations -m "Hello calibrations."

6. Loading Conditions
The basic steps for loading new conditions are generally the following.
Generate a text file in the correct format which usually has a mapping between channel ID (crystal for ECAL, strip for SVT) and conditions information (e.g. calibrations, etc.)
ecal_channel_id pedestal noise
[etc.]
Load the conditions into the database using the HPS Java command line tool. (Example below)
java -cp hps-distribution-bin.jar org.hps.conditions.cli.CommandLineTool -p connection.prop load -c 1 -f ecal_calibrations.txt -t ecal_calibrations
Add a conditions record for the new collection to register it by run number.
java -cp hps-distribution-bin.jar org.hps.conditions.cli.CommandLineTool add -c 1 -r e t ecal_calibrations -m "Calibrations loaded for fun and profit."

7. Notes about Loading Conditions
You need to provide a connection properties file in order to make updates to the database. The contents of this properties file should look something like the following.
user: hpscalibrations
password: xxxxxxxxxxxxxx
database: hps_conditions
hostname: jmysql.jlab.org
The account hpscalibrations should be used for inserting new records and for making deletions when necessary.
The password on this account should be directly requested from Maurik or me if you need it.
The input text files with conditions data need to be formatted properly.
The first line should contain column headers that match exactly the column names in the database.
Then the data values should be listed separated by spaces, with one record per line.
The id and collection_id fields should not be included in this file, as they are assigned automatically by the conditions manager.
One needs to take care that the conditions loaded into the database make sense according to what is already there.
For instance, it can be quite easy to inadvertently add multiple conditions sets that have overlapping or conflicting run numbers where this is not intended.
In other words, one should understand first what is in the conditions database for a given run before attempting to update it with new information!

8. Using Drivers to Load Conditions
It is possible to directly insert conditions into the database from LCSim using code in a Driver.
There is an example here for doing this.
ecal-recon/src/main/java/org/hps/recon/ecal/EcalCalibrationsDriver.java
The example Driver calculates pedestal and noise values from ECAL raw mode data and inserts a new conditions set into the database for the run number that was processed.
The conditions API can be used directly in Java code to create the conditions collection, load it into the database, and then add a conditions record with the run validity.
For conditions that we might want to have for many different runs, this approach is likely preferable to working with text files.
It is much easier to automate this process and generate calibrations for every run rather than needing to create/load/add external text files.
It is preferable to work within LCSim where existing conditions are available, as well as detector information, reconstruction algorithms, analysis utilities, etc.

9. Connecting to the Database Directly
The MySQL client can be used to connect directly to the database from a machine with a jlab.org hostname.
mysql -h jmysql.jlab.org -uhpscalibrations -pXXXXXXXX -D hps_condition
From there you may execute queries directly.
mysql> select * from ecal_calibrations;
The hpscalibrations account has select, delete and insert privileges on the database.
Currently, Maurik and I are functioning as the database administrators with full privileges on the conditions database.
If you feel that you need more privileges than hpscalibrations allows, then a user account can be added for your name with the additional grants.

10. Copying the Database
I do not really recommend this but it is not hard to replicate the conditions database to your local machine.
Need to have correct database account permissions to create a SQL dump or get a SQL file from one of the admins. (SQL backups might be periodically checked in to SVN as well.)
# dump the db to a SQL file (must be on a jlab.org machine)
mysqldump -h jmysql.jlab.org -ujeremym -pXXXXXXX --disable-lock-tables hps_conditions &> conditions_db.sql
# now on your local machine load the SQL file into a db
mysql -h localhost -P ujeremym -pXXXXXXX -Dhps_conditions < conditions_db.sql

11. Group Responsibilities
The responsibility for loading conditions into the database by run and verifying their correctness lies with the sub-system groups (ECAL, SVT).
I have been directly assisting with this process for the ECAL, as the exact procedures have still been under development during the commissioning runs.
In the future, I will not be able to directly verify all the data going into the database, so each of the groups will need a “czar” responsible for doing this, as well as assigning related sub-tasks within their group (such as authoring/running calibration scripts).
SVT = Omar?
ECAL = Nathan?

12. TODO
Conditions records with overlapping time validity need to be handled better. (At this time, the behavior is partially unspecified!)
We can use tags here to disambiguate between different conditions sets. This feature is supported currently but needs further testing.
If a tag is not provided and there are overlapping conditions sets, then the most recently updated one could be used by default or some other disambiguation strategy employed.
The classes that load conditions onto the detector in an easily accessible form need to be updated to support overlapping run validity. This makes sense for some conditions such as bad channels, which could all be concatenated together, whereas for others such as the DAQ mapping, it does not make any sense to have overlapping conditions.
Currently, the SLAC database is running on my Linux machine (!), because I have full db admin privileges there. I am trying to coordinate with the database admins to move this to our MySQL SLAC and give me the appropriate permissions to be able to manage it there.
The cloning of the JLab database to SLAC should occur periodically on a fixed schedule, such as once per day. I plan to add a cron job that will do this from JLab.
Where can I setup cron jobs at JLab that will be run? ifarm?
The Java framework has various entry points (monitoring, EvioToLcio, steering etc.) and each of these requires the conditions system to be setup correctly in order to work.
Need to be able to override run numbers from data with a user specified run number (working on this in monitoring; already supported in steering via ConditionsDriver)
Run numbers from data files in EVIO headers and LCIO events need to be pushed to the conditions system so it can decide whether to update or not (this is working in monitoring and EvioToLcio)
Might have a default strategy of using the most recent run number when none is specified or available from data (have not done this yet but it could be useful)
MC is typically generated with run number 0, and this value needs to be disambiguated between Test Run and Engineering Run data (perhaps using tags?)
We should completely move away from using text files for sub-system conditions information, especially files that are not checked into the SVN.
Using text files that are not in source control for conditions data makes it difficult for the collaboration to access this information and therefore interpret the event data correctly for physics reconstruction!

13. And finally….
What color do you want that database?