1. AMI – Status November 2010. Solveig Albrand Jerome Fulachier
AMI Databases
AMI – Status November 2010.
Solveig Albrand
Jerome Fulachier
Fabian Lambert
21/07/2018
S.A.

2. Summary Server & ORACLE problems and improvements. Bugs.
AMI Databases
Summary
Server & ORACLE problems and improvements.
Bugs.
Metadata Workshop report.
Developments.
Security & Information Protection.
21/07/2018
S.A.

3. No server problems
And the long standing ORACLE bug concerning the incompatibility of case sensitive searches and escaping the underscore character has been fixed.
A search for a string with an underscore in it now works correctly both at CCIN2P3 and CERN.

4. Performance improvements
We continue to make improvements in the reponse times of AMI. (For more details see TC talk of Jerome Fulachier in SIT )
All response times were reduced by ~0.5 seconds by caching the XSLT. (XSLT time now negligible)
Typical response time (99% of commands)
Overview page < 1 s. (35000 executions in Oct 2010)
DQ2GUID < 4 s.
The next slides show response times without XSLT!

5. Response time for AMITCEasyView
Ntimes
Response time for AMITCEasyView
One server is faster than the other ?
nVals 23763
24755
secs

6. AverageTime in seconds
Dataset Overview Page
Number of Executions
AverageTime in seconds
There seems to be a slight upturn in the last 2 months which we will be observing carefully!

7. Some bugs found in Pile Up datasets –
Problem with provenance – due to an over simple algorithm the calculation was missing generated event input when the pile-up was seen first. (FIXED)
Search by keyword = "pile-up" to see them.
Some recent pile-up not marked because the parameter name in the transform was changed.(Not yet fixed).
IT IS VERY IMPORTANT THAT AMI KNOWS ABOUT TRANSFORMATION CHANGES.

8. Metadata Workshop Report
30/31 August – report only made available (to me) last week, but we started work already on the requests.
Main request I retained : Get more run based information in AMI.
Read Access on the COMA tables (Elizabeth Gallas).
Rendered these tables browsable in AMI
Using aliases and views to "AMIfy" them.
Can now exploit them with AMI code.
Will be able to link dataset info with run info directly. But don't want to duplicate the runBrowser of course.
Already getting run period info through COMA– and we have plans for closer collaboration COMA/AMI on this issue. (see Elizabeth G.'s talk)

9. AMI thinks that COMA is an AMI catalogue
 Can use AMI generic code

10. "Show information about missing events and lumi blocks in AMI."
Work on Real Data
"Show information about missing events and lumi blocks in AMI."
Requirements from a meeting with Data Prep + metadata coordinators.
This is a vast project and only the first few steps have been done; comparing the number of events in a RAW dataset with the number of events of all its AOD, ESD and TAG descendants.
Need to synchronize on the file consistency service, COOL, and need "file to file" (LB) provenance tracking.

11. MC requirements
MD workshop "Meta-data interface looks a bit technical for the end user"
Transporting cross section values along the MC production chain (less clicks to get the values!)
Some requests are not possible to implement at present. E.g. No "automatic" way of marking datasets "September reprocessing". We cannot (will not) read this information from TWIKI pages or s.
Related development. : Reworking the "dataset numbers" broker, and extending it to hold production requests.

12. Other Developments Synchronizing with DQ2 :
AMI client for DQ2 stomp Active MQ service. ( done by a Summer student)
Used for Dataset & Dataset container deletion.
We would like to extend this service to
File consistency
Add/Remove primary datasets from dataset containers.

13. Security and Information Protection
Until last week most of the information in AMI was world readable.
Following a security audit of the AMI web site at CERN we were asked to put the access to the AMI replica behind SSO and to clean up some rather ugly responses to error conditions or attempts to inject java script. This has all been done with a lot of help from Flavia, and we now have as clean a bill of health as possible from CERN IT security.
But of course this does not protect any information from non-ATLAS members, and the main site remains world readable.

14. Towards Information Protection
What's more – the replica Web service is no longer accessible from outside CERN
What we plan to do in the short term, is to restrict world readable rights to the top page, and to permit only members of ATLAS VOMS to read AMI catalogues.
When this is done we can probably come out from behind SSO at CERN.  LATEST NEWS: CERN security cannot fix the web service problem so agrees that we come out from behind SSO, as long as they can monitor connections.
We would then have the same Authentication & Authorisation mechanism on both sites.