ICAT Integration at DLS. Alun Ashton

What were the requirements? Integrate with current business system Collect Data and Metadata relating to a proposal Security Long term storage of data (Multi?) Institutional repository Searchable Metadata Potentially extendable with future e-science infrastructure Scalable

What were the restrictions? Only relies on Beamline Network. Must not prevent scientist collecting data. Data is collected by putting files onto a UNIX file system. Use e-Science research output. Interest, vision, priorities and finances.

What was the solution? Diamond, Modified by e-Science GDA DArc and Storaged Nexus & Data DUO DLS ICAT SRB DataPortal Diamond Proposal Web pages Atlas Data Store DUO Desk Active Directory People DB IKitten Data / metadata

What was the solution? Diamond, Modified by e-Science GDA DArc and Storaged Nexus & Data DUO DLS ICAT SRB DataPortal Diamond Proposal Web pages Atlas Data Store DUO Desk Active Directory People DB IKitten Data / metadata

What goes into the ICAT/Ikitten Who When Where ‘what’ –Abstract –Visit Triggers I think moves the data to the iKitten DLS ICAT DUO Desk IKitten

What does the GDA read from the IKitten Metadata read only –Fedid – the GDA knows the users Fedid and asks the DB for the current visit for that user, used for setting up data paths. –Instrument, abstract GDA IKitten Data / metadata

Role Based access and application sharing. Definable access Definable role If this is not right…

What does the GDA write out? GDA does not currently write a NeXus file, –Detector files and anything….. Currently implemented only on MX beamlines GDA Nexus & Data

‘GDA’ needs to initiate next step In the MX GDA whenever an image file is created it runs a script Bash script creates an XML file as input to DArc The GDA can produce a script itself but no beamlines currently do today… maybe next week? GDA Nexus & Data

DArc Watches a directory Converts file to XML Ingest format Initiates storaged Copies a filelist To a location Moves XML to A log Python Based EDNA Server that checks <>5s Runs XML ingest Success?

The XML file, minimum! MX307 MX i04 dont need it experiment icr/fernando EXPERIMENT_RAW unknown FG2_3_MS_3_154.img srb://srb-mcat-i18.esc.rl.ac.uk:5518/dls-2/i04/data/data/2009/mx /icr/fernando/FG2_3_MS_3_154.img unknown T02:58:21

What is our status?

New Eclipse Based GDA

Viewing and choosing the Metatdata Relation to NeXus

By the by… Did try a view, now use a script DLS ICAT DUO Desk ISPyB IKitten

ISPyB and ICAT ICAT –Archive –Generic ISPyB –LIMS –Domain specific

How robust is the system? GDA DArc and Storaged Nexus & Data DUO DLS ICAT SRB DataPortal Diamond Proposal Web pages Atlas Data Store DUO Desk Active Directory People DB IKitten Data / metadata

How robust is the system? GDA DArc and storaged Nexus & Data DUO DLS ICAT SRB DataPortal Diamond Proposal Web pages Atlas Data Store DUO Desk Active Directory People DB IKitten Data / metadata

Future Work Use it!!! Portal and API integration into in house tools Sample information from user office (top level information about the sample, and ISPyB (the instance of the sample). –Internal ICAT. Reduced data and richer metadata. Archive old data

Acknowledgements Diamond Light Source: Tobias Richter, Stuart Campbell, Karl Levik, Marc Basham, Karl Levik, Bill Pulford (and others in groups who have contributed at various times on various beamlines) STFC e-Science: Michael Gleaves, Roger Downing, Rik Tyer, Glen Drinkwater, Shoaib Sufi, Phil Couch, Kerstin Kleese Van Dam, Keir Hawker, Carmine Cioffe, Gordon Brown, Lisa Blanshard, Kevin O’Neill. STFC Daresbury: Steve Kinder and Karen Ackroyd But especially, Roger, Keir, Carmine, Michael, Mark and Tobias. BTW we have 64Tb of raw data already