ICAT- Experience and activities at ISIS Tom Griffin, STFC ISIS Facility 4th PaN-Data & CRISP Harmonisation Meeting XFEL/DESY January 2013 tom.griffin@stfc.ac.uk

Summary Data flows Authentication Authorisation Data policy Problems Business data - Other systems Experiment data Authentication Authorisation Data policy Problems Future plans

Getting data into ICAT Pre-experiment C# Creates empty investigations and datasets ready for data Must update ICAT after changes

Getting data into ICAT During the experiment ‘ICATIngest’ Subscribes to file system events ReadDirectoryChangesW() Invokes ‘WriteRaw’ and ‘NxIngest’ Caches groups of files for performance Extracts ‘science’ metadata

Data files ISIS RAW files Nexus Custom binary format + custom reader Outputs ICAT friendly XML Nexus Not really application definition compliant Use mapping files to convert nexus to ICAT friendly XML

Getting Data out - Web Access Main point of access for ISIS data HTTP file serving – pluggable architecture Zipped or non-zipped

TopCAT – basic web interface

Getting Data out - Mantid More convenient for users Must work on and off site Derived data upload Potential for provenance tracking

Authentication ICAT 4.x plugin architecture Web services to ISIS user database 15 minutes to write Single sign on version too Much easier to reuse existing auth systems

Authentication -Umbrella For specific permissions will require local account stub Facilities may require local registration for ‘public’ data At ISIS will all go through local user database Single authentication provider in ICAT

Authorisation My implementation not complete Increasing test coverage…. So far just under 100 rules Not everything is possible Limitations around devolving control

Data Policy ICAT implements a data policy in rules Who can access what data when ISIS policy published and implemented ICAT is flexible http://www.isis.stfc.ac.uk/user-office/data-policy11204.html tiny.cc/isisdp

DOIs – data citation When to issue?

Problems Tracking reference numbers numbers Linking calibration data ‘Fixing’ data that has been wrongly assigned Migration from 3.3 -> 4.2.2

Future Complete migration to 4.2.2 / 4.3 Replace current data server with IDS Continue integration with Mantid Improve and share tools (fixing data) Federation of catalogues Not just ICATs OAI-PMH etc

Summary 3.3 in use for several years Upgrading to 4.2.2 Data upload key to future success Easy to link to user database and umbrella Hardest part is integration with business and experiment data

Questions...