4CeeD: Private Cloud and Data Cyber-Infrastructure for Scientific Instruments Steve Konstanty, Senior Research Programmer, CSL

Overview What is 4CeeD Storage Facility and 4CeeD Services Why are 4CeeD Services important to MRL/MNTL researchers How does MRL/MNTL researcher work with 4CeeD Services How does MRL/MNTL researcher sign for 4CeeD Services

4CeeD Storage Facility and Services Goals Address Scientific Digital Data Acquisition, Curation and Sharing prior to Scientific Publication of Results via Private Cloud Storage Facility

Scientific Digital Data Acquisition and Workflow Challenges Data come of various types and multimodal formats Each type requires a different data processing workflow Sample output data from SEM microscopy SiO2 Mask Deposition Diffusion Plasma Etching Oxidation Device Characterization Metallization Lithography SiNx Deposition SiNx Removal Profilometry Ellipsometry SIMS SEM Profilometry SEM SEM Profilometry SEM Optical microscopy SPA Optical microscopy What are the main challenges for building an advanced cyberinfrastructure for long-tail scientific data? Long-tail data is often small/medium in size, but heterogeneous & multimodal in formats Analysis on long-tail data often requires real-time response

4CeeD System Architecture Scientific Data Input/Output Scientific Devices User Office Uploader Curator Client LAN/WAN Uploader/Curator Server Task/Resource Coordinator Service Private Cloud Compute/Storage Services github.com/4ceed

4CeeD Private Cloud Compute/Storage Service What we looked for Redundancy Availability Scalability Storage Layer 40 TB (20 TB per investor) Replicated for redundancy Compute Layer Docker container orchestration (Kubernetes) Single master (High Available masters in future) 4CeeD Cloud Storage 4CeeD Cloud Compute

(Simple and Speed-Up Usage at Microscopes) 4CeeD Uploader Service (Simple and Speed-Up Usage at Microscopes) Simple steps, with support for advanced usage 1. Choose or select a collection. 2. Load template and enter user defined metadata to create a dataset. 3. Upload files to cloud coordinator. Accurately and quickly record metadata through use of templates. Users can create, edit, and share templates.

4CeeD Curator Service (Speed-Up Curation) File View Dashboard View Previewer Download, Delete, or Subscribe for updates Hierarchy of file. Add tags. Example of system generated metadata (cropped from list of 100’s) Owner information [Preview, annotate, download, extracted metadata] [Dashboard management]

4CeeD Smart Data Management 4CeeD Data Model organizes projects into collections, datasets, and files. These can then be shared in spaces. 1. Picture of the process used by researchers to collect data from a sample. 2. Example of how the 4ceed data model stores data and metadata, There’s no reason for users to have to convert their data now. The 4CeeD Data Model organizes projects into collections, datasets, and files. These can then be shared in spaces.

Why 4CeeD? Easy to use tool for collecting data and metadata for storage and sharing with research group

Current Situation in MRL Facilities Current situation for experimental data involves manual processes for data capture and storage leading to poor documentation of results Data transfer is often done via “sneaker-net” techniques using flash-drives or email No data file conversion is available “Best” results and images are kept, but what is “best” is determined by a narrow, specific scientific objective. “Imperfect” data is often discarded or not available for others to review.

Current State of Data Capture Fabricate experimental sample Prepare analytical sample Bring sample to instrument for analysis Extract data (File conversion) Transport data to office computer Analyze data Repeat per iteration Extract data (File conversion) Transport data to office computer Limits of home analysis -Only what you have -Connect notes to data (after time delay) -Retroactive not reactive Analyze data ‘Sneakernet’ -Security risk at both ends -Very limited transport space -Lost or forgotten flash drives Metadata Loss -Excessive file name schemes -Manual notes -What metadata is important? Instrument (MRL/MNTL) Flash Drives Office

4CeeD Data Capture Direct web interface Real time Lossless data Fabricate experimental sample Prepare analytical sample Bring sample to instrument for analysis Extract data Transport data to office computer Analyze data Repeat per iteration Extract data (NO FILE CONVERSION) Transport data to office computer (DIRECT) Analyze data (REAL TIME) Direct web interface -Highly secure data storage -Large transport volume -Easy to use/ annotate Real time -Metadata paired with data -Reactive interpretation -Easy interpretation/searching Lossless data -More simple file names -Automatic note taking -All metadata included Instrument (MRL/MNTL) Laptop Collaborators Campus PC Office

Access to 4CeeD in MRL Access to 4CeeD through web-browser (https:/4CeeD.illinois.edu) Extraction of meta-data automatically performed Storage of data on RAID storage cloud maintained by Engr-IT All researchers of MRL Facilities (and their collaborators) will be allowed access to the MRL space

4CeeD advantages One stop location for all of your research data Sharable between you, your advisor, and your research partners Visual interpretation of data without having to open specialty software Cloud based  you can access your data anywhere Provides machine specific meta-data on measurement for future replication of measurements Provides templates which allow for the entry of metadata specific to YOUR project

Extractors for: TEM/SEM/AFM/SIMS/XRAY How does it work? Web based application for uploading, sharing, downloading, and working with data. Extractors for: TEM/SEM/AFM/SIMS/XRAY

Process of signing up, and confirming through email Process of signing up, and confirming through email. Must have an account to share.

Dashboard. File tree view Dashboard. File tree view. Copy, Move, Hover Preview, Delete (multi) coming soon.

View files in dataset. Share, copy, download.

View dataset user metadata and make changes.

View file preview, and system extracted metadata. Apply tags if desired.

Select a collection. Create or use an existing dataset Select a collection. Create or use an existing dataset. Templates enable shareable and reusable recipes to collect user metadata. Drag files in to window.

For archive or complete automation, zip up your files, and we’ll take care of everything for you.

Standard free text search. Faceted filtered search coming.

Create virtual spaces to share out data and assign role based permissions to users.

Advanced Materials Characterization Workshop June 6 & 7, 2017

Partner with Engineering IT for Maintenance 4CeeD private cloud for MRL/MNTL Production cloud with Engineering IT staff maintenance 40 TB cloud for Material Research and Semiconductor Research at UIUC Purchase and Installation Started in Spring 2017 Extensive Testing during Spring/Summer 2017 Educational Material to Use the 4CeeD facility are available on 4CeeD website https://4ceed.github.io/ http://t2c2.csl.illinois.edu/ Workshops to Use the 4CeeD facility will be offered in Summer and Fall 2017 4CeeD Services are available now 4CeeD Services will be in full production mode in August 2017 (beginning of Fall semester) 24 Dev Ops Center, the full power and support of Engineering IT. Upcoming 4CeeD workshops and educations materials. SIGN UP NOW!!!