Managing Data with iPlant Introduction to Uploading, Downloading, Sharing, and Metadata in the Data Store

Background on the iPlant Data Store iPlant Learning Center provides quick tutorials, slides, and documentation for everything you will see here iPlant Learning Center Backbone of the iPlant CI Connects to all iPlant services Appropriate for any (research) file types of any size Cloud Based, Backed Up, Initial 100 GB expandable to 1 TB Built on iRODS Folder = Collection Other than that, you don’t have to think about this if you don’t want to Demo 1 – Navigating the Data Store in the Discovery Environment

Upload and Download In the Discovery Environment  ‘Simple’, for small files (~ 5 files, <1.9 GB)  ‘Bulk’, for larger files and folders (<10GB)  Import from URL (no size limit) Advantage + Disadvantage - Covers most upload/download sharing needs Point and Click Some size/speed limitations Demo 2 – Transferring Data from the Discovery Environment

Tips Spaces /Special Characters Many software packages are sensitive to spaces in file names and/or the special characters below. Users may wish to rename uploaded files before using them in an analysis. Good advice for any transfer method. ~` $ % ^& *()+ = {}[]|\:;"'<>,?/

Bulk Transfers Requires Java 6 or later be enabled in your browser ( Not currently compatible with Google Chrome Window and web browser must remain open and active until the transfer is complete Import from URL Monitor Notifications to check that the URL import has been submitted – you will receive a notification when import is complete Tips

iDrop Desktop  Drag and Drop files and folders  File sizes up to your total allocation  Fast transfers  Synchronize folders with Data Store  Download and Installation Instructions Download and Installation Instructions  Can demo installation Advantage + Disadvantage - Upload/download large file sizes and numbers of files Sharing and permission features more complex Demo 3 – Transferring Data with iDrop Desktop

iDrop Requires Java 6 or later ( At the bottom of the iDrop window you can monitor, pause, and restart transfers. You can also view additional details by clicking Manage. When iDrop Desktop is open/running there will be an icon in your system tray or menu along with other background programs (e.g. Wi-Fi, Bluetooth, etc.). You need to close iDrop from this icon to completely close the program. Tips

iCommands Advantage + Disadvantage - Customizability Requires at least some command line expertise  Ability to script and automate  Access from terminal/server  Can resume transfers  Download and installation instructions Download and installation instructions  Can demo installation Demo 4 – Accessing the Data Store with iCommands

Tips -f force - overwrite local files -P output the progress of the download. -r recursive - retrieve subcollections (directories) -T renew socket after 10 min.(use with large files) -V Very verbose Useful iget / iput options:

Sharing Files in the Data Store Discovery Environment Sharing Sharing via Public Link Share files/folders instantly Control access permissions Manage sharing between collaborators No iPlant account required Limited to individual files URLs are public (less secure, can revoke) 2 Easy ways to share data within the Discovery Environment Demo 5 – Data Sharing in the DE

Tips When sharing via the Discovery Environment, use the following chart to decide the permissions you wish to grant: PermissionReadDownloadMetadataInfo TypesRenameMoveDelete Read Write Own

Need help? iPlant Learning Center provides quick tutorials, slides, and documentation for everything you will see here iPlant Learning Center

Analyses in the Discovery Environment Using Bioinformatics Apps

Background on the iPlant Discovery Environment iPlant Learning Center provides quick tutorials, slides, and documentation for everything you will see here iPlant Learning Center So far we have mainly looked at the Data Tab DE is also a powerful interface for Apps and Analyses This is where scalability and extensibility really come into play Can customize and create new apps Seamlessly integrated with high-throughput computing Apps are linked to iPlant wiki for documentation Demo 1 – Apps and analyses in the DE

Tips Mark an App as a favorite and it will appear in your workspace. Use the Apps menu to: Rate Apps to provide feedback Click the info icon to see the user manual for an App Re-launch a job by clicking on the App name in the Analyses menu. The App will re-launch populated with the last parameters used, giving you the option to alter the settings you want.

Tips View a file containing metadata (settings, etc.) connected to your analysis. Select the a job and click View Parameters.

Tips Can customize tools in the DE (simplifying here) App installation / modification involves happens at two levels: Apps (and dependencies) are installed on DE Cluster (done by iPlant support) DE Interface (created by App integrator/user) and published to DE Detailed instructions with videos, manuals, documentation in Learning CenterLearning Center

Viewing and Editing Metadata In the DE User metadata stored AVUs Attribute – Value – Unit Template-based metadata Can view and edit from with iCommands (tomorrow) Demo 2 – Metadata in the DE

Tips Currently can only use one template at a time Can create custom metadata templates Metadata in the DE

Need help? iPlant Learning Center provides quick tutorials, slides, and documentation for everything you will see here iPlant Learning Center

Syncing Folders With iDrop iPlant Learning CenteriPlant Learning Center provides quick tutorials, slides, and documentation for everything you will see here

iDrop Desktop Synchronization 1.Click on Settings 2.Click on the Synchronization tab 3.Click New to start a new synchronization 4.Enter a name (e.g. Project 1 sequence data) 5.Click Choose Local Folder to select a folder 6.Select a synchronization mode 7.Select a frequency for synchronization 8.Click Choose iRODS Folder to select the location to synch to 9.Click Update to save your synchronization Demo – DropBox Sync

Tips  Choose one of three methods to synchronize files: Local > iRODS – files will copied at synch to Data Store Local < iRODS – files in Data Store copied to local computer at synch Local <> iRODS – files in Data Store and local computer both synched If syncing to a DropBox folder, Dropbox must be paused during set up

Need help? iPlant Learning Center provides quick tutorials, slides, and documentation for everything you will see here iPlant Learning Center

Advanced Topics Searching in the DE, iCommands and Metadata, Data Commons Plans and Progress

Searching in the Discovery Environment Demo 1 – Basic and advanced searching Basic search bar lets you search all files and folders where you have permission Advanced search features allows searching based on metadata, permissions, and share status

iCommands to view and edit metadata Demo 2 – imeta commands Ability to interact with metadata at the command line Already installed as part of iCommands Documentation is a little wimpy Try Here and HereHere

Tips At the moment, metadata added via template is not available from the command line

Data, Project, Research Management In the Data Commons Tools and for sharing, managing, and publishing data A home for high-value public dataset to be used with iPlant analysis tools A way to manage projects Metadata templates and workflows for common analysis types Working hard to lay the groundwork, with development starting early 2015

Data, Project, Research Management In the Data Commons Additional layer on top of the Data Store for Users to publish packages of data and metadata to the Data Commons and supported external repositories with appropriate long-term identifiers and licenses Data will be static, searchable, discoverable, and linked to external repositories Based on Data Strategy, current CI, Developer input, recognized need for additional components to support Data Commons effort

Data Commons Development Plans

Goal: Data models and workflows for the entire data lifecycle specimen collection analysis project creation publication

Interface Mockups are in Review Demo 3 – Data Commons Interface Mockups

Metadata Templates for G2F Open discussion now that we are all updated on G2F and iPlant infrastructure Project Interface needs for G2F? Metadata needs for G2F? Other needs? Perhaps start here?here

Thanks! iPlant Learning Center provides quick tutorials, slides, and documentation iPlant Learning Center