Virtual Global File System

Background & Short Story Launched in 2011 as a skunkworks project at Cambridge Computer Spun out into a separate company in January, 2014 $5M investment Inspired by SRB and IRODS as well as home grown software by our many clients 100% original code, built from the ground up by Starfish Storage Early customers (running like a managed service by our R&D team) Harvard Med School US Library of Congress

2016 – Moving Toward GA Release The software has been running in production for approximately 5 years. Foundation for managed services Archiving Fixity checking Other file systems rules automation Professional services tool File systems analysis Data migrations We are working aggressively toward a GA release But in the meantime, the product is very usable Get in on our early adopter program!

Technical Leadership Jacob Farmer, Founder and Chief Evangelist CTO for Cambridge Computer Storage industry veteran, 30 years’ experience Don Preuss, CTO for Starfish Storage Former CTO of NIH (US National Institutes of Health) Formerly head of systems at the NIH National Center for Biotechnology Information (NCBI) at the National Library of Medicine. Pubmed dbGaP SRA

Target Markets Research Computing Digital Libraries Policy-based management of all files across a large, diverse institution. Capture metadata throughout scientific pipeline Curate files in place It is not necessary to transfer files to an archival storage facility Digital Libraries File system middle-ware for automating the housekeeping of digital collections. Enterprise Computing Storage management for NAS and large-scale file systems

The Grand Vision Manage the Lifecycle of Institution-scale File Collections Publication/Preservation (Librarians, Archivists, Curators) Open Links / DOIs Metadata Extraction Curation Workflows Version Controls Access Controls Fixity Checks Content Creation (Scientists, Engineers, Artists) Metadata Tagging Workflow Automation Data Management Plans Open Access Data Reusability Collaboration IT Operations (Storage & Backup Administrators, IT Governance) Data Movement Tiered Storage Backup Restore Data Migration Governance Permissions Management Auditing Chargeback / Show-back Reporting Capacity Planning Aging / Utilization File System Analysis

What Starfish Does And How it Works Modular Architecture Designed to Enable Highly Customized File-based Workflows Also Designed to be Useful Right out of the Box

Starfish is Made Up of 3 Main Components File System Catalog Syncs the metadata of your file systems to a database Allows you to assign additional metadata to files and directories via API Jobs Manager Runs batch jobs that act on your files Enforces policies Access Gateways Provides alternative methods for addressing and accessing files Allows programmatic manipulation of access controls

The File System Catalog Starfish scans your file systems and make a database that reflects the metadata contents of your file systems. You can add additional metadata (via API) that better describe the files and directories.

What Makes the File System Catalog Awesome? Massively scalable Handles billions of files Multi-threaded and multi-host for greater parallelization Highly tunable and configurable Agents for specific file systems Agents capture file system events reducing the need to crawl and compare Agents capture device-specific metadata Versioning We track version changes in the file system The next release of our catalog supports directory tree versioning Navigate the file system tree throughout time Reconcile path names, even if the directory has been moved or renamed

What Can You Do With Metadata? Generating more meaningful reports Utilization, aging, trending, chargeback/ show-back Inventory management for your files Content classification Deeper understanding of file system contents Rules-based administration Backup, archiving, publishing, processing Addressing and retrieval A virtual global namespace Search Tracking provenance Facilitate Collaboration Define meaningful subsets of files and directories Enable more purposeful access controls (compared with LDAP/POSIX/ACLs)

Where Does the Metadata Come From? Metadata can be added via the API Metadata can be inherited from a parent directory A job could be executed that extracts metadata from a file and updates the database via API Examples Reading tags from a TIFF file Extracting CODECs from video files An externally defined workflow might make API calls to add metadata to the catalog as files are created and manipulated. Worst case, metadata can be data entered through a GUI

The Jobs Manager

The Jobs Manager Jobs work by querying the database on an ad hoc or scheduled basis. Query results go into a queue Agents take work out of the queue if they can mount the file systems in question You can have any number of agents They are easy to install. The essential components discover each other automatically They can run on dedicated hardware or virtual machines

What Kinds of Jobs Can You Run? Copy-Move-Delete With versioning enabled a copy command is just like enterprise backup, except much more powerful and flexible. To and from POSIX file systems and Object / Cloud Stores Calculate hashes Fixity checking (data integrity checking) Duplicate file detection Content addressing (address files by their hashes) Metadata extraction Format conversion Anything you can imagine Refactor your batch scripts as Starfish commands

Starfish System Topology

Out-of-band for Live File Systems In-band for backup and Archive Confidential

A “Virtual” Global File System

Common Use Cases For Digital Libraries Backup and Restore for Large File Systems Granular control of backup policies Ensure data integrity with hash compares Direct files to appropriate backup target based on metadata Tiered Storage Migrate files to archival tiers while maintaining namespace Reporting Utilization, Aging, Duplicate File Access controls Fixity Checking Metadata Extraction

Bragging Rights The software typically installs in less than 10 minutes. The major components discover themselves. We can perform an initial scan of a billion files in typically less than a day. Our largest single installation has over 8 billion file system objects. The upgrade process is invoked by a single command from CLI. We have over 25 sites using the software as of October, 2016. Most are top tier data centers and/or household names