NeST: Network Storage Flexible Commodity Storage Appliances

NeST: Network Storage Flexible, commodity based, software-only storage appliances Building a NeST should be as easy as: Finding a networked machine “Dropping” some software on it Self-configuring to best utilize machine

NeST resource management Dynamic user account creation provides storage for migratory grid users Storage reservations and quota systems intelligent scheduling of data intensive applications Matchmaking match storage opportunities & storage consumers

Condor NeSTs Better, smarter checkpoint servers Checkpoints are just another data file Data replicated/migrates to different NeSTs Condor jobs access data from closest NeST Flexible policy support for managing disk and network resources

New worlds, New problems Diverse hardware, software platforms - Netapp, EMC advantage fewer platforms, control over OS - Our approach Automate configuration to each host system H/W (Disks): use file system or self-manage S/W (File System): use either read/write or mmap Implication: Flexibility at a cost? Key is design of the software

NeST structure Modules for communication, transfer & storage Protocol layer Pluggable protocols allow diverse protocols to be mapped into common control flows Transfer layer Different concurrency architectures to maximize system throughput on diverse platforms Storage layer Provides abstract interface to disks

Concurrency Architecture NeST Structure Protocol Layer GFTP NeST WiND HTTP NFS Control Logic Concurrency Architecture Multi-process Multi-threaded Nonblocking Storage Layer Raw disk Local FS RAID Memory

Many Protocols, Single Server Single point of control - Storage quotas/guarantees can be supported - Bandwidth can be controlled & QoS provided Single administrative interface Set policies, Manage user accounts Maintainable S/W Shared code base reduces replication, increases maintainability

Protocol layer implementation Each protocol listens on well-defined port Central control accepts connections Protocol layer reads from connection and returns generic request object Analagous to Linux V-nodes Add new protocol by writing a couple of methods

Virtual protocol class class nestProtocol { public: virtual nestRequest* receiveRequest( fd_set *ready ) = 0; virtual ssize_t Receive( char *buffer, ssize_t n )= 0; virtual ssize_t Send( char *buffer, ssize_t n ) = 0; virtual bool sendAck( ) = 0; virtual bool sendErrorMessage( int error ) = 0; virtual bool sendDirectoryList( NestFileInfo *list ); virtual bool sendPwd( char *pwd ); virtual bool sendTransferInfo( class nestRequest *req ); virtual bool sendFilesize( int retval, long size) = 0; virtual bool sendQuotaInfo( struct UserUsage* usage ); virtual bool hasSocket() = 0; virtual int getSocket() = 0; };

Virtual protocol class Plus some static methods static int listen( int port ); static nestProtocol* accept( int socket, sockaddr_in *address ); static NestReplyStatus initiateThirdParty( nestClient *client, nestProtocol **connection );

Grid FTP virtual protocol Good news? Hard work already done Already implemented libraries have support for parallel streams, authentication, etc. Bad news? Slight mismatch to integrate callback model into synchronous API

Grid FTP - listen if ( Spipe( pipes ) < 0 ) { goto ERROR; } // static initialization routine int nestProtocolGridFtp::listen( int port ) { globus_result_t result; globus_ftp_control_server_t *server; globus_module_activate( GLOBUS_FTP_CONTROL_MODULE ); server = (globus_ftp_control_server_t *) Malloc( sizeof( struct globus_ftp_control_server_s ) ); result = globus_ftp_control_server_handle_init( server ); if ( result != GLOBUS_SUCCESS ) { goto ERROR; } short unsigned Port = (short unsigned) port; result = globus_ftp_control_server_listen( server, &Port, listenCallback, NULL ); if ( result != GLOBUS_SUCCESS ) { goto ERROR; } if ( Spipe( pipes ) < 0 ) { goto ERROR; } }

Grid FTP - receiveRequest nestRequest* nestProtocolGridFtp::receiveRequest( fd_set *ready ){ // ignore the fd_set char msg[MESSAGESIZE]; snprintf( msg, MESSAGESIZE, "%s", "Gftp receiveRequest called: " ); nestRequest *req = this->request; if ( req != NULL ) { snprintf( &(msg[strlen(msg)]), MESSAGESIZE - strlen(msg), "%s\n", "Actually have a request to give.\n" ); } else { "No request to give.\n" ); } nest_debug( 1, msg ); return req;

Grid FTP - sendFilesize bool nestProtocolGridFtp::sendFilesize( int retval, long size){ assert( reqType == FILESIZE_REQUEST ); switch( retval ) { case NEST_SUCCESS: snprintf( buffer, FILEBUFFER, "213 %ld\r\n", size ); break; case NEST_REMOTE_FILE_NOT_FOUND: snprintf( buffer, FILEBUFFER, "550 File not found\r\n" ); default: assert( 0 ); } return sendMessage( buffer );

Grid FTP - sendMessage bool nestProtocolGridFtp::sendMessage( const void *vptr ) { globus_result_t result; waitIdle(); setWait( true ); result = globus_ftp_control_send_response( handle, (char *)vptr, commandCallback, this ); if ( result != GLOBUS_SUCCESS ) { fprintf( stderr, "globus_ftp_control_send_response error: %s\n", resultToString( result ) ); free( handle ); return false; } return true;

Grid FTP - commandCallback void nestProtocolGridFtp::commandCallback( void * arg, struct globus_ftp_control_handle_s * handle, globus_object_t * error) { nest_debug( 1, "command_callback called - set wait false\n" ); if( error != GLOBUS_SUCCESS ) { fprintf(stderr,">> command_callback: error %s\n", globus_object_printable_to_string( error ) ); return; } // set the wait flag to false nestProtocolGridFtp *connection = (nestProtocolGridFtp *)arg; connection->setWait( false );

Concurrency architecture Three difficult goals Low latency High bandwidth Multiple simultaneous clients No single portable solution Multiple models provide solutions on a range of different platforms Multi-threaded, multi-process, single process nonblocking

Concurrency architecture Flexible Concurrency Concurrency architecture Nonblocking Multi-process Multi-threaded Control logic creates transfer object Virtual connection from the protocol layer Virtual connection from the storage layer Transfer object passed to concurrency layer Control logic informed when transfer ends

Concurrency Models Nonblocking model Pre-allocated pool of processes Performs only non-blocking I/O Selects on file-descriptors / sockets from each transfer object Pre-allocated pool of processes Descriptors are passed over a pipe Transfer object recreated on other side Each process does a blocking transfer Pre-allocated pool of threads Transfer object enqueued by server Dequeued by an available thread

Concurrency Models and GFTP Nonblocking model Not yet supported. Grid FTP libraries do not expose socket on which to select. Pre-allocated pool of processes Haven’t figured out how to send a GridFTP connection over a pipe Pre-allocated pool of threads No problem. Fully integrated. Other models could work . . .

Storage Layer Three needed areas of flexibility File systems interfaces Example: read()/write() or mmap() Abstract storage models RAID, JBOD, etc. Memory storage model also a possibility Provide file system interface to remote memory Useful for buffering systems like Kangaroo Virtual storage model akin to virtual protocol layer User account administration Creation and removal Quotas and guarantees for users and groups