1. NeST: Network Storage Flexible Commodity Storage Appliances

2. NeST: Network Storage
Flexible, commodity based, software-only storage appliances
Building a NeST should be as easy as:
Finding a networked machine “Dropping” some software on it Self-configuring to best utilize machine

3. NeST resource management
Dynamic user account creation
provides storage for migratory grid users
Storage reservations and quota systems
intelligent scheduling of data intensive applications
Matchmaking
match storage opportunities & storage consumers

4. Condor NeSTs Better, smarter checkpoint servers
Checkpoints are just another data file
Data replicated/migrates to different NeSTs
Condor jobs access data from closest NeST
Flexible policy support for managing disk and network resources

5. New worlds, New problems
Diverse hardware, software platforms
- Netapp, EMC advantage
fewer platforms, control over OS
- Our approach
Automate configuration to each host system
H/W (Disks): use file system or self-manage
S/W (File System): use either read/write or mmap
Implication: Flexibility at a cost?
Key is design of the software

6. NeST structure Modules for communication, transfer & storage
Protocol layer
Pluggable protocols allow diverse protocols to be mapped into common control flows
Transfer layer
Different concurrency architectures to maximize system throughput on diverse platforms
Storage layer
Provides abstract interface to disks

7. Concurrency Architecture
NeST Structure
Protocol Layer
GFTP
NeST
WiND
HTTP
NFS
Control
Logic
Concurrency Architecture
Multi-process
Multi-threaded
Nonblocking
Storage Layer
Raw disk
Local FS
RAID
Memory

8. Many Protocols, Single Server
Single point of control
- Storage quotas/guarantees can be supported
- Bandwidth can be controlled & QoS provided
Single administrative interface
Set policies, Manage user accounts
Maintainable S/W
Shared code base reduces replication, increases maintainability

9. Protocol layer implementation
Each protocol listens on well-defined port
Central control accepts connections
Protocol layer reads from connection and returns generic request object
Analagous to Linux V-nodes
Add new protocol by writing a couple of methods

10. Virtual protocol class
class nestProtocol {
public:
virtual nestRequest* receiveRequest( fd_set *ready ) = 0;
virtual ssize_t Receive( char *buffer, ssize_t n )= 0;
virtual ssize_t Send( char *buffer, ssize_t n ) = 0;
virtual bool sendAck( ) = 0;
virtual bool sendErrorMessage( int error ) = 0;
virtual bool sendDirectoryList( NestFileInfo *list );
virtual bool sendPwd( char *pwd );
virtual bool sendTransferInfo( class nestRequest *req );
virtual bool sendFilesize( int retval, long size) = 0;
virtual bool sendQuotaInfo( struct UserUsage* usage );
virtual bool hasSocket() = 0;
virtual int getSocket() = 0; };

11. Virtual protocol class
Plus some static methods
static int listen( int port );
static nestProtocol* accept( int socket, sockaddr_in *address );
static NestReplyStatus initiateThirdParty( nestClient *client,
nestProtocol **connection );

12. Grid FTP virtual protocol
Good news?
Hard work already done
Already implemented libraries have support for parallel streams, authentication, etc.
Bad news?
Slight mismatch to integrate callback model into synchronous API

13. Grid FTP - listen if ( Spipe( pipes ) < 0 ) { goto ERROR; }
// static initialization routine
int nestProtocolGridFtp::listen( int port ) {
globus_result_t result;
globus_ftp_control_server_t *server;
globus_module_activate( GLOBUS_FTP_CONTROL_MODULE );
server = (globus_ftp_control_server_t *)
Malloc( sizeof( struct globus_ftp_control_server_s ) );
result = globus_ftp_control_server_handle_init( server );
if ( result != GLOBUS_SUCCESS ) { goto ERROR; }
short unsigned Port = (short unsigned) port;
result = globus_ftp_control_server_listen( server, &Port, listenCallback, NULL );
if ( result != GLOBUS_SUCCESS ) { goto ERROR; }
if ( Spipe( pipes ) < 0 ) { goto ERROR; } }

14. Grid FTP - receiveRequest
nestRequest* nestProtocolGridFtp::receiveRequest( fd_set *ready ){
// ignore the fd_set
char msg[MESSAGESIZE];
snprintf( msg, MESSAGESIZE, "%s", "Gftp receiveRequest called: " );
nestRequest *req = this->request;
if ( req != NULL ) {
snprintf( &(msg[strlen(msg)]), MESSAGESIZE - strlen(msg), "%s\n",
"Actually have a request to give.\n" );
} else {
"No request to give.\n" ); }
nest_debug( 1, msg );
return req;

15. Grid FTP - sendFilesize
bool nestProtocolGridFtp::sendFilesize( int retval, long size){
assert( reqType == FILESIZE_REQUEST );
switch( retval ) {
case NEST_SUCCESS:
snprintf( buffer, FILEBUFFER, "213 %ld\r\n", size );
break;
case NEST_REMOTE_FILE_NOT_FOUND:
snprintf( buffer, FILEBUFFER, "550 File not found\r\n" );
default:
assert( 0 ); }
return sendMessage( buffer );

16. Grid FTP - sendMessage
bool nestProtocolGridFtp::sendMessage( const void *vptr ) {
globus_result_t result;
waitIdle();
setWait( true );
result = globus_ftp_control_send_response(
handle, (char *)vptr, commandCallback, this );
if ( result != GLOBUS_SUCCESS ) {
fprintf( stderr, "globus_ftp_control_send_response error: %s\n",
resultToString( result ) );
free( handle );
return false; }
return true;

17. Grid FTP - commandCallback
void nestProtocolGridFtp::commandCallback( void * arg,
struct globus_ftp_control_handle_s * handle,
globus_object_t * error) {
nest_debug( 1, "command_callback called - set wait false\n" );
if( error != GLOBUS_SUCCESS ) {
fprintf(stderr,">> command_callback: error %s\n",
globus_object_printable_to_string( error ) );
return; }
// set the wait flag to false
nestProtocolGridFtp *connection = (nestProtocolGridFtp *)arg;
connection->setWait( false );

18. Concurrency architecture
Three difficult goals
Low latency
High bandwidth
Multiple simultaneous clients
No single portable solution
Multiple models provide solutions on a range of different platforms
Multi-threaded, multi-process, single process nonblocking

19. Concurrency architecture
Flexible Concurrency
Concurrency architecture
Nonblocking
Multi-process
Multi-threaded
Control logic creates transfer object
Virtual connection from the protocol layer
Virtual connection from the storage layer
Transfer object passed to concurrency layer
Control logic informed when transfer ends

20. Concurrency Models Nonblocking model Pre-allocated pool of processes
Performs only non-blocking I/O
Selects on file-descriptors / sockets from each transfer object
Pre-allocated pool of processes
Descriptors are passed over a pipe
Transfer object recreated on other side
Each process does a blocking transfer
Pre-allocated pool of threads
Transfer object enqueued by server
Dequeued by an available thread

21. Concurrency Models and GFTP
Nonblocking model
Not yet supported.
Grid FTP libraries do not expose socket on which to select.
Pre-allocated pool of processes
Haven’t figured out how to send a GridFTP connection over a pipe
Pre-allocated pool of threads
No problem. Fully integrated.
Other models could work . . .

22. Storage Layer Three needed areas of flexibility
File systems interfaces
Example: read()/write() or mmap()
Abstract storage models
RAID, JBOD, etc.
Memory storage model also a possibility
Provide file system interface to remote memory
Useful for buffering systems like Kangaroo
Virtual storage model akin to virtual protocol layer
User account administration
Creation and removal
Quotas and guarantees for users and groups