MDTM Application Implementation (Discussion Topics) 1

Topic 1: Overall BBCP Changes Control agent’s function Interface with the data transfer modules How to create and manage control/data channels? How to create multiple threads (readers/writers, senders/receivers)? Synchronization and buffer management, locking, performance issues 2

Topic 2: Preprocessing Module Implementation Preprocessing: how many threads, sequential processing? What kind of layout info we can get from the MDTM? Metadata access for different file/storage systems. Example: local disks, how to get the location info? Should we consider load balancing with large groups? How? How to represent the groups with data structure, and the result of preprocessing? 3

Topic 3: Thread/flow Management Implementation How many threads for each group? – Reader -> sender – Receiver -> writer Load balancing consideration here: large groups might be divided into small groups, and may utilize remote resources The information passed to MDTM for binding/scheduling What is the results of return from the calls to MDTM, and what is the general steps for our application program? 4

Topic 4: Data Access/Transmission Implementation Synchronization among all threads for accessing common resources/data Implementation of communication between storage/network threads – IPC, locks, buffer size (and number) Direct I/O support in various Linux (and other systems) NUMA aware cache, and bypass page cache, any technical difficulties? Cache management organization, algorithm and policy issues, performance from previous tests Sendfile/splice support in Linux – buffer to network, any synchronization issues too? 5