EUROPEAN UNION Polish Infrastructure for Supporting Computational Science in the European Research Space Towards scalable, semantic-based virtualized storage resources provisioning Kornel Skałkowski, Renata Słota, Dariusz Król, Michał Orzechowski, Bartosz Kryza, Jacek Kitowski ACC Cyfronet AGH, Krakow, Poland KU KDM 2012 : fifth ACC Cyfronet AGH users' conference : Zakopane, March 07–09, 2012

2 Outline  Introduction  The QStorMan toolkit overview  The QStorMan toolkit architecture  QStorMan usage  Recent improvements  Current status of QStorMan  Test results  Future Work

Introduction  Data intensive applications and the 4 th science paradigm  Resources virtualization becomes ubiquitous  Storage resources virtualization is often provided by cluster file systems like Lustre  IT infrastructure users expect more and more computing and storage power as well as an appropriate QoS level 3

The QStorMan toolkit  Main goal is to provide virtualized storage resources with QoS warrianties for data intensive applications  Users can define QoS requirements concerning storage resources on three levels: application, user, virtual organization  Currently we support the following non-functional requirements:  Average Read/Write transfer rate,  Current Read/Write transfer rate,  Free capacity,  Result cachability – dedicated for application, which generates a large number of small files.  The toolkit consists of three components:  Knowledge base (GOM) which stores semantic descriptions concerning storage resources and synchronizes the descriptions with a grid middleware  Dedicated monitoring service (SMED) which performs continuous, real-time monitoring of virtualized storage resources with semantic support  Intelligent resources matching service (SES) which combines information obtained from the GOM and SMED services as well as advanced semantic support in order to perfectly match a virtualized resource from the resources mesh 4

The QStorMan toolkit architecture 5

QStorMan usage  Using system C library (libses-wrapper):  declare your non-functional requirements in the GOM knowledge base  export LD_PRELOAD= 2. Using C++ programming library (libses): #include using namespace lustre_api_library; LustreManager manager; StoragePolicy policy; policy.setAverageReadTransferRate(50); policy.setCapacity(100); int descriptor = manager.createFile(„nazwa_pliku.dat”, &policy); 6

Recent improvements  General purpose of the improvements is to provide a scalable, fully semantic-based solution for efficient provisioning of virtualized storage resources  SMED improvements:  Utilization of the enhanced C2MS storage resources semantic model for description of high-level QoS parameters  Application of semanatic reasoners on the monitoring level  SES improvements  Cache mechanism on demand – supporting large number of files generation  Automatic registration of users in knowledge base – decrease required administration effort  GOM improvements  Security enhancements  Scripts for administration 7

The QStorMan toolkit current status  Test installation is running at ACC Cyfronet AGH from over 1 year now  A lot of tests were performed and no major bugs were found  We have passed operational and security audits in PL-Grid succesfully  We now waiting for official deployment in ACC Cyfronet, PCSS Poznan, TASK Gdansk, and ICM Warsaw  Official tutorials, workshops and other material are on the way  Integrated with QoSCosGrid middleware from PCSS  We are willing to cooperate with anyone, who would like to test QStorMan in practice with an exisiting data intensive application 8

Test description Synthetic test  The toolkit evaluation was performed by simulation of 8 users which were executing their applications on the Grid infrastructure  3 users used the QStorMan toolkit during the applications execution, the others used plain Lustre file system  Every user periodically saved and read a 60 GB file with random sleep periods between the succeeding operations (10 reads and 10 writes)  Users started their applications with random delays in order to simulate real conditions in a Grid environment Test with real user’s application  Simulation of sound wave propagation inside human head  Out-of-core computations  No source code modifications  5 instances of application running in parallel in order to generate enough load for storage system 9

Synthethic test results 10 12% speedup between two fastest applications 26% speedup on average (~7:20 h vs ~10 h) No source code modification

Real user’s application test result 11 15% speedup on average Running on production infrastructure No source code modification

Future work  Support for domain-oriented virtualized computing environments  Implementation of new storage resources selection strategies  Orientation toward Cloud computing environments  Dissemination and exploitation among possible users 12

Questions? 13