Working Items and plan Hideki Miyake (KEK) June 30 th, 2014 KISTI, Korea

The list shown in May AMGA topics related to Belle II - replication doesn't work - to avoid table collisions - stuck due to deep directory structure (below /belle/user/wenjing) - too many sessions for one AMGA operation - due to disconnect() - Can GSI socket keep sessions? --> significant improve - Can increase number of FE process (i.e. amgad) - connection pool? - Can decrease number of BE process (e.g. AMGA threading mode)? - Should check current I/O load which limits BE performance - performance against accesses during very short period - an API to scan AMGA directories - AMGA server was down by memory piled up - worry about future massive accesses - performance estimation - Initial goal: 10Hz (>30K jobs/1h duration) writing+deleting? - Should estimate more reasonable goal (production+analysis) - 1 second access time per directory is not acceptable - any idea to reduce access time? - synchronization between FC + metadata - relevant to AMGA survival model (LFC+AMGA, DFC+AMGA...) - future support - major concern from Belle II collaborators - AMGA development plan independent from Belle II? - AMGA development plan dedicated for Belle II? - activity invisibility from Belle II 2 Recent Belle II topics2014/6/30

The list shown in May AMGA topics related to Belle II - replication doesn't work - to avoid table collisions - stuck due to deep directory structure (below /belle/user/wenjing) - too many sessions for one AMGA operation - due to disconnect() - Can GSI socket keep sessions? --> significant improve - Can increase number of FE process (i.e. amgad) - connection pool? - Can decrease number of BE process (e.g. AMGA threading mode)? - Should check current I/O load which limits BE performance - performance against accesses during very short period - an API to scan AMGA directories - AMGA server was down by memory piled up - worry about future massive accesses - performance estimation - Initial goal: 10Hz (>30K jobs/1h duration) writing+deleting? - Should estimate more reasonable goal (production+analysis) - 1 second access time per directory is not acceptable - any idea to reduce access time? - synchronization between FC + metadata - relevant to AMGA survival model (LFC+AMGA, DFC+AMGA...) - future support - major concern from Belle II collaborators - AMGA development plan independent from Belle II? - AMGA development plan dedicated for Belle II? - activity invisibility from Belle II 3 Recent Belle II topics2014/6/30 Evaluated Good sign! Agreed tentative goal 100 operations/sec for both reading and writing

Proposal for this week First of all we should classify the items for short term (in this week) and others Short and easy items should be solved during the week –Want to have session to work together –Show our issue –Tackle the issue For the others we should determine rough plan and milestone (which should be shown in any Belle II meeting) 4 Recent Belle II topics2014/6/30

Memo Writing Limited number of concurrent access GSI patch significantly improved the performance since # session is dramatically decreased as expected (can drop disconnect()!!) Scalability issue still exists (# jobs  100K) AMGA proxy (DIRAC service) AMGA threading (multiple AMGA vs one DB) Access speed (O(1)) Reading Access speed (O(1)) for one directory  request two order faster Tricky API to read the directory

Memo Metadata Practical use case Attribute migration? Backend robustness Connection pool? Possibility of DFC+AMGA proposal What is advantage compared with DFC

Memo Development plan for AMGA server Development plan for AMGA client Tuning suggestion for PGSQL Possibility of MySQL? AMGA separation (federation) /belle/data /belle/MC /belle/user (or all others)

Memo Replication failed due to complicated directory Correct procedure Modification? Multi layered dataset Possible? Any suggestion

Open Question The lowest cost to check connectivity? “whoami” looks a bit complicated… Simple way like ping? Variable array? Or fixed array example? Best tuning parameter for PostgreSQL Backend possibility of MySQL? DB expansion (when records exceed disk size) 2014/6/30Recent Belle II topics 9

Backend benchmark Check transaction performance (not AMGA but backend DB) pgbench –t 5 (average of 5 measurements) Recent Belle II topics 10 Peak processing power < 100 connections? Most important thing is to handle a few thousands connections without error But performance optimization should be also important Is there room to optimize? Need your help! Limited by DISK I/O? Transaction / sec (TPS) # of clients Backend benchmark 2014/6/30

Possible solutions Recent Belle II topics 11 AMGApgpool DB1 DB2 replication AMGA1 AMGA2 - Connection pooling + multiple backends - AMGA federation DB1 DB2 -DB1 and DB2 are independent pooling >1000 ~100 /belle/MC /belle/data 2014/6/30