Unified Model I/O Server : Recent Developments and Lessons Paul Selwood © Crown copyright Met Office

Recent Developments Paul Selwood © Crown copyright Met Office

Increasing Server Parallelism Original Server Parallelism Over units/files Course grain load spreading Units are allowed to ‘hop’ servers based on current load balance of servers Uses a second ‘priority’ protocol queue for fast enquiry operations Hints are provided with file_open() calls to signal that future writes to the unit are dependency from prior ones © Crown copyright Met Office

Parallelism Within Server North-South domain decomposition Not model decomposition Maps N/S model subdomains to servers Pro – reduced message count Con – load imbalance Allows packing parallelism Multiplies effective server FIFO queue size Provides a communication hierarchy © Crown copyright Met Office

Subtlety: Temporal Reliability The operational environment needs to know what's going on But I/O is running in arrears Messages to trigger archiving and pre-processing Model state consistency (which is my last known good state?) Introduce a new generic ‘epoch’ pseudo operation Server owning the epoch must wait until all other servers have processed the epoch – effectively a barrier. Provides limited temporal ordering between independent operations where needed at the expense of stalling one server. © Crown copyright Met Office

Subtlety: Temporal Reliability 4 1 2 3 Existing work Checkpoint Complete! Epoch marker (owner is 4) Temporally critical operation New work Server 4 waiting for epoch on 1 & 2 © Crown copyright Met Office

Now Use Quadruple Buffering! Buffer 1: Aggregation over repeated requests to the same unit Evolved from aggregation over levels in a 3D field Trade off: message counts, bandwidth, latency, utilisation Buffer 2: Client side dispatch queue Allows multiple in flight transactions with server. Insulates against comms latency and a busy server. Buffer 3: I/O Server FIFO Used to facilitate hardware comms offload Buffer 4: Aggregation of write data into well formed I/O blocks Filesystem tuneable © Crown copyright Met Office

Lots of implementation … All deterministic global configurations UKV – 1.5km model (24 node) EURO 4km model (32 node) All ensemble configurations using > 4 nodes Atmosphere-only climate runs on > 4 nodes © Crown copyright Met Office

Reducing Messaging Pressure © Crown copyright Met Office

I/O in ESM OASIS Coupler NEMO Ocean CICE Sea Ice JULES Land Surface UKCA Chemistry UM Atmosphere OASIS Coupler UM I/O Server XIOS NEMO Ocean CICE Sea Ice © Crown copyright Met Office

Deployment Nightmare Component model decomposition tunings and restrictions I/O server tuning Coupled system load-balancing Mapping onto nodes Iterative process potentially using lots of compute time © Crown copyright Met Office

Input “helper threads” Can we use a spare thread to help read costs? Initial conditions, Boundary Conditions, Forcings, … Thread reads blocks ahead of requests Tuneable buffer sizes and number of blocks to read ahead Easier to implement on a new read layer with intrinsic buffering design Initial attempt implemented on I/O server © Crown copyright Met Office

CPU 0 I/O Server Helper Read Block 1 Read Req. Deliver Read Block 2 Scatter Deliver Read Block 2 Read Block 3 Deliver Deliver © Crown copyright Met Office

Read and Distribute Time (s) Apparent Read Rate (MB/s) Read Helper Results N512 forecast model start + timestep 0 incremental analysis update on 16 nodes of p775, 4 threads/task Read and Distribute Time (s) ApparentRead Time (s) Apparent Read Rate (MB/s) Original I/O layer 21.4 13.7 927 New layer, Input on I/O Server, High Polling delay 19.6 3.9 3143 New layer, Input on I/O Server,Low Polling Delay 16.6 4.5 2710 New layer, no I/O Server or helper thread 13.9 5.3 2288 © Crown copyright Met Office

So what is happening? Helper thread is working – read rates are good Latencies between I/O server and model are killing performance Same for larger data sets? Can we aggregate levels to reduce latencies? Helper thread on model cores? New I/O level is an improvement anyway! © Crown copyright Met Office

Lessons Paul Selwood © Crown copyright Met Office

MPI Issues and Opportunities Threading support Need MPI_THREAD_SERIALIZED, prefer MPI_THREAD_MULTIPLE iGatherv Data transfer currently via iSend/iRecv Reduce message rate pressure Message Priority Want QoS features to prioritise messages Available at system level for many IB implementations MPI 3 communicator hints? © Crown copyright Met Office

OpenMP Issues Subtle failures with FLUSH Thread polling load with SMT / HT (backoff) How to use > 2 threads when atmosphere does Packing? Nested OpenMP support uneven. Don’t. Awkward deployment. Input helper thread? © Crown copyright Met Office

Other Issues Haven’t fully appreciated bottlenecks until partial system in place and scale is pushed Memory accounting is important and easy to get wrong Cost overheads exist Message contention NUMA effects Fully asynchronous processes difficult to debug Complex systems get (wrongly) blamed for any failures! © Crown copyright Met Office

Lots of tuneable parameters… Number and spacing of I/O servers Core capability and mapping to hardware Buffering Choices; Where to consume memory FIFO for I/O servers, Client queue size, Aggregation level Platform characteristics, MPI characteristics, output pattern Load balancing options Timing tunings Thread back offs, to avoid spin loops on SMT/HT chips Standard disk I/O tunings (write block size) etc Profiling: lock metering, loading logs, transaction logs, comms timers © Crown copyright Met Office

Server FIFO queues frequently hit max capacity … but how to choose? Fast: 4x14 Slow: 4x2 Time vs. Time step Straight line indicates no impact of model evolution from IO intensive steps Model stalls on IO intensive steps, Servers trailing by up to 600 seconds. Backlog vs. Times Server FIFO queues frequently hit max capacity © Crown copyright Met Office

Results © Crown copyright Met Office

What to measure? In practise we measure “Stall Time” Overall model runtimes Tend not to worry about apparent I/O rates etc Minimising stall time ensures process is as asynchronous as we can get it. © Crown copyright Met Office

Results (PS27) Config Performance (relative No-IOS) P575 P775 1st Operational configuration 8 serial I/O servers 760 Atmosphere tasks (20x38) QU06 Global model (PS27 Code level) 120GB output total 4 x 10.6GB restart files, 25 files total Good performance on POWER6 Migration to POWER7 slows down Atmospheric model faster Exposes previously completely hidden packing time Config Performance (relative No-IOS) P575 P775 IO Disabled 42% No I/O Server 100% Proxy I/O Server 90% Asynchronous I/O Server 73% 154% © Crown copyright Met Office

Results (Parallel Servers) Same QU06 model @24 Nodes on P775. Parallel servers remove the packing blockage For the same floor space, more CPUs allocated to IO gives better performance Moving to 26 Nodes is a super linear speedup. Insignificant observable I/O time mid-run, 20 seconds at run termination whilst the final I/O discharges Config Performance (relative No-IOS) Original 20x38 atmosphere grid 8x1 154% 4x2 114% 2x4 77% 1x8 84% Smaller 22x34 atmosphere grid 10x2 85% 5x4 58% 4x5 63% 2x10 55% © Crown copyright Met Office

Results (N768 ENDGAME) Potential future global model 96 Nodes, 6144 threads 2.25x more grid points Untuned first run! Use 4 servers to reflect rough breakdown of files Config Performance (relative No-IOS) 4x1 <timeout> 4x2 130% 4x4 75% 4x8 62% 4x14 60% © Crown copyright Met Office

Questions and answers © Crown copyright Met Office