Scaling up R computation with high performance computing resources

+ =

What to do if the computation is too big for a single desktop A common user question: –I have an existing R solution for my research work. But the data is growing to big. Now my R program runs days to finish (/runs out of memory) Three strategies –Using automatically offloading with multicore/GPU/MIC. –Break big computation with multiple job submission –Implement code using parallel packages.

Hardware Acceleration with Computation Offloading Hardware supported: –Multiple cores on CPU –Intel Xeon Phi coprocessor (on Stampede) –GPGPU (on Stampede/Maverick) Libraries supporting automatic offloading –Intel Math Kernel Library (MKL) Available on stampede and maverick for users –HiPlarB Open source and freely available

MIC and MKL MKL provides BLAS/LAPACK routines that can “offload” to the Xeon Phi Coprocessor, reducing total time to solution User R script/function R interpreter Code execution with pre-built library MKL BLAS

MKL-MIC: Submission script to use MIC #!/bin/bash #SBATCH -J benchmark25-R #SBATCH -o slurm.out%j #SBATCH -p vis #SBATCH -t 01:30:00 #SBATCH -A TACC-DIC #SBATCH -e slurm.err #SBATCH -N 1 #SBATCH -n 10 #set environment module purge module load TACC module load cuda module load intel/ module load Rstats # enable mkl mic offloading export MKL_MIC_ENABLE=1 # from 0 to 1 the work division export MKL_HOST_WORKDIVISION=0.3 export MKL_MIC_WORKDIVISION=0.7 # make the offload report big to be visible export OFFLOAD_REPORT=2 # set the number of threads on host export OMP_NUM_THREADS=16 export MKL_NUM_THREADS=16 # set the number of threads on MIC export MIC_OMP_NUM_THREADS=240 export MIC_MKL_NUM_THREADS=240 # Run a script Rscript r-benchmark-25-MIC.R

HiPLAR HiPLAR (High Performance Linear Algebra in R) use the latest multi-core and GPU libraries to give substantial speed-ups to existing linear algebra functions in R. Two projects that aims to achieve high performance and portability across a wide range of multi-core architectures and hybrid systems. Parallel Linear Algebra for Scalable Multi-core Architectures (PLASMA) Matrix Algebra on GPU and Multi-cores Architectures (MAGMA ) Need additional package installation

HiPLAR: example using regular matrix multiplication A <- matrix(rnorm(2048 * 2048), nrow=2048, ncol=2048) B <- matrix(rnorm(2048 * 2048), nrow=2048, ncol=2048) system.time(C <- A %*% B) user system elapsed library(HiPLARb) hiplarb_mode_magma()# to use GPU mode user system elapsed

R benchmark 2.5

Advantage: –No code changes needed –User can run R solution as before without knowledge of the parallel execution. Limitations: –Only support limited computational operations.

Break Big Computations with multiple R jobs Running R in non-interactive session User can submit multiple R jobs with different command Line parameters –Similar to run R batch mode –Parameters is specified on the command line –Good for repeated runs of same computations or running script partially Manually or via other tools

Hadoop and Spark Hadoop: –an open source project designed to support large scale data processing across clusters of computers – inspired by Google’s MapReduce-based computational infrastructure Spark –a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN.

MapReduce - Word Count from Revolution Analytics’ Getting Started with RHadoop course

Rhadoop and SparkR Rhadoop –an open source project sponsored by Revolution Analytics –package –rmr2 - all MapReduce-related functions –rhdfs - interaction with Hadoop’s HDFS file system – rhbase - access to the NoSQL HBase database SparkR –an R package that provides a light-weight frontend to use Apache Spark from R –exposes the Spark API through the RDD (Resilient Distributed Datasets) class

Text Analysis of HathiTrust Corpus (‘tm’ package, ~1M books) Guangchen Ruan, et al.

Advantages –Utilize efficiency of other data intensive processing framework –Each job can use existing R code Limitations –A “data-parallel” solution that may not suitable for simulation based analysis Supported on Wrangler and Rustler.

Running R with parallel packages

Parallel package Multicore –Utilizes multiple processing core within the same node. –Replace several common functions with parallel implementations apply  mcapply –lapply(1:30, rnorm) -> mclapply(1:30, rnorm) –Scalability is limited by the number of core and memory available within single node SNOW –Developed Based on Rmpi package, –Simplify the process to initialize parallel process over cluster. –Need to create “cluster“, e.g. cl <- makeCluster(4, type='SOCK’) parSapply(cl, 1:20, get("+"), 3)

Advantage –Do whatever you want with them –Get the best performance Limitations –Need code development –In some case, the analysis workflow may need be changed.