1. Spark with R Martijn Tennekes
THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

2. Spark and R: best of both worlds
Apache Spark
Cluster-computing framework
Supports SQL syntax
Built-in machine learning
High performance R
Many tools and packages for data analysis
Interactive documents
Powerful visualizations
Interactive dashboards

3. Using Spark in R RStudio
Spark objects are shown in the Connections window in the same way as local R objects are shown in the Environment window.
Button to access the Spark Web Console
R, package sparklyr
A dplyr back-end for Spark
Extendable, e.g. rsparkling for H2O (machine learning using Spark)
Easy to configure a Spark cluster, or to set up a local Spark cluster

4. Data science workflow
Source: R for Data Science - Garrett Grolemund, Hadley Wickham

5. Import data Connect to Spark sc <- spark_connect() Import data
From csv file x_tbl <- spark_read_csv(sc, "x.csv")
From json x_tbl <- spark_read_json(sc, "x.json")
From parquet (HDFS) x_tbl <- spark_read_parquet(sc, "name", path)
From R object x_tbl <- copy_to(sc, x)
The returned object (x_tbl) is a tbl_spark object, equivalant to a normal tbl (data.frame)
Retrieving Spark tables
Get table names src_tbls(sc)
Get tbl_spark object x_tbl <- tbl(sc, "table")

6. Process data Process data with dplyr functions, e.g.
SQL
Description
select
SELECT
Select columns
filter
WHERE
Filter rows
arrange
ORDER
Order rows
summarise
AVG, SUM, MIN, etc.
Aggregate columns
mutate
+, *, LOG, etc.
Create new columns
Process data with dplyr functions, e.g.
filter to filter rows
select to select columns
mutate to create new variables
group_by to specify the groups
summarize to summarize the values
left_join to join tables

7. Process data Process data with the main dplyr functions: dplyr SQL
Description
select
SELECT
Select columns
filter
WHERE
Filter rows
arrange
ORDER
Order rows
mutate
+, *, LOG, etc.
Create new columns
group_by
GROUP BY
Group rows
summarise
AVG, SUM, MIN, etc.
Aggregate columns
Recommendation: use the pipe operator %>%
tmp <- functionA(x, paramA1, paramA2) y <- x %>%
y <- functionB(tmp, paramB) functionA(paramA1, paramA2) %>%
functionB(paramB)

8. Visualize data Approach:
Process the data such that the output is exactly that is needed for the plot
Use collect() to copy this data into R’s memory
Use general purpose R packages to create the plot:
ggplot2 for any type of plot (line chart, bar chart, scatter plot, heatmap, etc)
tmap for (interactive) maps

9. Model data Any R package H2O Spark MLlib
Many implemented methods (over CRAN packages)
Downside: data needs to be collected into R’s memory (see previous slide).
H2O
Open-source software for big data analysis (statistical methods / machine learning)
Integration with Spark via Sparkling Water
Sparkling Water is accessable with the R package rsparkling
Spark MLlib
Native machine learning algortihms
All functions have the prefix ml_ for instance ml_random_forest
Functions tidy, augment, and glance from the broom package are implemented for these methods. These functions summarize the method output at different levels. The format is always a data.frame.

10. Communicate results R Markdown Shiny
Markdown is a lightweight and easy to use markup language
With R Markdown, chunks of R code can be embedded. Each chunk can either be shown or hidden in the document. Also, the output of the chunk can either be shown or hidden.
Interactive visualizations can be included as well
Shiny
Interactive dashboards
Can be run locally, or hosted on a server
Free servers available, i.e. shinyapps.io

11. Overview of sparklyr functions
Function naming:
Feature transformation functions: prefix ft_
Machine learning functinos (Spark MLlib): prefix ml_
Spark DataFrame functions*: prefix sdf_
Configuration / read / write functions: prefix spark_
Streaming data functions: prefix stream_
* Note that the functions can often be provided without prefix. For instance, by using sdf_copy_to on a Spark data.frame, copy_to is used.

12. References
R for Data Science - Garrett Grolemund and Hadley Wickham
sparklyr: R interface for Apache Spark – Rstudio