Open Source Analytics Visualization and Predictive Modeling of Big Data with R Michael E. Driscoll, Ph.D. July 22, 2009 OSCON

(from Jessica Hagys thisisindexed.com) Hard-working Middle Class Hypothesis

gdp <- read.csv('gdp.csv') hours <- read.csv('hours.csv') gdp.hours <- merge(hours,gdp) gdp.hours$freetime < gdp.hours$hours attach(gdp.hours) plot(freetime ~ gdp) m <- lm (freetime ~ gdp,data=gdp.hours) abline(m,col=3,lw=2) pm <- loess(freetime ~ gdp) lines(spline(gdp,fitted(pm))) Munge & Model OECD Data

Visualize the Analysis: is it True?

modeling Big Data

100 thousand gene measures

1 million transactions during this presentation

If You Liked ____, Youll Love ___ !

1 billion clicks during this presentation

1 million pitches thrown since 2007

A Tale of Two Pitchers Hamels Webb

xyplot(x ~ y, data=pitch)

xyplot(x ~ y, groups=type, data=pitch)

xyplot(x ~ y | type, data=pitch)

xyplot(x ~ y | type, data=pitch, fill.color = pitch$color, panel = function(x,y, fill.color, …, subscripts) { fill <- fill.color[subscripts] panel.xyplot(x,y, fill= fill, …) })

xyplot(x ~ y | type, data=pitch, fill.color = pitch$color, panel = function(x,y, fill.color, …, subscripts) { fill <- fill.color[subscripts] panel.xyplot(x, y, fill= fill, …) })

visualizing Big Data

ggplot2 = grammar of graphics

qplot(carat, price, data = diamonds)

qplot(log(carat), log(price), data = diamonds) qplot(carat, price, log=xy, data = diamonds) OR

qplot(log(carat), log(price), data = diamonds, alpha = I(1/20))

qplot(log(carat), log(price), data = diamonds, alpha=I(1/20)) + facet_grid(. ~ color)

R on the cloud

Data Desktop

CodingClicking vs

Linux Apache MySQL R

Final thoughts