1. Steven Gollmer Cedarville University
Big Data
Steven Gollmer
Cedarville University
Picture from wikipedia - Dice

2. Working with Large Data
Accessing data
Collection and calibration assumptions
Selecting appropriate parameters
Formatting
Calculation
Testing hypothesis

3. Hipparcos Space Astrometry
Main Page
Data Catalogues
Software
Desktop -
Search tool -
Data Format
Flexible Image Transport System (FITS) -

4. Sloan Digital Sky Survey
Main Page
Data
9th Data Release -
Archive Server -
Software
IDL -

5. Weather Data NOAA National Climatic Data Center
Popular Data -
Environmental Modeling Center

6. TERRA/AQUA http://terra.nasa.gov http://aqua.nasa.gov Data Format
LARC DAAC -
LAADS Web -
Format
NetCDF -
HDF -

7. Other Topics of Interest
Extra-Solar Planets
Asteroid Mapping and Near Earth Detection
Earthquakes
Agencies and Products
NASA -
ESA -
USGS -
GOES -
Paleoclimatology -

8. Hypothesis Testing P-value T-test
Probability of a value being found assuming the null hypothesis.
Usually reject the null hypothesis if p < 0.05 or (5% or 1%)
May have more stringent criteria for rejection.
T-test
Assume a normal distribution
One-sample test 𝑡= 𝑥 − 𝜇 0 𝑆/ 𝑛
Two-sample test 𝑡= 𝑀 𝑥 − 𝑀 𝑦 𝑆𝑥2 𝑛𝑥 + 𝑆𝑦2 𝑛𝑦
Check significance using T distribution table
Compare t value and degrees of freedom
1 sample df = n sample df = n1 + n2 – 2
S – Estimate of standard deviation
M – Estimate of the mean
n – Number of samples

9. Example Hypothesis Statistics Result 2 tail rejection
Data is from a distribution with mean m = 2.5
Statistics
X = 3.317
S =
df = 5
Result
T = 2.80
2 tail rejection
p = 0.05 is 2.571
p = 0.02 is 3.365
Data
2.3
4.2
3.6
3.1
2.8
3.9

10. Z-Value Assume a normal random variable Z – Value
x ~ (m, s2)
m – mean
s – standard deviation
Z – Value
z ~ (0, 1)
If number of samples is large, then z-test will work on one-sample test instead of a t-test.
erf(x)= 2 𝜋 0 𝑥 𝑒 − 𝑢 2 𝑑𝑢
One Tail: p=1/2(1+erf(z/ 2 ) Two Tail: p=erf(z/ 2 )
𝑓 𝑥 = 1 𝜎 2𝜋 𝑒 − (𝑥−𝜇) 2 2 𝜎 2
𝑧~ 𝑥−𝜇 𝜎