1. data on marble: what and where
how to process (visualise, analyse, etc) the data:
ncdump, python, ferret, ...

2. : what ??? data, what? cmip3 (IPCC-AR4) cmip5 (IPCC-AR5) cmip5_25x25
MIP == Model Intercomparision Project
The homepage of the Coupled Model Intercomparison Project (CMIP).
Here you find links to CMIP Phase 3 (CMIP3) and CMIP Phase 5 (CMIP5).
Recommended is the Guide to CMIP5 where you find also links to other MIPs like GeoMIP.
GeoMIP Geoengineering - sres special report on emission scenarios – rpc representation concentration pathway

3. : where ??? data, where? data_dir = /data/marble
CMIP3: data_dir/cmip3/IPCC_AR4/list (list with (soon) available data)
CMIP5: data_dir/cmip5/experiment/temporal-resolution/variable/model/realisation/*.nc CMIP5_25x25: same like cmip5; 25×25 indicates the regular horizontal grid of 2.5×2.5 deg
GeoMIP: data_dir/GeoMIP/experiment/temporal-resolution/mon/variable
/institution/model/realisation/*.nc

4. : what and where ??? data_dir/cmip5/experiment/
CMIP5-experiments: piControl, rcp26, rcp45, rcp60, rcp85, historical, historicalExt, historicalGHG, historicalMisc, historicalNat, 1pctCO2, abrupt4xCO2, esmFdbk1, esmFixClim2, esmHistorical, esmrcp85, amip
GeoMIP-experiments: piControl, rcp45, 1pctCO2, abrupt4xCO2, db, G1, G1oceanAlbedo, G2, G3, G3S, G4, G4cdnc, G5
data_dir/cmip5/experiment/temporal-resolution/
As temporal resolution we have at our disposal: monthly data from the Atmosphere, Ocean and Land (Amon, Omon and Lmon) and fixed data (fx) containing e.g. area (areacello) and volume (volcello) information about the model.
data_dir/cmip5/experiment/temporal-resolution/variable/
e.g. ~$ ls /data/marble/cmip5/piControl/Omon: fgco2 hfds so thetao tos zos zostoga
e.g. ~$ ls /data/marble/GeoMIP/piControl/Amon/mon: cSoil lai pr ra rsdt so tasmax thetao ts gpp orog prsn rlut rsut tas tasmin tos zos
Two useful pdf-files about the model experiments and variables: experiment controlled vocabulary and standard_output. The latter is quite comprehensive
and is reduced to our available variables (based on the pre-industrial control experiment (piControl)): standard_output_red.xlsx (this list also holds for our GeoMIP data).
e.g. fgco2 downward CO2 flux at ocean surface (positive into the ocean) – hfds downward net heat flux into the ocean (positive net ocean heat gain) – gpp gross primary carbon production on land

5. : what and where ???
data_dir/cmip5/experiment/temporal-resolution/variable/model/
e.g. $ ls /data/marble/cmip5/piControl/Amon/ts:
ACCESS CESM1-CAM5-1-FV2 FGOALS-g2 GISS-E2-R-CC MIROC5
ACCESS CESM1-FASTCHEM FGOALS-s2 HadGEM2-AO MIROC-ESM
bcc-csm CESM1-WACCM FIO-ESM HadGEM2-CC MIROC-ESM-CHEM
bcc-csm1-1-m CMCC-CESM GFDL-CM HadGEM2-ES MPI-ESM-LR
BNU-ESM CMCC-CM GFDL-ESM2G inmcm MPI-ESM-MR
CanESM CMCC-CMS GFDL-ESM2M IPSL-CM5A-LR MPI-ESM-P
CCSM CNRM-CM GISS-E2-H IPSL-CM5A-MR MRI-CGCM3
CESM1-BGC CNRM-CM GISS-E2-H-CC IPSL-CM5B-LR NorESM1-M
CESM1-CAM5 CSIRO-Mk GISS-E2-R MIROC4h NorESM1-ME
CMIP5_modeling_groups.pdf
data_dir/cmip5/experiment/temporal-resolution/variable/model/realisation/*.nc
e.g. $ ls /data/marble/cmip5/historical/Omon/zostoga/ACCESS1-3: r1i1p1 r2i1p1 r3i1p1
(you always can check the availability of data with e.g. 'ls /data/marble/cmip5/* > out.dat')

6. : what and where ???
example file (3d, x,y,t) time series at each areal grid-point
cd /data/marble/cmip5/rcp85/Omon/hfds/ACCESS1-0/r1i1p1
~$ ls
-> hfds_Omon_ACCESS1-0_rcp85_r1i1p1_ nc
example file (4d, x,y,z,t) time series at each volumetric grid point
cd /data/marble/cmip5/rcp85/Omon/thetao/ACCESS1-0/r1i1p1
-> thetao_Omon_ACCESS1-0_rcp85_r1i1p1_ nc
thetao_Omon_ACCESS1-0_rcp85_r1i1p1_ nc ...

7. ... some plots ...
1/ hfds_Omon_ACCESS1-0_rcp85_200601_ nc 2/ hfds_Omon_ACCESS1-0_rcp85_200601_ nc
3/ thetao_Omon_ACCESS1-0_historical_r1i1p1_200001_ nc 4/ zostoga_Omon_ACCESS1-0_historical_r1i1p1_ nc
hfds

8. : netcdf ??? netCDF: network Common Data Form info faq
meta-data: ~$ ncdump –h file_name
example: ~$ ncdump –h /data/marble/cmip5/rcp45/Omon/zostoga/ACCESS1-0/r1i1p1/zostoga_Omon_ACCESS1-0_rcp45_r1i1p1_ nc
netcdf zostoga_Omon_ACCESS1-0_rcp45_r1i1p1_ {
dimensions:
time = UNLIMITED ; // (1140 currently)
bnds = 2 ;
variables:
double time(time) ;
time:bounds = "time_bnds" ;
time:units = "days since " ;
double time_bnds(time, bnds) ;
float zostoga(time) ;
zostoga:standard_name = "global_average_thermosteric_sea_level_change" ;
raw-data: ~$ ncdump –v variable_name file_name
example: ~$ ncdump –vzostoga /data/marble/cmip5/rcp45/Omon/zostoga/ACCESS1-0/r1i1p1/ostoga_Omon_ACCESS1-0_rcp45_r1i1p1_ nc
zostoga = , , , , , , , , ....
Typically you would use ncview to get a quick and easy, push-button look at your netCDF files.
NO data analysis tool

9. : netcdf ??? first visualisation step: ~$ ncview file_name
e.g. ~$ ncview /data/marble/GeoMIP/piControl/Amon/mon/ra/NASA-GISS/
GISS-E2-R/r1i1p1/ra_Lmon_GISS-E2-R_piControl_r1i1p1_ nc
non-script-based
netcdf_software
panoply

10. data visualization with ferret
What data means without proper visualization? Unprocessed data is bunch of jargon to many. Cleansing can make it edible. And yes, not everyone is a data scientist:
data visualization with ferret
ferret_users_guide
start ferret: ~$ ferret -nojnl load data: yes? use "file_dir/file_name show data: yes? sh  variable_name plot data[1d]: yes? plot variable_name shade data[2d]: yes? shade variable_name[i=,j=,z=,t=] make (non-weighted) averages: yes?
visualising data tool – limited analysis tool – script-based

11. (/home/UNIMELB/kdommenget/ferret_scripts/examples)
ferret examples
for shading, plot a time series and profile plot and how to remove the seasonal climatology
(/home/UNIMELB/kdommenget/ferret_scripts/examples)
1/ shading_tave_ex.jnl, 2/ plot_tseries_aave_ex.jnl, 3/ plot_tave_profile_ex.jnl, 4/ rem_seasonal_clim_ex.jnl
1/ shading_tave_ex.jnl
ca da/all
ca var/all
!""" contours/shades from your monthly time series, based on: """
!""" usage go shade_tave_ex.jnl dir file_name var_name tb te """
!dir = /data/marble/cmip5/rcp85/Omon/hfds/ACCESS1-0/r1i1p1
!file_name = hfds_Omon_ACCESS1-0_rcp85_r1i1p1_ nc
!var_name = hfds
!tb = 2006
!te = 2100
define symb dir = $1
define symb file_name = $2
define symb var_name = $3
define symb te = $4
define symb tb = $5
!""" example data set: monthly downward heat flux at sea water surface (in W/m**2) """
!""" comment: this is the net heat entering the liquid water column through its upper surface. """
use "($dir)/($file_name)"
!""" (optional) set the time range of data ""“
efine axis/calendar=gregorian/t="15-jan-($tb)":"15-dec-($te)":1/unit=months tmon
let var_name_of_interest =
!""" compute the temporal mean """
let var_name_ave =
!""" contour/shade variable """
shade/lev="(-inf)(-100,100,20)(inf)" var_name_ave, lon, lat
!""" save file """
frame/file=test.gif
sp convert test.gif shade_tave_ex.pdf
sp \rm test.gif]
2/ plot_tseries_ex.jnl
ca da/all
ca var/all
!""" plots your monthly time series, based on: """
!""" usage go plot_tseries_ex.jnl dir file_name var_name tb te """
!dir = /data/marble/cmip5/rcp85/Omon/hfds/ACCESS1-0/r1i1p1
!file_name = hfds_Omon_ACCESS1-0_rcp85_r1i1p1_ nc
!var_name = hfds
!tb = 2006
!te = 2100
define symb dir = $1
define symb file_name = $2
define symb var_name = $3
define symb te = $4
define symb tb = $5
!""" example data set: monthly downward heat flux at sea water surface (in W/m**2) """
!""" comment: this is the net heat entering the liquid water column through its upper surface. """
use "($dir)/($file_name)"
!""" (optional) set the time range of data ""“
let var_name_of_interest =
!""" plot variable """
let/title="global mean"
let/title="cold tongue"
let/title="Labrador Sea"
plot var_gm, var_ct, var_ls
!""" save file """
frame/file=test.gif
sp convert test.gif plot_tseries_ex.pdf
sp \rm test.gif
3/ plot_tave_profile_ex.jnl
ca da/all
ca var/all
!""" plots profile from your monthly time series, based on: """
!""" usage go plot_tave_profile.jnl dir file_name var_name tb te """
!dir = /data/marble/cmip5/historical/Omon/thetao/ACCESS1-0/r1i1p1
!file_name = thetao_Omon_ACCESS1-0_thetao_r1i1p1_ nc
!var_name = thetao
!tb = 2000
!te = 2004
define symb dir = $1
define symb file_name = $2
define symb var_name = $3
define symb te = $4
define symb tb = $5
!""" example data set: monthly potential temperature (in K) """
use "($dir)/($file_name)"
!""" (optional) set the time range of data ""“
define axis/calendar=gregorian/t="15-jan-($tb)":"15-dec-($te)":1/unit=months tmon
let var_name_of_interest =
!""" define depths levels """
let zz = z[gz=var_name_of_interest]
define axis/z/units=m/depth zlevn = zz
!""" plot variable """
let/title='global mean'
let/title='cold tongue'
let/title='Labrador Sea'
plot var_gm, var_ct, var_ls
!""" save file """
frame/file=test.gif
sp convert test.gif plot_tave_profile.pdf
sp \rm test.gif
4/ rem_seasonal_clim_ex.jnl
ca da/all
ca var/all
!""" removes seasonal climatology (climatological annual cycle) from your monthly time series, based on: """
!""" """
!""" usage go remove_seasonal_clim.jnl dir file_name var_name tb te """
!dir = /data/marble/cmip5/historical/Omon/zostoga/ACCESS1-0/r1i1p1
!file_name = zostoga_Omon_ACCESS1-0_historical_r1i1p1_ nc
!var_name = zostoga
!tb = 1850
!te = 2005
define symb dir = $1
define symb file_name = $2
define symb var_name = $3
define symb te = $4
define symb tb = $5
!""" example data set: monthly global mean thermosteric sea level """
use "($dir)/($file_name)"
!""" get the climatological axis that you want to use """
!""" here we will use the pre-defined axis ("month-reg") """
use climatological_axes
cancel data climatological_axes
!""" (optional) set the time range of data to be used in the climatology """
define axis/calendar=gregorian/t="15-jan-($tb)":"15-dec-($te)":1/unit=months tmon
let var_name_of_interest =
!""" compute the climatology """
let var_name_climatology =
!""" subtract to get the anomaly """
!""" regrid the anomaly back to the original time axis to guarantee success """
!""" (subtle interpolation errors may occur on irregular time axes if regridding isn't done.) """
let var_name_anomaly = var_name_of_interest -
plot
plot/ov var_name_of_interest
plot/ov

12. where to find ferret examples and demo-scripts?
resulting plots ...
1/ shading_tave_ex.jnl 2/ plot_tseries_aave_ex.jnl
3/ plot_tave_profile_ex.jnl 4/ rem_seasonal_clim_ex.jnl
where to find ferret examples and demo-scripts?
/data/marble/fer_dsets/data
/usr/local/ferret/examples (in ferret: go example_file.jnl)

13. : python ??? python faq – stackoverflow
a python tutorial for climate scientist (by ARC-CCS center of excellence in climate system science)
a netcdf data reading routine with python (>2.7) (readnetcdf_example.py):
main file:
## this is an example for the header of the main program import scipy.io.netcdf as spn import glob from cmip5_rd import * ## read netcdf-data and process further with a1.keys() and look at the key-variables with a1['var'][:] or firstly, to get the shape: a1['var'].shape a1 = rdnetcdfdata('filename_as_string')
subroutine: ## this could be subroutine "cmip5_rd.py" import scipy.io.netcdf as sp def rdnetcdfdata(filename):         """reads netcdf-data: output is a dictionary process further with keys etc """         fobj = sp.netcdf_file(filename, mode='r')         outputarray = fobj.variables         fobj.close()         return outputarray
visualising and analysis tool – script-based

14. : python ??? read_and_write_netcdf_files:
some plots ...
example files in /home/UNIMELB/kdommenget/python_scripts
no basemap yet
read_and_write_netcdf_files:
examples for a variety of programming languages including python and matlab.

15. What is the difference between
Data, Presentation, Information and Knowledge?