Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth Sciences bozkurtd@itu.edu.tr The International Training Course on “Climate Analysis and Applications” Alanya 10-19 October 2011

Climate Datasets Datasets

Digital representation “11010001” Climate Datasets Raw data “abcd12345” Input system Digital representation “11010001”

Climate Datasets Modeled data: gridded multi-dimendisonal data Consisting of variables, dimensions and attributes Observations (station, reanalysis)

Climate Datasets Climate data can be stored as different formats in different platforms: Character format Packed binary format

Climate Data Formats - Character Format ASCII (American Standard Code for Information Interchange) It refers to a “text” file that is readable by the naked eye (it only contains the letters a-z, numbers, carriage returns, and punctuation marks. Each character is stored as digital = Binary 1001000 1100101 1101100 1101111 0101100 0100000 1110111 1100111 1110010 1100100 H e l o , w r d

Climate data formats – ASCII

Climate data formats – ASCII Factors restrict the use of ASCII: Weak data storage and huge data size: “9” 1 byte, “679.43” 6 byte, -0.123456E+05 13 byte No or not practical metadata

Climate data formats – Packed Binary A common data format for saving and storing of large size of the data “3.1” 24 bits (ASCII) can be packed as 5 bit Can be processed very quickly and effectively in data analysis

Climate data formats – Packed Binary

Some standard data formats NetCDF (network Common Data Form) Format Self-describing data format All the information about data is stored inside of the data No need for an additional information for the identification of the contents of the file Machine independent Can be readable by various applications NCL, IDL, GrADS, Matlab, Ferret, C, C++, Java, Fortran Komut satırı operatörler: NCO, CDO Most commonly used format in the oceanographic and atmospheric science for observational data and numerical modeling (IPCC datasets, NASA, NOAA, NCAR, UCAR…)

Climate data formats - NetCDF metadata dimensions variables Variable properties attributes

Climate data formats - NetCDF Data storage in NetCDF format 3D (x,y,t) or 4D (x,y,z,t)

Climate data formats - NetCDF Data storage in NetCDF format netcdf mynetcdf { dimesions: x=4; y=4; time=UNLIMITED; variables: float x(x); float y(y); int time(time); float temperature(time,x,y); data: x = 10, 20, 30, 40; y = 110, 120, 130, 140; time = 31, 59, 90; }

Climate data formats - NetCDF Data storage in NetCDF format netcdf mynetcdf { dimesions: x=4; y=4; time=UNLIMITED; variables: float x(x); float y(y); int time(time); float temperature(time,x,y); data: x = 10, 20, 30, 40; y = 110, 120, 130, 140; time = 31, 59, 90; Temperature = 111, 211, 311, 411; } time = 1 x = 1 to 4 y = 1

Climate data formats - NetCDF Data storage in NetCDF format netcdf mynetcdf { dimesions: x=4; y=4; time=UNLIMITED; variables: float x(x); float y(y); int time(time); float temperature(time,x,y); data: x = 10, 20, 30, 40; y = 110, 120, 130, 140; time = 31, 59, 90; Temperature = 111, 211, 311, 411, 121, 221, 321, 421, 131, 231, 331, 431, 141, 241, 341, 441; } time = 1 x = 1 to 4 y = 1 to 4

Climate data formats - NetCDF Data storage in NetCDF format netcdf mynetcdf { dimesions: x=4; y=4; time=UNLIMITED; variables: float x(x); float y(y); int time(time); float temperature(time,x,y); data: x = 10, 20, 30, 40; y = 110, 120, 130, 140; time = 31, 59, 90; Temperature = 111, 211, 311, 411, 121, 221, 321, 421, 131, 231, 331, 431, 141, 241, 341, 441, 112, 212, 312, 412, 122, 222, 322, 422, 132, 232, 332, 432, 142, 242, 342, 442, 113, 213, 313, 413, 123, 223, 323, 423, 133, 233, 433, 143, 243, 343, 443; } time = 1 to 3 x = 1 to 4 y = 1 to 4

Climate data formats - NetCDF 4D (x,y,z,t)

Climate data formats - NetCDF some NetCDF commands ncdump file_name | less, to see the entire contents of the file ncdump –h file_name | less, to see the front portion of the data ncdump –v var_name file_name | less, to see the specific data within the file ncl_filedump to see the contents of different data formats

Climate data formats - GRIB GRIB (GRIdded Binary) Format World Meteorological (WMO) standard, commonly used in meteorology Bit-oriented data format: very efficient for transmission/archival - netCDF (float) 2-4 times larger than corresponding GRIB two dimensional horizontal grids only - each grid has scale/offset for optimal precision requires external parameter table (s)

Climate data formats - GRIB Basically consists of: Product definition section(PDS) Creater of the data, unit of the data, time stamp, vertical system of the data Grid description section (GDS), optional Grid description, map projections Bit map section (BMS), optional Defines which grid points will be presented Binary data section (BDS) Contains the packed binary data End section

Climate data formats - GRIB Files are typically 1/2 to 1/3 of the size of normal binary files: Y: gridded data D: scale factor R: reference value E: binary scale factor

Climate data formats – ArcGIS Shapefile NetCDF data: Raster Table Feature Directly readable

Climate data formats – ArcGIS Shapefile

Climate data formats – ArcGIS Shapefile Raster Feature Table

Climate data formats – ArcGIS Shapefile

Climate data formats – ArcGIS Shapefile http://www.gisclimatechange.org/