1. Precipitation As An EWV
G.J. Huffman
NASA/Goddard Space Flight Center
Input Data Sources
Data Sets
Dominant Controls on Performance
IMERG Output Data Fields
Issues
Final Comments

2. Distance from nearest gauge
1. Input Data Sources – Surface
Precip gauges are considered the standard, but
• irregularly spaced
• restricted access to data
• latency of days to months
• suffer from undercatch
• 5-150%
• worse in wind, drizzle, snow
• can (and should) be corrected
Surface radars provide local coverage, but
• relatively sparse
• diverse operational schemes
• radar artifacts
• not direct – still a retrieval
• gauge-adjusted radar generally considered the best (MRMS)
[Kidd et al. (2016), Fig. 1]
Distance from nearest gauge
We don’t use regional input data, such as radar networks, but this *global* product could serve as a base for a regional value-added product.
The overpass-time graphic is very popular and freely available – send mail to to get the digital image and/or be put on the update mailing list
[Heistermann et al. (2013), Fig. 1]

3. 1. Input Data Sources – Satellite
A diverse, changing, uncoordinated set of input precip estimates, with various
• periods of record
• regions of coverage
• sensor-specific strengths and limitations
infrared microwave
latency min 3-4 hr
footprint 4-8 km km
interval min (to ≤3 hr) hr (~3 hr)
“physics” cloud top hydrometeors
(weak) (strong)
• additional microwave issues over land include
• scattering channels only
• issues with orographic precip
• estimates over snow/ice are research topics
We don’t use regional input data, such as radar networks, but this *global* product could serve as a base for a regional value-added product.
The overpass-time graphic is very popular and freely available – send mail to to get the digital image and/or be put on the update mailing list
[Knapp et al. (2011), Fig. 1a]

4. 1. Input Data Sources – Models and Reanalyses
Numerical schemes start with the real equations governing the atmosphere
• but then computational reality intervenes
• precipitation is profoundly local, with large-scale controls
Personal observations:
• models tend to enforce grid-scale balance too rigorously
• diurnal cycle of convection follows the Sun too closely
• model estimates in convective regimes lose to “good” satellite estimates
• precip is generally frequent, very light
• but at higher latitudes in cool season the precipitation is often not convective
• previous-generation satellite estimates lose to models (Ebert et al. 2007)
We don’t use regional input data, such as radar networks, but this *global* product could serve as a base for a regional value-added product.
The overpass-time graphic is very popular and freely available – send mail to to get the digital image and/or be put on the update mailing list

5. IPWG data listings: http://www.isac.cnr.it/~ipwg/data/datasets.html
2. Data Sets – Publicly Available, Quasi-Operational, Quasi-Global Precipitation Estimates
IPWG data listings:
Table 1 Combination Data Sets with Gauge Data
CAMS/OPI
CMAP
CMORPH V1.0 BIAS-CORRECTED
CMORPH V1.0 BLENDED
GPCP One-Degree Daily (Version 1.2)
GPCP pentad (Version 1.1)
GPCP Version 2.2 Satellite-Gauge (SG)
GSMaP Gauge-calibrated (GSMaP_Gauge) V7
GSMaP Reanalysis Gauge-calibrated (GSMaP_RNL_Gauge) V7
H05
IMERG Final Run V3
MSWEP
PERSIANN-CDR
RFE
TRMM Plus Other Data (3B43 Version 7)
TRMM Plus Other Satellites (3B42 Version 7)
Table 4 Precipitation Gauge Analyses
APHRODITE Water Resources
CPC Unified Gauge-based Anal. of Global
Daily Precip.
CRU Gauge
CRU TS 3.23 Gauge
Dai Gauge Dataset 2
GHCN+CAMS Gauge
GPCC Monitoring
GPCC Full Data Reanalysis Version 6
GPCC VASClimO Version 1.1
Table 2 Combination Data Sets
AIRG2SSD
AIRX2SUP
AIRX2SUP_NRT
AIRX3SPD, AIRX3SP8, AIRX3SPM
CMORPH
CMORPH V1.0 RAW
GSMaP Near-real-time　(GSMaP_NRT)
GSMaP Realtime (GSMaP_NOW)
GSMaP Standard (GSMaP_MVK) V7
GSMaP Reanalysis (GSMaP_RNL) V7
H03
IMERG Early Run V3
IMERG Late Run V3
MPE
 MSWEP NRT
NRL Real TIme
PERSIANN
PERSIANN-CCS
SCAMPR
TCI (3G68)
TOVS
TRMM Real-Time HQ Version 7 (3B40RT)
TRMM Real-Time VAR Version 7 (3B41RT)
TRMM Real-Time HQVAR Version 7 (3B42RT)

6. 2. Data Sets – Two Styles
Climate Data Records (CDR)
• emphasize homogeneity over fine-scale accuracy
• examples
• GPCP SG, 1DD
High Resolution Precipitation Products (HRPP)
• emphasize fine-scale accuracy over homogeneity
• but homogeneity still valued
• CPC Morphing algorithm (CMORPH)
• Global Satellite Map of Precipitation (GSMaP)
• Integrated Multi-satellitE Retrievals for GPM (IMERG)
• Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks (PERSIANN)
• Self-Calibrating Multivariate Precipitation Retrieval (SCaMPR)
• TRMM Multi-satellite Precipitation Analysis (TMPA)

7. 3. Dominant Controls on Performance
Each product should tend to follow its calibrators at a few weeks and longer
• over land (for GPCP, TMPA, IMERG) – GPCC gauge analysis
• under-catch corrected
• month-to-month (post-real time) or climatological (near-real time)
• [ over ocean – satellite calibrator ]
• climatological calibration only sets long-term bias, not month-to-month behavior
• Zhou et al. (2015) uncovered regional variations over time between month-to- month and climatological calibrations
Short-interval variations
• land and ocean: precip events are driven by the individual input datasets
• inter-satellite calibration attempts to enforce consistency in distribution
• event-driven statistics depend on satellites, e.g. bias in frequency of occurrence
• differences between sensors tend to be noticeable
• different sensors “see” different aspects of the same scene
• limited opportunities to “fix” problems with the individual inputs on the fly
• satellite sensors tend to be best for tropical ocean
• satellite sensors and rain gauge analyses tend to have more trouble in cold areas and complex terrain

8. 4. IMERG Output Data Fields
Example of a modern combination dataset
Output dataset includes intermediate data fields
• users and developers require
• processing traceability
• support for algorithm studies
0.1° global CED grid
• 3600x1800 = 6.2M boxes
• fields are 1-byte integer, scaled 2-byte integer, or 4-byte real
• but, dataset compression means smaller disk files
• and PPS provides subsetting
“User” fields in italics, darker shading
HDF5 and other formats
• quasi-standard metadata

9. New in V05 by user request: “simple” index of “quality”
4. IMERG Output Data Fields
Example of a modern combination dataset
Output dataset includes intermediate data fields
• users and developers require
• processing traceability
• support for algorithm studies
0.1° global CED grid
• 3600x1800 = 6.2M boxes
• fields are 1-byte integer, scaled 2-byte integer, or 4-byte real
• but, dataset compression means smaller disk files
• and PPS provides subsetting
“User” fields in italics, darker shading
HDF5 and other formats
• quasi-standard metadata
New in V03 by user request: precip phase as (diagnostic) probability of “liquid” (including “mixed”)
New in V05 by user request: “simple” index of “quality”

10. 5. Issues (1/2)
Specific product names and versions are required in papers and discussions to prevent confusion
• “TRMM” is not an acceptable shorthand for “3B42-RT V7”
Latency for global satellite products > 3-4 hours
• driven by latency of data downlinks from satellites
• faster delivery is being developed regionally
• but, “faster” is usually “more approximate”
No general “suitability for use” metrics
• behavior by climate zone and weather regime can be different
• regional datasets can be more focused than global datasets
• different versions can have rather different qualities
• check with developers
• the state of the art is generally ahead of the refereed literature
• IMERG V05 will have a “quality index”
• detailed error representations are still a work in progress

11. 5. Issues (2/2)
Current research topics
• error representation
• precipitation in complex terrain
• precipitation in coastal zones
• rain and snowfall over snowy/icy/frozen surface
• key topic in GPM
• several new concepts have surfaced
• use of daily gauge data (vs. monthly)
• fully global precip data sets
• need rain and snowfall over snowy/icy/frozen surface
• need “morphing” vectors at high latitudes
• possible contributions by models/reanalyses
Long-term support is required
• many products are only “best effort”
• episodic retrospective processing for continued homogeneity
• continued launches to maintain the microwave constellation
• continued precip gauge observations

12. 6. Final Comments
Precip gauges, surface radars, satellites, and models/reanalyses can be used to create global precip datasets
Combination datasets using microwave satellite, infrared satellite, and precip gauge input are the current state of the art
CDR and HRPP precip data sets have (somewhat) different goals
Calibration schemes (should) control bias for intervals over a few weeks
Short-interval events driven by satellite input data sets
Suitability for use is still determined case by case
• error representations are still a work in progress

13. Backup Slides

14. • limit microwave calibrator to 6 a.m./p.m. satellite
2. GPCP
CDR-like GPCP
• limit microwave calibrator to
6 a.m./p.m. satellite
• consistent diurnal bias
• use to “adjust” GPI
At higher latitudes use Susskind-type scheme with TOVS & AIRS
sounding retrievals
Before the DMSP era use OLR Precip Index at all latitudes
GPCP gauge analysis throughout
• Full and Monitoring products relatively consistent
Emission
Microwave
Satellite/Gauge
Scattering
TOVS/ AIRS
Microwave/IR
Calibration
Gauge
Gauge-Adjusted
Satellite
Multi-Satellite
Geo
Low-Orbit
OPI
Matched
3-hr
Geo-IR GPI
Merged IR
GPI
1986-present
1987-present
Microwave/Other
Fusion
AGPI

15. Computed in both real and post-real time, on a 3-hr 0.25° grid
3. TMPA – Flow Chart (1/2)
Computed in both real and post-real time, on a 3-hr 0.25° grid
Microwave precip:
• intercalibrate to TMI/PR combination for P
• intercalibrate to TMI for RT
• then combine, conical- scan first, then sounders
IR precip:
• calibrate with microwave
Combined microwave/IR:
• IR fills gaps in microwave
GPCP Version 2.2 is out in provisional form and due for final form “soon”

16. Computed in both real and post-real time, on a 3-hr 0.25° grid
3. TMPA – Flow Chart (1/2)
Computed in both real and post-real time, on a 3-hr 0.25° grid
Microwave precip:
• intercalibrate to TMI/PR combination for P
• intercalibrate to TMI for RT
• then combine, conical- scan first, then sounders
IR precip:
• calibrate with microwave
Combined microwave/IR:
• IR fills gaps in microwave
Instant- aneous
SSM/I
TRMM
AMSR
AMSU
MHS
HQ coefficients
3-hourly merged HQ
3- hourly IR Tb
Hourly HQ-calib IR precip
3-hourly multi-satellite (MS)
Monthly gauges
Monthly SG
Rescale 3-hourly MS to monthly SG
Rescaled 3-hourly MS
Calibrate High-Quality (HQ) Estimates to “Best”
Merge HQ Estimates
Match IR and HQ, generate coeffs
Apply IR coefficients
Merge IR and HQ estimates
Compute monthly satellite-gauge combination (SG)
30-day IR coefficients
Monthly Climo. Adj. Coeff.
The TMPA Version 7 was released in June.

17. 3. TMPA – Final Calibration Production TMPA
• monthly MS and GPCC gauge analysis combined to Satellite-Gauge (SG) product
• weighting by estimated inverse error variance
• 3-hrly MS rescaled to sum to monthly SG
Real-Time TMPA
• 3-hrly MS calibrated using climatological TCI, 3B43 coefficients
Each product should tend to follow its calibrators
• over land – the GPCC gauge analysis
• over ocean – satellite calibrator
• climatological calibration only sets long-term bias, not month-to-month behavior
• Zhou et al. (2015) uncovered regional variations over time
The TMPA Version 7 was released in June.

18. Product scale and coverage
5. GPCP V3 – Products and Inputs
Product scale and coverage
Input data
6 a.m./p.m. GPROF-SSMI/SSMIS August 1987–present
Monthly Susskind TOVS/AIRS August 1987–present
GridSat-B –2010 (to be extended)
Monthly 1979–present
CPC 4-km Merged IR 2011–present
0.5° global
OPI 1979–present
GPCC Full Gauge Analysis 1979–2010 (to be extended)
GPCC Monitoring Gauge Analysis 2011–present
TRMM Composite Climatology
Pentad 1979–present
CMAP Pentad Precipitation 1979–present
GPCP Daily 1982–present
GPCP Monthly 1982–present
Daily 1982–present
6 a.m./p.m. GPROF-SSMI/SSMIS August 1987–present
0.5° 60°N-S 1982–1996
Daily Susskind TOVS/AIRS October 1996–present
0.5° global 1996–present
Gridsat-B (to be extended)
GPCP 3-hr 1998–present
3-hr 1998–present
GPCP Monthly 1998–present
0.1° 60°N-S (to be extended)
GPM IMERG 1998–present

19. It is planned to provide the following data fields for all products:
5. GPCP V3
– Output Data Fields
The standard monthly precipitation estimate will be a satellite-gauge combination, while all the shorter-interval products will be calibrated to the monthly satellite-gauge product.
It is planned to provide the following data fields for all products:
• accumulated precipitation (mm/d)
• estimated error
• precipitation phase (percentage probability of liquid)
The monthly dataset will also provide:
• multi-satellite accumulated precipitation (e.g., without month-to-month gauge information; mm/d)
• gauge relative weighting in the final satellite-gauge product
The 3-hourly dataset will also provide:
• snapshot precipitation rate (expressed in mm/d)

20. – Observation-Model Products
Experience shows that numerical model-based precipitation estimates are more skillful than observational products in cold-season mid- and high-latitude regions.
In parallel with (not replacement of) the observational products, this project plans to create monthly observation-model products:
• start work using the NASA Modern-Era Retrospective Analysis for Research and Applications (MERRA)
• blend the GPCP multi-satellite and MERRA precipitation fields following Sapiano et al. (2008)
• carry out the gauge combination with the blended satellite-model field
Shorter-interval combinations currently lack a developed scheme for use. Recent work by Xie and collaborators on merging daily satellite, gauge, and radar data might provide the necessary framework.
5. GPCP V3

21. Interpolate between PMW overpasses, following the
cloud systems. The current state of the art is
• estimate cloud motion fields from geo-IR data
• move PMW swath data using these displacements
• apply Kalman smoothing to combine satellite
data displaced from nearby times
Currently being used in CMORPH, GSMaP (Japan)
Introduces additional correlated error
30-min HQ Precip (mm/h) 00Z 15 Jan 2005
6. IMERG – Introduction
The “good stuff” (microwave) is sparse
• 30 min of data shows lots of gaps
• extra gaps due to snow in N. Hemi.
• 4 imagers, 3 sounders
TMI
TCI
Aqua
AMSR-E
N16
AMSU
F13
SSMI
F14
N17
F15
N15
GPM developed the concept of a unified U.S. algorithm that takes advantage of
• Kalman Filter CMORPH (lagrangian time interpolation) – NOAA
• PERSIANN with Cloud Classification System (IR) – U.C. Irvine
• TMPA (inter-satellite calibration, gauge combination) – NASA
• all three have received PMM support
Integrated Multi-satellitE Retrievals for GPM (IMERG)

22. 6. IMERG Design – Notional Requirements
Resolution – 0.05°~0.1° [i.e., roughly the resolution of microwave, IR footprints]
Time interval – 30 min. [i.e., the geo-satellite interval, then aggregated to 3 hr]
Spatial domain – global, initially covering 60°N-60°S
Time domain – 1998-present; later explore entire SSM/I era (1987-present)
Product sequence – early sat. (~4 hr), late sat. (~12 hr), final sat.-gauge (~2 months after month) [more data in longer-latency products] unique in the field
Instantaneous vs. accumulated – accumulation for monthly; instantaneous for half- hour
Sensor precipitation products intercalibrated to TRMM before launch, later to GPM
Global, monthly gauge analyses including retrospective product – explore use in submonthly-to-daily and near-real-time products; unique in the field
Error estimates – still open for definition; nearly unique in the field
Embedded metadata fields showing how the estimates were computed
Operationally feasible, robust to data drop-outs and (strongly) changing constellation
Output in HDF5 v1.8 – compatible with NetCDF4
Archiving and reprocessing for near- and post-RT products; nearly unique in the field

23. 6. IMERG Design – Box Diagram
The flow chart shown is for the final product
• institutions are shown for module origins, but
• package will be an integrated system
• “the devil is in the details”
• (near-)RT products will use a cut-down of this processing
• we should examine the utility of climatological calibrations for the (near-)RT products
UC Irvine
GSFC
CPC
Compute even-odd IR files
(at CPC) 3
IR Image segmentation
feature extraction
patch classification
precip estimation 9
Receive/store
even-odd IR
files 1
Compute IR
displacement vectors 4 11
Recalibrate
precip rate
Build IR-PMW precip calibration 10
Import PMW data;
grid; calibrate; combine 2
Forward/backward
propagation 5
Build Kalman
filter weights 7
Apply Kalman
filter 8
Import mon. gauge;
mon. sat.-gauge
combo.;
rescale short-interval
datasets to monthly
Apply climo. cal. RT
Post-RT 13 12
prototype6

24. 6. IMERGE Design – Multiple Runs
Multiple runs serve different users’ needs for timeliness
• more delay usually yields a better product
• pioneered in TMPA
Early – first approximation;
flood, now-casting users
• current input data latencies at PPS support ~4-hr delay
• truly operational users (< 3 hr) not well-addressed
Late – wait for full multi-satellite; crop, flood, drought analysts
• driver is the wait for microwave data for backward propagation
• expect delay of hr
Final – after the best data are assembled; research users
• driver is precip gauge analysis
• GPCC gauge analysis is finished ~2 months after the month

25. 4. FUTURE – Transitioning from Version 3 to Version 4
Version 3 IMERG is available
• Final Run from mid-March 2014 to February 2016
• Late Run from 7 March 2015
• Early Run from 1 April 2015
Early February 2017: Version 4, first-generation GPM-based IMERG archive, March 2014–present
Mid-2017: Version 5 IMERG, March 2014–present
Late 2017: TRMM V.8/GPM V.5 TRMM/GPM-based IMERG archive, 1998–present
Winter : Legacy TMPA products retired

26. 7. IMERG Implementation – What Next?
The clear goal for Day-1 is operational code meeting GPM deadlines; after that …
• implement a high-latitude scheme
- develop high-latitude precip estimates
- calibration schemes for high-latitude precip estimates
- leo-IR–based displacement vectors
- parallel observation-model combined product
• use sub-monthly (daily, pentad, or dekad) gauge analyses
• refined precipitation type estimates
• alternative scheme for computing displacement vectors
• address cloud growth
• convective/stratiform classification
• address orographic enhancement
• error estimates
- bias and random
- scale and weather regime dependence
- user-friendly formats and cutting-edge science
• intercalibrate across sensors with different capabilities
• revise precipitation gauge wind-loss corrections
science project
possible model input
science project
science project
science project
science project
science project

27. 2. Data Sets – Produced or Used in the Precip Group

28. 2. Data Sets – Data Sets Produced or Used in the Precip Group (cont.)
[1] ftp://ftp-anon.dwd.de/pub/data/gpcc/html/gpcc_monitoring_v4_doi_download.html
[2] ftp://ftp-anon.dwd.de/pub/data/gpcc/html/fulldata_v6_doi_download.html
[3] ftp://ftp-anon.dwd.de/pub/data/gpcc/vasclimo_50y_precip_clim_v1_1.zip
[4] ftp://rsd.gsfc.nasa.gov/pub/1dd-v1.2/
[5] ftp://ftp.cpc.ncep.noaa.gov/precip/GPCP_PEN/
[6] ftp://precip.gsfc.nasa.gov/pub/gpcp-v2.2/psg/
[7] ftp://jsimpson.pps.eosdis.nasa.gov/data/imerg/early/; automatic and free registration required the first time
[8] ftp://jsimpson.pps.eosdis.nasa.gov/data/imerg/late/; automatic and free registration required the first time
[9] ftp://arthurhou.pps.eosdis.nasa.gov/gpmdata/YYYY/MM/DD/imerg/ where YYYY, MM, DD are 4-digit year, 2-digit month, 2-digit day; automatic and free registration required the first time
[10]
[11] ftp://trmmopen.nascom.nasa.gov/pub/merged/calibratedIR/

29. 2. Data Sets – Heritage
The Adjusted GPI (early ‘90’s) led to GPCP
GPCP concepts were first used in TRMM, then blended with new multi-satellite concepts for TMPA
IMERG adds morphing, Kalman smoother, and neural-network concepts
TRMM V4,5 3B42
TRMM 3B42RT
Time
thin arrows
denote
heritage
CMORPH
IMERG late
IMERG final
IMERG early
GPCP V1
SGM
AGPI
TRMM V4,5 3B43
GPCP V2,2.1 SG
GPCP V1,1.1 1DD
GPCP V1,1.1 Pentad
V6,7 TRMM 3B42
V6,7 TRMM 3B43
GPCP V3
PERSIANN
PERSIANN-CCS
KF-CMORPH