1. Grace Peng Software Engineer, CISL/DSS
NCAR/CISL/DSS
Datasets and Services available from the Research Data Archive (RDA) at NCAR
Grace Peng Software Engineer, CISL/DSS
WRFDA Tutorial
July 26, 2017
Good morning. I’m Grace Peng, one of the data specialists at the NCAR Research Data Archive and
I’ve been invited here to tell you about some of our data sets and services that are useful for WRFDA users

2. Road Map Overview of the RDA DA-related datasets overview
Dataset access options
Datasets in more detail
RDA user support
I’ll start with a brief overview about the RDA and some datasets we believe will be most useful for WRFDA users.
I’ll introduce you to a few of our data access options.
Then I’ll go into more depth about the datasets mentioned earlier
I’ll close with some of the user support services we provide.

3. Highlights Overview of the RDA DA-related datasets overview
Dataset access options
Datasets in more detail
RDA user support
First, a little bit about us

4. Overview of the RDA NCAR’s Research Data Archive Purpose Collections
Established in the 1960s
Purpose
Support climate & weather research at NCAR and UCAR universities with reference datasets
Collections
Ocean & atmospheric observations, analyses,
reanalyses, operational NWP products
600+ datasets, 8M files, 1.5 PB
70+ datasets growing daily-monthly
Free and open access
Worldwide usage
Science educated staff
One of many data assets at NCAR
The RDA is a part of NCAR’s Computational & Information Systems Laboratory, formerly the Scientific Computing Division
We’ve been here helping people with data since the 1960s.
We are just one of the data assets at NCAR; each lab runs their own and we are the one for CISL.
We maintain a large and growing collection of data useful to a broad user base.
We offer free and open-access data to users around the world
Registration is free and required.
Our science-trained staff are cross-trained in data engineering and diverse disciplines including
atmospheric science, physics, oceanography, mathematics and civil and electrical engineering

5. Highlights Overview of the RDA DA-related datasets overview
Dataset access options
Datasets in more detail
RDA user support
We’ve selected a list of data sets that we believe are useful for WRFDA, but it
is not an exhaustive list. Please visit our website to explore more.
We offer many ways to search and browse for data.

6. DA Related Datasets at rda.ucar.edu
NCEP Climate Forecast System, Full Ingest Data (ds099.0)
NCEP ADP Global Upper Air and Surface Weather Observations (PREPBUFR format), May 1997 – Continuing (ds337.0)
NCEP ADP Global Upper Air Observational Weather Data, October 1999 – continuing (ds351.0)
NCEP ADP Global Surface Observational Weather Data, October 1999 – continuing (ds461.0)
NCEP GDAS Satellite Data April 2004 – continuing (ds735.0)
These are the five observational RDA datasets that we believe will be of most interest to the WRFDA community.
I will discuss them in more detail later.
We give each dataset a unique ID #s or call number that never changes.
They have no specific meaning, but
are useful for locating the data on disk or on the web.
But don’t worry if you can’t remember them today. We offer free-text and faceted searches for data.
All of these come from NOAA’s NCEP, which means they are free for everyone and come with no usage restrictions
All are global datasets and can be used in DA. The are also used by the GSI user community
The second one can also be used for model verification.

7. DA Related Datasets Obs data that can be used for verification
NCEP ADP Global Upper Air and Surface Weather Observations (PREPBUFR format), May 1997 – Continuing (ds337.0)
Data can be converted to DTC Model Evaluation Tool (MET) compliant NetCDF format through the “Get a Subset” web interface
We offer translation of some of our datasets if a broad community of users need that.
For instance, we offer translation of NCEP ADP Global… to NetCDF for use in
Developmental Testbed Center’s MET tools

8. 24 Reanalysis dataset collections 400 TB of disk accessible data
DA Related Datasets
Reanalysis data produced by data assimilation
Title
Spatial Coverage
Temporal Coverage
NCEP/NCAR (NNR)
Global (T62, 28 levels)
NCEP Climate Forecast System (CFSR)
Global (T382, 64 levels)
present
NCEP North American Regional Reanalysis (NARR)
North American (32km, 45 levels)
NCAR Global Climate Four-Dimensional Data Assimilation Hourly (CFDDA)
Global (40km, 28 levels)
Arctic System (ASR)
Northern Hemisphere (30km, 71 levels)
NOAA/CIRES Twentieth Century Version 2c
Japanese 25 Year (JRA-25)
Global (T106, 40 levels)
Japanese 55 Year (JRA-55)
Global (T319, 60 levels)
ECWMF-40 (ERA-40)
Global (T159, 60 levels)
ECMWF-Interim (ERA-Interim)
Global (T255, 60 levels)
ECMWF 20th Century (ERA-20C)
Global (T159, 91 levels)
24 Reanalysis dataset collections
400 TB of disk accessible data
We also offer many reanalyses that you can use for model verification with MET tools.
We’re known around the world for providing 24 reanalysis collections in one easy to use portal.
Most of them are on disk for instant download.
Many of them can also be used to initialize WRF.
(Jake asked for reanalysis data sets that ingest data)

9. Highlights Overview of the RDA DA-related datasets overview
Dataset access options
Datasets in more detail
RDA user support
We offer many ways to access our data.
I’m not going to get into our machine-to-machine communication methods today.

10. Data Access Options
This is what one of our dataset home pages looks like.
Wherever you are for a dataset, you will always see the tabs
Click on the Data Access tab to be taken to our data access matrix.

11. Data Access Options The options may vary slightly between datasets.
In general, the publically available options are on the left and the NCAR internal options are on the right
I’ll go into more detail about the options in later slides
Select an option.
From the Web File listing, you can download whole files or, in some cases, perform a
Faceted and spatial search to get a subset.
You can use Request Globus Transfer to initiate drag and drop downloads of whole files
Or range of files via GridFTP
Even if there isn’t a Request Globus Transfer option in the matrix, you can see one later.
The Get a subset column is available, often with special data services on top of subsetting.
Finally, if you have an NCAR internal account, you can directly access our data on /glade
And /hpss

12. Data Access Options Direct File Transfer
HTTP
GridFTP (Globus Online)
Subset/Format conversion requests (select datasets)
Web interface
Download via Globus or scripts
CISL HPC account –Direct file access from disk
U.S. University users
Most of our most popular and recent data are cross-archived on our webserver for immediate access via HTTP and GridFTP
We recommend setting up Globus, a highly robust and fast GridFTP implementation.
We also recommend using our custom data subsetting services to reduce data volumes.
The custom data products can be downloaded using Globus (the recommended way) or our auto-generated csh or perl scripts that rely on wget
Those of you with NCAR accounts can also directly access the web files off /GLADE.

13. RDA User Services Data Transfer Options Globus Advantages
Reliable, Secure, High-performance file transfer
“Fire and Forget”
Automatic fault recovery
Powerful GUI, APIs and CLI
Integrated with RDA file lists
Authenticate with RDA identity
Globus.org
Globus = “Personal download assistant”
It was originally created for use at DOE, which moves large volumes of data between labs.
It’s being used at more and more labs around the world.
It’s free for personal use and we highly recommend it.
You set up a Globus endpoint on your end just one time.
Login using your RDA credentials to download a personal cookie, and
Your personal data assistant or genie does most of the work.
Thereafter, data transfers are ’fire and forget’
You can select a range of RDA whole files and where you want them to go
using an intuitive interface.
The files will appear on your endpoint.
You’ll also receive an when the transfer is complete
You can learn more and Download it from globus.org

14. Highlights Overview of the RDA DA-related datasets overview
Dataset access options
Datasets in more detail
RDA user support
Now let’s look at each of the five datasets I mentioned earlier in more detail

15. NCEP Climate Forecast System (CFS), Full Ingest Data (ds099
Gridded analyses: T, Precip Rate, Snow (Cover, Depth), Ice Concentration, Land Cover
Obs data ingested into CFS: satellite, surface, upper air
Entire or subset BUFR files
All times
Organized in 6-hourly files by type
Ds099.0 has the daily data ingested into CFS. It has gridded analyses as well as satellite, aircraft, conventional data.
It comes in BUFR format, so you may need to do a little bit of work to use the non-satellite data.
However, we do offer translation to ASCII, which you can convert easlity to LITTLE_R

16. NCEP ADP Global Upper Air and Surface Weather Observations (PREPBUFR format), May 1997 – Continuing (ds337.0)
Can be used for DA and verification
T, TD, P, GeoZ, winds
Surface data: land, marine
Upper air: radiosonde, pibal, aircraft
Translation into NetCDF for use in Model Evaluation Tool (MET) or ASCII
Entire files or subset
Ds337.0 contains quality controlled PREPBUFR data prepared for use in NCEP’s myriad models.
It can be used as-is in PREPBUFR for DA.
The NetCDF translation can also be used for model verification

17. NCEP ADP Global Upper Air Observational Weather Data, October 1999 – continuing (ds351.0)
T, TD, P, GeoZ, Winds
Upper Air: raob, pibal, dropsonde, aircraft, satellite cloud-motion winds
Entire files or subset in BUFR or ASCII
All times, variable frequency
Daily or sonde-only data files
Entire global daily LITTLE_R translations
The NCEP ADP… are the global upper air observations collected by NCEP off the Global Telecommunications System (GTS)
They encompass all times but frequency varies by data source
Most importantly, we’ve already translated them into LITTLE_R for you.

18. NCEP ADP Global Surface Observational Weather Data, October 1999 – continuing (ds461.0)
T, TD, P, GeoZ, Winds, Precip, Visibility, Cloud & Wave Info
Surface: fixed & mobile land and buoys, ships
Entire files or subset in BUFR or ASCII
All times, variable frequency
Daily data files
Entire global daily LITTLE_R translations
These are the surface obs analog to the prior dataset
The download options are very similar.
Again, note that this is already translated into LITTLE_R for you

19. NCEP GDAS Satellite Data April 2004 – continuing (ds735.0)
Satellite-related data ingested into GDAS
Radiances, Tb, Refractivity (GPSRO & GPSIPW), Surface Winds, WV, TPW, P, GeoZ, O3, Imagery, Cloud Frequency
Entire BUFR files for each type
Organized in 6-hourly files
Files may contain data from multiple satellites
Text files
TCVITALS, ABIAS, ABIAS_PC
Last, but not least, there are the satellite and remotely sensed obs ingested into NCEP’s GFS GDAS system,
This is a collection of over 24 types of data files useful for WRFDA and GSI
They are tied to the GFS data assimilation cycle and organized for each 6-hourly analysis cycle
With a time window centered around the analysis time.
So be sure to request/download times bracketing your model’s run times
OK, we’re going to have a pop quiz about each data set

20. Data set highlights CFS Analyses & Ingested Obs
ADP UA/Surface PREPBUFR
ADP Upper Air
ADP Surface
GDAS Ingested Satellite
Dataset ID #
ds099.0
ds337.0
ds351.0
ds461.0
ds735.0
Date Range
1997 – con’t
1999 – con’t
2004 – con’t
Native Format
BUFR
PREPBUFR
LITTLE_R translation
N/A
Daily Global File
‘Get a Subset’ output options
ASCII
NetCDF*
‘Faceted Browse’ subset
Yes**
Yes
Entire File
Just kidding.
I’ve summarized the key information about each of these datasets
Because you’ll need to get the data into LITTLE_R, we’ve already done the work for you with the surface and UA data.
but you can only download whole global daily files.
You can prepare your own LITTLE_R from ADP PREPBUFR, ds337.0, with prepbufr2littleR.exe
The PREPBUFR dataset can also be translated to NetCDF for use with MET (Model Evaluation Tool)
** Spatial subset available for obs only. Gridded data comes in global daily files.

21. Highlights Overview of the RDA DA-related datasets overview
Dataset access options
Datasets in more detail
RDA user support
I’ll close with a plug about our user support
There are many data centers that serve data.
We’re a research data archive embedded in an education and research organization.

22. RDA user support: Advise user of best data set collection for research purpose
As such, we provide both data and guidance.
Each dataset is assigned to a data specialist with relevant scientific expertise.
They prepare a personalized guide to each dataset with all the information that they anticipate users need to get started.
If you have any questions that aren’t answered in our help materials, we tell you who to contact at the top of every page.
In this case, it’s Tom, the hipster over on the right.

23. RDA user support: Advise user of best data set collection for research purpose
As such, we provide both data and guidance.
the Climate Data Guide is a resource where experienced researchers give opinions on which
Datasets they find most useful .
They give guidance about the strengths and weaknesses of the datasets.
Where applicable, our data sets are cross linked to the expert guidance in the Climate Data Guide.
If a dataset is linked to the CDG, you should click on the link and read it before starting to use the data.
If you have any questions about which data set is most appropriate for your research goals, consult with us.
We provide this service frequently for our users.

24. RDA user support http://ncarrda.blogspot.com/ @ncar_rda
We announce news on FB, Twitter and on our blog.
We post longer items such as new dataset announcements, backgrounders and tutorials on our blog.
Tutorial videos are posted to the NCAR CISL channel, and are also embedded on our blog.
Many datasets have helpful blog posts with background information, news and tutorials.
You can find all relevant ones by clicking the blog link at the bottom of the dataset’s home page

25. RDA user support Dataset Consultant listed on each dataset homepage
General help
Blog posts: ncarrda.blogspot.com
User video tutorials
If you have a question that is not answered in the data web pages or our blog, contact us by or phone using the link or # listed here.
If you have a general Q or are not sure which dataset is appropriate, use rdahelp

26. RDA user support: Server generated data citation
Access from
user dashboard or
data set page
Maintain user access history, citation recall at any time
A keystone of reproducible science is to cite your data sources. We make this easy.
We provide a data citation widget that generates a citation in many popular styles for easy cut and paste into your publication.
You can access this from your user dashboard or from the dataset main page.

27. Key Takeaways
The RDA offers many data services to help you be more productive
Information-rich home web page for each data set
Science-trained staff assigned to assist with each data set
Our collection is always evolving
We are open to adding data or data services that have broad user bases
Cite your data and where you got it
I’ve given you a glimpse of some of our data services, but we offer many more.
Do browse our collections.
We try to anticipate what information users want to know.
But, if you have additional questions that aren’t answered on the data web pages, feel free to contact the data specialist by or phone
Each of us has relevant experience. For instance, I’m responsible for our satellite datasets.
I have a PhD in physics and worked in satellite meteorology for the Air Force before joining NCAR.
We are not a fossil. If you have an idea of something we should offer, tell us about it and who you think would benefit from it.
If it is useful to a broad community, we may offer it in the future.
Lastly, cite your data and where you got it. This is a keystone of reproducible science.

28. Questions? https://rda.ucar.edu rdahelp@ucar.edu Grace Peng
Questions? Visit us.
ESA Sentinal-1 Satellite launched 2014

29. Backup Globus Options Slides
Select and download entire files in background
Receive when complete
Make custom data request
When arrives about job completion, select Globus
Fire and forget
Or watch progress as files appear at your local endpoint

30. Data Access Options Select an option.
Notice you can get a subset or see the full file listings.
Even if there isn’t a Request Globus Transfer option in the matrix, you can see one later.
Get a subset column is available special data services on top of subsetting.
On many RDA data sets you can also get a subset by using the Web File Listing’s Faceted search option.

31. Download options
Once your custom data product is ready, you’ll be ed a link to a page that looks like this.
You can download the data with Globus, or use our auto-generated csh or perl script.

32. If you select Globus, then click on this link to initiate it.

33. Then select the location where you want to transfer your files.