1. WRFDA Regression Testing Suite
Michael Kavulich
August 21, 2014

2. WRFDA Regression Testing Suite
The WRFDA Regression Testing Suite (RTS) is designed to detect errors in various parts of the WRFDA system,
The testing system consists of the following:
A main script (regtest.pl)
A test data file (testdata.txt)
A test database (WRFDA-data-EM)
A baseline database (BASELINE.NEW)

3. WRFDA Regression Testing Suite
You can get the scripts and text files from the following repository:
Download it by using “svn checkout”
svn checkout
The test database and baseline database are very large, and must be downloaded from the website:

4. What the test data file looks like
REGTEST]$ vi testdata.txt
###########################################################################################
#ARCH MACHINE NAME SOURCE COMPILER PROJECT QUEUE DATABASE BASELINE
Linux x86_64 yellowstone WORKDIR/ _changes/multi-level_and_gps_fixes/wrfda.tar ifort P caldera WRFDA-data-EM BASELINE.NEW
#INDEX EXPERIMENT TYPE CPU OPENMP PAROPT
afwa_t7_ssmi DVAR serial|dmpar
afwa_t7_ssmi_ DVAR serial|dmpar
ASR_prepbufr DVAR serial|dmpar
ASR_airs DVAR dmpar
cv3_guo DVAR serial|dmpar
cv3_guo_ DVAR serial|dmpar
cwb_ascii DVAR serial|dmpar
cwb_ascii_outerloop_rizvi 3DVAR serial|dmpar
cwb_ascii_thinning DVAR serial|dmpar
iasi_kavulich DVAR dmpar
# lat_lon_outerloops DVAR dmpar
outerloop_bench_guo DVAR serial|dmpar
outerloop_ztd_bench_guo 3DVAR serial|dmpar
radar_meixu DVAR serial|dmpar
radar_meixu_thinning DVAR serial|dmpar
rainfall_outerloops DVAR dmpar
rttov_4dvar_madagascar DVAR dmpar
seviri_varbc VARBC|3DVAR dmpar
sfc_assi_2_outerloop_guo 3DVAR serial|dmpar
t44_liuz DVAR serial|dmpar
t44_prepbufr DVAR serial|dmpar
tut_xinzhang_4dvar DVAR dmpar
tut_xinzhang_fgat FGAT dmpar
tut_xinzhang_obsproc OBSPROC|3DVAR dmpar
tut_xinzhang_rttov_genbe GENBE|3DVAR dmpar
...
The main script reads this file to find the parameters of each test, such as test name, parallelism, number of cores, test type (3DVAR, VARBC, etc.). Other important data such as project number, Yellowstone queue, and locations of the databases are specified here as well
There is a separate section for each compiler/platform combination
To run only specific tests, you can simply comment out a line to omit that test

5. What the test database looks like
REGTEST]$ ls WRFDA-data-EM
afwa_t7_ssmi cwb_ascii_outerloop_rizvi radar_meixu_thinning t44_prepbufr
afwa_t7_ssmi_32 cwb_ascii_thinning rainfall_outerloops tut_xinzhang_4dvar
ASR_airs iasi_kavulich rttov_4dvar_madagascar tut_xinzhang_fgat
ASR_prepbufr lat_lon_outerloops seviri_varbc tut_xinzhang_obsproc
cv3_guo outerloop_bench_guo sfc_assi_2_outerloop_guo tut_xinzhang_rttov_genbe
cv3_guo_ outerloop_ztd_bench_guo share wind_sd
cwb_ascii radar_meixu t44_liuz
Each of those is a directory containing the files needed as input for the section of WRFDA being tested. For 3DVAR tests this can be as simple as a first guess, be.dat, and namelist.input, but for others there can be many more

6. What the baseline database looks like
REGTEST]$ ls BASELINE.NEW
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi_32.dmpar.gfortran
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi_32.dmpar.ifort
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi_32.dmpar.pgi
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi_32.serial.gfortran
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi_32.serial.ifort
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi_32.serial.pgi
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi.dmpar.gfortran
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi.dmpar.ifort
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi.dmpar.pgi
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi.serial.gfortran
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi.serial.ifort
wrfvar_output.Linux.yellowstone.afwa_t7_ssmi.serial.pgi
wrfvar_output.Linux.yellowstone.ASR_airs.dmpar.gfortran
wrfvar_output.Linux.yellowstone.ASR_airs.dmpar.ifort
wrfvar_output.Linux.yellowstone.ASR_airs.dmpar.pgi
...
Each of those is a wrfvar_output file to be compared. There is one baseline file for each test, parallelism, and compiler combination.

7. What the test looks like
Experiment Paropt Job type CPU_MPI Status Walltime(s) Compare ================================================================================================= ASR_airs dmpar 3DVAR 16 running 0 -- ASR_prepbufr dmpar 3DVAR 16 running 0 -- ASR_prepbufr serial 3DVAR 16 pending 0 -- cwb_ascii dmpar 3DVAR 4 done 27 match cwb_ascii serial 3DVAR 4 running 0 -- cwb_ascii_outerloop_rizvi dmpar 3DVAR 4 running 0 -- cwb_ascii_outerloop_rizvi serial 3DVAR 4 pending 0 -- cwb_ascii_thinning dmpar 3DVAR 4 done 32 match cwb_ascii_thinning serial 3DVAR 4 pending 0 -- iasi_kavulich dmpar 3DVAR 16 running 0 -- lat_lon_outerloops dmpar 3DVAR 16 error 6 Output missing outerloop_ztd_bench_guo dmpar 3DVAR 1 done 14 match outerloop_ztd_bench_guo serial 3DVAR 1 done 9 match radar_meixu dmpar 3DVAR 8 done 31 match radar_meixu serial 3DVAR 8 pending 0 -- rainfall_outerloops dmpar 4DVAR 16 running 0 -- rttov_4dvar_madagascar dmpar 4DVAR 64 running 0 -- seviri_varbc dmpar VARBC|3DVAR 16 running 0 -- sfc_assi_2_outerloop_guo dmpar 3DVAR 16 running 0 -- sfc_assi_2_outerloop_guo serial 3DVAR 16 pending 0 -- t44_liuz dmpar 3DVAR 2 done 18 match t44_liuz serial 3DVAR 2 running 0 -- tut_xinzhang_4dvar dmpar 4DVAR 32 running 0 -- tut_xinzhang_fgat dmpar FGAT 16 running 0 -- tut_xinzhang_obsproc dmpar OBSPROC|3DVAR 16 running 0 -- tut_xinzhang_rttov_genbe dmpar GENBE|3DVAR 10 running 0 -- wind_sd dmpar 3DVAR 16 running 0 -- wind_sd serial 3DVAR 16 pending 0 -- dmpar job for outerloop_bench_guo was completed in 41 seconds. Comparing 'outerloop_bench_guo/wrfvar_output.Linux.yellowstone.outerloop_bench_guo.dmpar.gfortran' to '/glade/p/work/kavulich/REGTEST/BASELINE.NEW/wrfvar_output.Linux.yellowstone.outerloop_bench_guo.dmpar.gfortran'

8. When the test is complete
The full test takes about 20 minutes to complete (8-12 minutes compilation, ~5 minutes for running tests)
The compiled code was compiled separately for 3DVAR and 4DVAR, as well as a separate directory for each parallelism option:
WRFDA_3DVAR_serial/
WRFDA_3DVAR_dmpar/
WRFDA_4DVAR_dmpar/
If you chose the option “--upload=yes”, or the source of the code is the repository, the test summary will be uploaded to the website
At the conclusion of the test, you will receive an with the test results

9. Script options
There are many options which can be specified by providing arguments to the main script
--compiler=COMPILER This one is required
--source=SOURCE_CODE.tar This is the location of the source code. Can also specify “--source=SVN” to check out the code from the main WRF repository.
--revision=#### If you use the “SVN” option, you can specify a specific revision number to test
--upload=yes Uploads a test summary to the website upon completion
--exec=yes Instead of compiling the code, use pre-compiled code. Make sure the directory names are correct!
--j=## For parallel compilation, specify the number of processors (default is 4)

10. Other notes
The source code tar file must contain a directory named “WRFDA” that contains the source code
The test removes directories that are removed for the release: chem, hydro, and dyn_nmm
All this information and more can be found in the README file found in the test repository and on the website