1. Assessing Uncertainty in FVS Projections Using a Bootstrap Resampling Method by Tommy F. Gregg Region 6 NR/FID and Susan Stevens Hummel PNW Research Station

2. Objective Develop process for assessing uncertainty in model projections. Develop process for assessing uncertainty in model projections. Create a program compatible with SUPPOSE and FVS to implement the process. Create a program compatible with SUPPOSE and FVS to implement the process.

3. Criteria for the Process Must be statistically valid. Must be statistically valid. Must be feasible given current technology. Must be feasible given current technology.

4. “…The oldest and simplest device for misleading folks is the barefaced lie. A method that is nearly as effective and far more subtle is to report a sample estimate without any indication of its reliability…” ( Frank Freese 1967 ) Why is this important?

5. Available variance estimators Simple random sample: Simple random sample: Stratified random sample Stratified random sample Double sampling Double sampling Multi-stage or cluster sample Multi-stage or cluster sample..and many more!..and many more!

6. Problem with available variance estimators They do not apply to model projections over time. They do not apply to model projections over time. They can not be used for making inferences about model complex results. They can not be used for making inferences about model complex results.

7. Means & Confidence Limits from a set of independent samples may look like this: 

8. Sample data are often projected through time without regard to sampling uncertainty x Time 

9. Bootstrap Resampling Method Developed in the 1980s (Efron), based on classical statistical theory from the 1930s. Developed in the 1980s (Efron), based on classical statistical theory from the 1930s. Computer intensive method for assessing uncertainty. Computer intensive method for assessing uncertainty. Used for complex problems in many fields. Used for complex problems in many fields.

10. Why Bootstrap Model Projections? Bootstrapping allows us to substitute computational power for classical statistical analysis. may be the only technical method for assessing uncertainty in model projections.Bootstrapping may be the only technical method for assessing uncertainty in model projections. is doable.Bootstrapping is doable.

11. Stand with 31 stand-exam inventory plots What is the Bootstrap Resampling Method?

12. The Process for Generating Monte Carlo Bootstrap Samples (1) Randomly select a sample of size n with replacement from the original empirical distribution (where n is the sample size for that original sample). (1) Randomly select a sample of size n with replacement from the original empirical distribution (where n is the sample size for that original sample). (2) Compute a bootstrap mean using the bootstrap sample. (2) Compute a bootstrap mean using the bootstrap sample. (3) Repeat steps 1 and 2 k times. (3) Repeat steps 1 and 2 k times.

13. An Example of a Bootstrap Sampling An Example of a Bootstrap Sampling

14. Generating a set of Nonparametric Bootstrap Confidence Intervals Confidence intervals are obtained from the Monte Carlo bootstrap distribution. Confidence intervals are obtained from the Monte Carlo bootstrap distribution. They are taken at appropriate percentiles from a sorted list of the k bootstrap means. They are taken at appropriate percentiles from a sorted list of the k bootstrap means. For example, a two-sided approximate 95% confidence interval about mean would be extracted at the 2.5 and 97.5 percentile. For example, a two-sided approximate 95% confidence interval about mean would be extracted at the 2.5 and 97.5 percentile.

15. LIST OF 200 Sorted BootStrap Means: LIST OF 200 Sorted BootStrap Means: ( 0) ( 1) ( 2) ( 3) ( 4) ( 5) ( 6) ( 7) ( 8) ( 9) ( 0) ( 1) ( 2) ( 3) ( 4) ( 5) ( 6) ( 7) ( 8) ( 9) 0 : - 1207.0 1204.0 1191.0 1181.0 1178.0 1173.0 1159.0 1117.0 1116.0 0 : - 1207.0 1204.0 1191.0 1181.0 1178.0 1173.0 1159.0 1117.0 1116.0 10 : 1082.0 1079.0 1072.0 1071.0 1067.0 1066.0 1065.0 1056.0 1051.0 1047.0 10 : 1082.0 1079.0 1072.0 1071.0 1067.0 1066.0 1065.0 1056.0 1051.0 1047.0 20 : 1042.0 1040.0 1032.0 1031.0 1031.0 1030.0 1018.0 1017.0 1013.0 1010.0 20 : 1042.0 1040.0 1032.0 1031.0 1031.0 1030.0 1018.0 1017.0 1013.0 1010.0 30 : 1010.0 1008.0 1004.0 1003.0 995.0 994.0 992.0 985.0 984.0 983.0 30 : 1010.0 1008.0 1004.0 1003.0 995.0 994.0 992.0 985.0 984.0 983.0 40 : 982.0 981.0 981.0 975.0 975.0 969.0 968.0 967.0 967.0 965.0 40 : 982.0 981.0 981.0 975.0 975.0 969.0 968.0 967.0 967.0 965.0 50 : 962.0 962.0 960.0 960.0 956.0 955.0 953.0 951.0 950.0 949.0 50 : 962.0 962.0 960.0 960.0 956.0 955.0 953.0 951.0 950.0 949.0 60 : 948.0 945.0 943.0 942.0 942.0 941.0 936.0 933.0 928.0 927.0 60 : 948.0 945.0 943.0 942.0 942.0 941.0 936.0 933.0 928.0 927.0 70 : 926.0 925.0 924.0 924.0 924.0 923.0 922.0 919.0 919.0 919.0 70 : 926.0 925.0 924.0 924.0 924.0 923.0 922.0 919.0 919.0 919.0 80 : 917.0 916.0 913.0 913.0 911.0 905.0 899.0 898.0 898.0 896.0 80 : 917.0 916.0 913.0 913.0 911.0 905.0 899.0 898.0 898.0 896.0 90 : 894.0 892.0 892.0 892.0 891.0 890.0 889.0 889.0 887.0 885.0 90 : 894.0 892.0 892.0 892.0 891.0 890.0 889.0 889.0 887.0 885.0 100 : 885.0 885.0 884.0 884.0 883.0 880.0 879.0 879.0 878.0 878.0 100 : 885.0 885.0 884.0 884.0 883.0 880.0 879.0 879.0 878.0 878.0 110 : 877.0 876.0 876.0 873.0 872.0 872.0 870.0 867.0 865.0 864.0 110 : 877.0 876.0 876.0 873.0 872.0 872.0 870.0 867.0 865.0 864.0 120 : 863.0 861.0 858.0 858.0 857.0 855.0 854.0 853.0 852.0 852.0 120 : 863.0 861.0 858.0 858.0 857.0 855.0 854.0 853.0 852.0 852.0 130 : 852.0 851.0 847.0 846.0 846.0 845.0 844.0 841.0 840.0 840.0 130 : 852.0 851.0 847.0 846.0 846.0 845.0 844.0 841.0 840.0 840.0 140 : 840.0 837.0 835.0 834.0 833.0 830.0 830.0 827.0 823.0 823.0 140 : 840.0 837.0 835.0 834.0 833.0 830.0 830.0 827.0 823.0 823.0 150 : 823.0 818.0 817.0 813.0 810.0 809.0 806.0 805.0 805.0 805.0 150 : 823.0 818.0 817.0 813.0 810.0 809.0 806.0 805.0 805.0 805.0 160 : 803.0 801.0 800.0 799.0 796.0 795.0 789.0 781.0 778.0 777.0 160 : 803.0 801.0 800.0 799.0 796.0 795.0 789.0 781.0 778.0 777.0 170 : 777.0 770.0 769.0 767.0 765.0 761.0 757.0 755.0 749.0 749.0 170 : 777.0 770.0 769.0 767.0 765.0 761.0 757.0 755.0 749.0 749.0 180 : 748.0 747.0 747.0 745.0 741.0 739.0 734.0 721.0 720.0 713.0 180 : 748.0 747.0 747.0 745.0 741.0 739.0 734.0 721.0 720.0 713.0 190 : 709.0 709.0 703.0 702.0 702.0 688.0 680.0 670.0 654.0 623.0 190 : 709.0 709.0 703.0 702.0 702.0 688.0 680.0 670.0 654.0 623.0 200 : 588.0 200 : 588.0

16. Program Description ( FVS2Boot.exe ) Builds bootstrap files from existing FVS input treelist files. Builds bootstrap files from existing FVS input treelist files. Interfaces seamlessly with SUPPOSE. Interfaces seamlessly with SUPPOSE. Processes FVS output: Processes FVS output: –.sum files –.out files Displays results. Displays results.

17. There are two sources of variation that can be used to characterize uncertainty in FVS The stochastic components in FVS (accessible to the user through the RANNSEED keyword). We call this FVS-Mean and FVS Prediction Interval (FVS-PI). The stochastic components in FVS (accessible to the user through the RANNSEED keyword). We call this FVS-Mean and FVS Prediction Interval (FVS-PI). Variation among sampling units. We call this Sampling Error Prediction Interval (SE-PI). Variation among sampling units. We call this Sampling Error Prediction Interval (SE-PI).

18. FVSBoot Main Menu

19. Open FVS Files and Directories

20. Bootstrap Option Screen

21. Suppose Main Menu:

22. Suppose Select Stands

23. Run FVS on all Bootstrap Samples

24. Select FVS Variables for Display

25. FVS Bootstrap Output Screen:

26. DISPLAY OF FVS MODEL STOCASTIC BEHAVIOR Model output data from FVS = Cycle( 5), TPA FVS-PI Mean = 467.55 FVS-PI Mean = 467.55 Number of FVS runs = 200 Number of FVS runs = 200 Standard Deviation = 1.48 Standard Deviation = 1.48 Median = 468.00 Median = 468.00 Max= 471.00 Max= 471.00 Min= 463.00 Min= 463.00 Range= 8.00 Range= 8.00 Frequency distribution for ( 201 ) bootstrap samples -------- --------- ---+------------------------------------------------------------ 1 463.50 2 |II 1 463.50 2 |II 2 464.50 14 |IIIIIIIIIIIIII 2 464.50 14 |IIIIIIIIIIIIII 3 465.50 32 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 3 465.50 32 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 4 466.50 49 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 4 466.50 49 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 5 467.50 53 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 5 467.50 53 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 6 468.50 34 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 6 468.50 34 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 7 469.50 11 |IIIIIIIIIII 7 469.50 11 |IIIIIIIIIII 8 470.50 6 |IIIIII 8 470.50 6 |IIIIII -------- --------- ---+------------------------------------------------------------ -------- --------- ---+------------------------------------------------------------

27. FVS PREDICTION INTERVAL FVS-PI CAUSED BY SAMPLE VARIATION Data from FVS Variable = Cycle( 5), TPA Data from FVS Variable = Cycle( 5), TPA FVS-PI Mean= 467.55 Sampling Error PI: Number of samples = 500 Number of samples = 500 Bootstrap Mean = 464.53 Bootstrap Mean = 464.53 Standard Deviation = 52.86 Standard Deviation = 52.86 Bootstrap Median = 461.00 Bootstrap Median = 461.00 Max outcome = 657.00 Max outcome = 657.00 Min outcome = 318.00 Min outcome = 318.00 Range of outcomes = 339.00 Range of outcomes = 339.00

28. Frequency distribution for ( 500 ) bootstrap samples for "Cycle(5), TPA" from FVS. Interval Midpoints Counts Interval Midpoints Counts -------- --------- ---+------------------------------------------------------------------------------ -------- --------- ---+------------------------------------------------------------------------------ 1 326.48 3 |III 1 326.48 3 |III 2 343.42 4 |IIII 2 343.42 4 |IIII 3 360.38 8 |IIIIIIII 3 360.38 8 |IIIIIIII 4 377.33 16 |IIIIIIIIIIIIIIII 4 377.33 16 |IIIIIIIIIIIIIIII 5 394.27 28 |IIIIIIIIIIIIIIIIIIIIIIIIIIII 5 394.27 28 |IIIIIIIIIIIIIIIIIIIIIIIIIIII 6 411.23 34 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 6 411.23 34 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 7 428.18 57 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 7 428.18 57 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 8 445.13 65 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 8 445.13 65 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 9 462.08 67 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 9 462.08 67 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 10 479.02 59 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 10 479.02 59 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 11 495.98 52 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 11 495.98 52 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 12 512.92 35 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 12 512.92 35 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 13 529.88 31 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 13 529.88 31 |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 14 546.83 20 |IIIIIIIIIIIIIIIIIIII 14 546.83 20 |IIIIIIIIIIIIIIIIIIII 15 563.78 10 |IIIIIIIIII 15 563.78 10 |IIIIIIIIII 16 580.73 4 |IIII 16 580.73 4 |IIII 17 597.67 3 |III 17 597.67 3 |III 18 614.63 2 |II 18 614.63 2 |II 19 631.58 0 | 19 631.58 0 | 20 648.53 2 |II 20 648.53 2 |II -------- --------- ---+-------------------------------------------------------------------------------

29. BOOTSTRAP SAMPLING ERROR PREDICTION INTERVALS (SE-PI): BOOTSTRAP SAMPLING ERROR PREDICTION INTERVALS (SE-PI): Variable Mean Percent Lower Upper Variable Mean Percent Lower Upper ---------------- -------- ----- -------- -------- ---------------- -------- ----- -------- -------- Cycle( 5), TPA 467.55 68 414.00 518.00 Cycle( 5), TPA 467.55 68 414.00 518.00 80 398.00 531.00 80 398.00 531.00 90 381.00 553.00 90 381.00 553.00 95 364.00 571.00 95 364.00 571.00 99 329.00 648.00 99 329.00 648.00 --------------------------------------------------------------------- ---------------------------------------------------------------------