Measuring Technical Progress and Potential Impact in Real Time Presented at the American Evaluation Association Conference 2009 by Jerald Hage, Wilbur Hadden Center for Innovation University of Maryland Gretchen Jordan Sandia National Laboratories

Why be concerned about this issue?  Papers and patents occur to late for effective interventions for research management  Metrics provide opportunities for expanding the number of social and economic benefits  Without better and more timely measures, there will be a considerable backlash against the research stimulus package  Need to shift focus from productivity to innovation Measuring Technical Progress and Potential Impact in Real Time

Current studies : NSF study of 6 morbidities at two Institutes of NIH (200 projects) (focus of this talk) NSF study of 5 research areas in 6 national laboratories N = 72 projects Past studies: Scientific and technological progress at Sandia National Laboratories (22 projects) New products at STAR at NOAA (5 projects)

Review of current cost-benefit studies of health research  Too much emphasis on mortality rates  Too much emphasis on economic returns  Too much emphasis on long time periods  Little consideration of the deconstruction of the streams of research

Correcting the Problems  View health care as a process: - prevention - diagnosis - treatment - post-treatment  Develop various social measures (e.g. reduction in home care, improved quality of life, etc.)  Focus on specific research projects as completed  Isolate core ideas in streams of research

A Framework for Understanding the Impacts of Health Research J. Hage October Health Research Biomedical Clinical Health services Population and Public health Cross cutting research Health Industry Well Being And Economic Prosperity Determinants of health (social, environmental, etc.) Public Information, Groups Knowledge Pool (Population) Health Status and Function Healthcare Appropriateness, Access, etc. Prevention, Diagnosis, Treatment, and Post Treatment Global Research Research Capacity Other Industries Government Research Agenda Interactions/Feedback Our focus for measures of potential impact Modified from CAHS Report on ROI for Health research available at

Measuring Health Outcomes as (percent change in) Prevention effectiveness of prevention intervention (decline in incidence of morbidity) severity of incidence of morbidity Intake and Assessments the speed of diagnosis (reduction in the number of tests and their duration) the accuracy of diagnosis (reduction in false positives or negatives) accuracy of prognosis (duration and quality of life, etc.)

Measuring Social/Economic Benefits Prevention Value of illness days saved from decline in morbidity incidence Value of reduction in cost of treatments for less severe morbidity incidence Intake and Assessments Reduction in the costs of tests for diagnosis Reduction in costs of false positives or negatives Reduction in the costs of futile interventions

Measuring Social/Economic Benefits: Example One In a clinical trial of a new staining technique for earlier diagnosis of metastasis, the treatment group had their sentinel lymph node removed. The treatment group had a much better survival rate after four years and with fewer new tumors Using the treatment groups experience, one takes the number of new patients in the next year and computes the percentage with metastasis that would be found.

Measuring Social/Economic Benefits: Example One Example: diagnostic technique in melanoma Formula for social value: new patients X % metastasis X% mortality = Life years* 53, % 22.5%1,888 * only in fourth year Formula for economic value: Life years X Value of one year =Economic value 1888 $45,000$85 million Does not include the value of reduction in the number of tumors

Measuring Social/Economic Benefits: Example One Assumptions: The percentage of sentinel nodes where metastasis has occurred in the study is the same as in the general population. No confounding differences between the populations of Australia (the majority of the patients), the U.S. or Europe. Reference: Morton et al. September 28, 2006 “Sentinel-Note Biopsy or Nodal Observation in Melanoma” New England Journal of Medicine, 355, 13:

Measuring Social/Economic Benefits: Example Two In a clinical trial of new patients with Alheizmer’s Disease those with depression were given anti- depressant drug The treatment group had a slower on set of Alheizmer’s disease by on average about three months Using the treatment groups’ experience, one takes the number of new patients in the next year and computes the savings in home care

Measuring Social/Economic Benefits: Example Two Metric for Prevention: Alheizmer’s Disease Formula for social value: new patients X % depression X Reduction in hours 454,000 28% 520 per case Or 2,754,237 days of home care Formula for economic value: Value per Value of Hour Total hours $10$66 million Does not include the value of reduced stress

Measuring Social/Economic Benefits: Example Two Assumptions: The percentage of depression among new patients is the same as existed in the study, which to us seems to be an underestimation. For new patients only home care is used, which again is a conservative estimate. Only 40 hours per week of home care for each of the three months or 13 weeks because the new patients do not need constant watching. An average wage of only $10. The assumption of an improvement of three months is based on the conservative estimate of the findings from five drugs that have been approved by the FDA for Alzheimer’s which reduced symptoms for six to 12 months for about 50 percent of the population, i.e. 50 % of six months or three months. Reference: Lu et al., June 16, 2009 “Donepezil delays progression of AD in MCI subjects with depressive symptoms” Neurology, 72, 24,

Measuring Technical Progress in Real Time Next Steps: 1Apply all health outcome and social/economic benefit metrics in six morbidity areas 2Determine what proportion of the funding was completed by NIH 3Determine the stream of development of ideas that led to the outcome

Measuring Technical Progress in Real Time Please contact us if you have questions, suggestions, or opportunities to collaborate. Jerald Hage: phone Wilbur Hadden: phone Gretchen Jordan: phone