Evaluating the Evidence Module Seven Evaluating the Evidence In this module we are going to discuss the evaluation of research evidence that you find in your search.

Quick Review of Research Evidence Research evidence may include: Individual outcome studies Program evaluations Meta-analyses Systematic reviews Case Studies ….and not all evidence is created equal As we discussed earlier in the training, research “evidence” may come in many different forms. Now that you have engaged in the search yourself, you may find that you have several different types of studies. The type and number of studies that you identify in your search likely reflects the type of question you have posed and the state of the research in a particular area. If an area of study is relatively new, you may find only a few individual studies that are relatively exploratory in nature. If your question taps into an area of research that has been subject to a substantial amount of study, you are much more likely to identify a systematic review or meta-analysis. Whatever the type of research you found on your search, you are going to have to make some decisions about its quality. This may actually be the most challenging part of the EBP process. Research is complicated, can come in many forms, and varies a great deal in terms of quality. We don’t expect that you will become an expert in research. However, you can become a more informed consumer of research by understanding some of the key ingredients that make a study stronger.

Quality Assessment Examples General indicators of good research Different folks (like Gibbs) have suggested different criteria or tools Different criteria apply to different types of research evidence There are some general indicators of good research evidence that you should keep in mind, and we have provided you with a quick tips handout to help you out. If the research evidence that you find meets these general quality indicators, it may be worth your time to take a deeper look. If not, you may not want to There are different tools that are available to you in order to help guide you through the process of reading and judging the quality of research evidence that you find. Gibbs has produced a number of tools. If you look at the “Evaluating Study Quality” handout, you will see that he has created tools that match the different types of COPES questions and research, which we discussed earlier. If you are, for example, asking a question related to effectiveness, you would use the Quality Study Rating Form (QSRF). This form can be downloaded, printed, and completed using any effectiveness study you have identified. Likewise there is a form that is aimed at studies testing client assessment and risk (CARE), prevention (QSRF-P), Meta-analysis (META) and description and qualitative studies. We have also included a “Appraising Treatment” handout, which is focused on the evaluation of treatment studies (effectiveness or prevention type). Each of these forms can seem intimidating. They are, however, aimed at helping you zero in on the key information that you need to identify in order to better understand the quality of the evidence. To help you out we have provided an EBP glossary, which includes definitions of terms that you may find on the forms or in the research studies that you identify, which you may not be familiar with. We have also uploaded an example of a real article that was identified by a team of practitioners who participated in this training along with two different forms that were used to evaluate the same study as an example. If you are working as a team you will want to determine whether everyone will read all of the articles and review their quality or if each person will read a fraction of the articles and come prepared to present those articles to the group for discussion.

Refinement of Search Plan Any search engines not covered? Any types of evidence missing? Assess progress. Whether you are working as a group or as an individual, you’ll want to take a look at the research evidence that you have found so far and consider whether or not you will keep searching for more evidence or end your search. If you have disappointing results take a look at your search tracking form. Is there a search engine that you did not try? Have you covered different types of evidence such as systematic reviews, meta-analyses as well as individual studies? If you are working as a team you may want to assess as a groups the type and nature of the research that you have thus far identified, set a goal for finding more pieces of evidence. Before beginning the next module you should end your search and amass all of the research evidence that you have thus far identified. You will also want to begin practicing using the evaluation tools. We do not expect that you will always use these tools when reading research, but that using these forms will help you become more adept at identifying the key components of research.