1. Qualitative Validity and Trustworthiness

2. Believing what you read?
How can you generalize from a small non random sample?
If someone else did this study, would they get the same result?
How do you know the researcher isn’t biased and just finding what he or she expects to find
If the researcher is the primary instrument for data collection and analysis, how can we be sure the researcher is a valid and reliable instrument?
What are some of the reasons we may reach inaccurate conclusions during our research?

3. Validity and reliability
Validity and reliability are related to the data collection methods and tools employed in the research and to the extent that the researcher has been able to limit any bias in data collection processes.

4. Reliability and Validity
Why do we bother?
Terms used in conjunction with one another
Quantitative Research: R & V are treated as separate terms
Qualitative Research: R & V are often all under another, all encompassing term
Semi-reciprocal relationship

5. Validity Three types of validity measures are
available to the researcher:
content validity
construct validity.

6. Validity Evidence
Content Validity: “that based on expert ratings of the items” in the test
Construct Validity: “that based on the degree to which scores statistically behave as we would expect a measure of information literacy to behave.” (quantitative)

7. Reliability
Reliability is the consistency with which a tool measures what it is intended to.
The researcher is interested in three measures of reliability:
stability of a measure
its internal consistency
its equivalence.

8. Reliability
In statistics or measurement theory, a measurement or test is considered reliable if it produces consistent results over repeated testings.
Refers to “how well we are measuring whatever it is that is being measured (regardless of whether or not it is the right quantity to measure).”

9. How can validity be established?
Quantitative studies:
measurements, scores, instruments used, research design
Qualitative studies:
ways that researchers have devised to establish credibility: member checking, triangulation, thick description, peer reviews, external audits

10. How can reliability be established?
Quantitative studies?
Assumption of repeatability
Qualitative studies?
Reframe as dependability and confirmability

11. Rigour and trustworthiness
Qualitative researchers aim to ensure rigour and trustworthiness.
Four key components are included:
credibility
dependability
confirmability
transferability.

12. Guba & Lincoln
Guba and Lincoln proposed four criteria for judging the soundness of qualitative research and explicitly offered these as an alternative to more traditional quantitatively-oriented criteria
Traditional Criteria for Judging Quantitative Research
Alternative Criteria for Judging Qualitative Research
internal validity
credibility
external validity
transferability
reliability
dependability
objectivity
confirmability

13. Credibility
The credibility criteria involves establishing that the results of qualitative research are credible or believable from the perspective of the participant in the research.
Since from this perspective, the purpose of qualitative research is to describe or understand the phenomena of interest from the participant's eyes, the participants are the only ones who can legitimately judge the credibility of the results.

14. Transferability
Transferability refers to the degree to which the results of qualitative research can be generalized or transferred to other contexts or settings.
the ability of research results to transfer to situations with similar parameters, populations and characteristics
From a qualitative perspective transferability is primarily the responsibility of the one doing the generalizing.

15. Transferability
The qualitative researcher can enhance transferability by doing a thorough job of describing the research context and the assumptions that were central to the research.
The person who wishes to "transfer" the results to a different context is then responsible for making the judgment of how sensible the transfer is.

16. Dependability
The traditional quantitative view of reliability is based on the assumption of replicability or repeatability.
Essentially it is concerned with whether we would obtain the same results if we could observe the same thing twice.
But we can't actually measure the same thing twice -- by definition if we are measuring twice, we are measuring two different things.
In order to estimate reliability, quantitative researchers construct various hypothetical notions (e.g., true score theory) to try to get around this fact.

17. Dependability
The idea of dependability, on the other hand, emphasizes the need for the researcher to account for the ever-changing context within which research occurs.
The research is responsible for describing the changes that occur in the setting and how these changes affected the way the research approached the study.

18. Confirmability
Qualitative research tends to assume that each researcher brings a unique perspective to the study.
Confirmability refers to the degree to which the results could be confirmed or corroborated by others.

19. Confirmability : Strategies
The researcher can document the procedures for checking and rechecking the data throughout the study.
Another researcher can take a "devil's advocate" role with respect to the results, and this process can be documented.
The researcher can actively search for and describe and negative instances that contradict prior observations.
And, after he study, one can conduct a data audit that examines the data collection and analysis procedures and makes judgements about the potential for bias or distortion.

20. Triangulation
Triangulation is a method used by qualitative researchers to check and establish validity in their studies by analyzing a research question from multiple perspectives.
Patton (2002) cautions that it is a common misconception that the goal of triangulation is to arrive at consistency across data sources or approaches; in fact, such inconsistencies may be likely given the relative strengths of different approaches.
these inconsistencies should not be seen as weakening the evidence, but should be viewed as an opportunity to uncover deeper meaning in the data.

21. Types of Triangulation
Data triangulation
Investigator triangulation
Theory triangulation
Methodological triangulation
Environmental triangulation

22. Data Triangulation
using different sources of information in order to increase the validity of a study.
Eg. these sources are likely to be stakeholders in a program—participants, other researchers, program staff, other community members, and so on.
In-depth interviews could be conducted with each of these groups to gain insight into their perspectives on program outcomes.

23. Data Triangulation
During the analysis stage, feedback from the stakeholder groups would be compared to determine areas of agreement as well as areas of divergence.
data triangulation is the most popular triangulation because it is the easiest to implement

24. Investigator Triangulation
Investigator triangulation involves using several different investigators in the analysis process.
Typically, this manifests as an evaluation team consisting of colleagues within a field of study wherein each investigator examines the program with the same qualitative method (interview, observation, case study, or focus groups).
The findings from each evaluator would then be compared to develop a broader and deeper understanding of how the different investigators view the issue.

25. Investigator Triangulation
If the findings from the different evaluators arrive at the same conclusion, then our confidence in the findings would be heightened.
While this is an effective method of establishing validity, it may not always be practical to assemble different investigators given time constraints and individual schedules.

26. Theory Triangulation
involves the use of multiple perspectives to interpret a single set of data.
Unlike investigator triangulation, this method typically entails using professionals outside of a particular field of study.
One popular approach is to bring together people from different disciplines; however, individuals within disciplines may be used as long as they are in different status positions.

27. Theory Triangulation
In theory, it is believed that individuals from different disciplines or positions bring different perspectives.
Therefore if each evaluator from the different disciplines interprets the information in the same way, then validity is established.
this method can be time-consuming and may not be feasible in all situations.

28. Methodological Triangulation
involves the use of multiple qualitative and/or quantitative methods to study the program.
For example, results from surveys, focus groups, and interviews could be compared to see if similar results are being found.
If the conclusions from each of the methods are the same, then validity is established.
While this method is popular, it generally requires more resources. Likewise, it requires more time to analyze the information yielded by the different methods.

29. Environmental Triangulation
This type of triangulation involves the use of different locations, settings, and other key factors related to the environment in which the study took place, such as the time, day, or season.
The key is identifying which environmental factors, if any, might influence the information that is received during the study.
These environmental factors are changed to see if the findings are the same across settings. If the findings remain the same under varying environmental conditions, then validity has been established.  
Unlike the other types of triangulation, environmental triangulation cannot be used in every case. It is only used when it is likely that the findings may be influenced by environmental factors.

30. Advantages of Triangulation
 “increasing confidence in research data, creating innovative ways of understanding a phenomenon, revealing unique findings, challenging or integrating theories, and providing a clearer understanding of the problem” (Thurmond, 2001, p. 254). These benefits largely result from the diversity and quantity of data that can be used for analysis.

31. Advantages of Triangulation
 For example,
Burr (1998) used multiple triangulations to obtain a more comprehensive view of family needs in critical care. Through the use of questionnaires and selective participant interviews, this researcher found that family members who were interviewed found the sessions therapeutic, but those who were not interviewed could only communicate their frustrations on questionnaires (Thurmond, 2001, p. 254).
Thus, using interviews as well as questionnaires added a depth to the results that would not have been possible using a single-strategy study, thereby increasing the validity and utility of the findings.

32. Disadvantages of Triangulation
One of the primary disadvantages of triangulation is that it can be time-consuming.
Collecting more data requires greater planning and organization—resources that are not always available to lead researchers
Other disadvantages include the “possible disharmony based on investigator biases, conflicts because of theoretical frameworks, and lack of understanding about why triangulation strategies were used” (Thurmond, 2001).

33. Strategies for Greater Consistency
Member checks
Peer/colleague examination
Statement of researchers’ experiences and assumptions
Submersion/Engagement in the research situation – enough period of time
Audit-trail – how data were collected, how categories were derived, how decisions were made