1. Rigor is disciplinary…
What words would you use to describe rigor in your discipline?
How would you know a study is more rigorous than another?
Free recall
Quality of Data…

2. Rigor is….
…a system of inquiry for making persuasive or convincing claims and anticipating the consequences of being wrong
…the principles you follow when making study design decisions
…influenced by moral, political, ideological, and technical value commitments
…a way of communicating the trustworthiness of your research findings
As well as … the precision you follow in utilizing techniques

3. Rigor: System of Inquiry
Observation
Interpretation
Quality of data
Evidence
Technique
Quality of reasoning
Other interpretations
Research Question
Theory
Assumptions
Lenses
Previous work

4. Rigor: Education Research Principles
Pose significant question that can be investigated empirically
Link research to relevant theory
Use methods that permit direct investigation
Provide coherent and explicit chain of reasoning
Replicate and generalize across studies
Public disclosure – peer critique and evaluation

5. Rigor: Communicating Quality
Issue
Quantitative
Assess effects of researcher bias
“Objectivity”
Consistency of findings
“Reliability”
Quality of procedures
“Validity”
Relative truth of findings
“Internal Validity”
Applicability of findings beyond study
“External Validity”
Sampling
Random
Qualitative ?

6. Rigor: Communicating Findings
Issue
Quantitative
Assess effects of researcher bias
“Objectivity”
Consistency of findings
“Reliability”
Quality of procedures
“Validity”
Relative truth of findings
“Internal Validity”
Applicability of findings beyond study
“External Validity”
Sampling
Random
Qualitative
“Confirmability”
“Dependability”
“Verification”
“Credibility”
“Transferability”
Purposeful

7. Rigor: Precision to a Technique
Merriam-Webster
1 : manifesting, exercising, or favoring rigor; very strict
2 : scrupulously accurate :PRECISE
MSN Encarta
exacting: extremely precise and exacting
2. logic precise: precise and formalized

8. Pragmatics Trade-off Decisions Available resources vs. Scope
“Richness” vs. Precision

9. Richness vs. Precision Richness Precision
(e.g., credibility, internal validity)
Precision
(e.g., transferability, external validity)

10. High precision, low richness
“Time spent preparing for class”
Percent of students reporting
From the National Survey of Student Engagement we can learn about what students say about their engineering education.
(Some implications)
These findings are also interesting in terms of what engineering students say about what helps them learn…maybe turn this slide into a chart where column 1 is what they say, column 2 is a comment for that issue in terms of whether or not it helps them learn. 6 - 1 1 6 - 2 2 6 - 3 1 - 5 1 1 - 1 5 2 1 - 2 5
> 3
Reported hours per week
National Survey of Student Engagement – 470 schools (2001)

11. High richness, Low precision
“an ability to communicate effectively”
Roger’s view
“At least in my world, you don’t define a problem, come up with a solution, and then publish it out and say this is what we’re going to do. Instead, you come up with a potential solution, and you say to people, what do you think of this? Have you got any ideas? Does this work for you? Does this meet your problem statement, which may be significantly different than mine? It may be that they see something from a different viewpoint. It may be from a financial viewpoint or from a liability viewpoint or from an ability to do it smoothly and consistently in a manufacturing environment. All of those are maybe different viewpoints than strictly an engineering viewpoint.” (Mosborg et al, 2005)

12. “Sort of like the story of the elephant and the blind men
“Sort of like the story of the elephant and the blind men [O]ne grabs the trunk and says, oh, an elephant is like a snake And the fifth one grabs the tail and says, oh, an elephant is like a rope. Well, I mean it’s – they’re all correct. It depends on what hunk of the elephant you grabbed. And it’s the same sort of thing: In order to have successful implementation of a complex project, you can’t look at it from one viewpoint. You have to have multiple viewpoints.”

13. High richness, Low precision
“an ability to communicate effectively”
Communicative expertise in design
Estimate fluently for others
Negotiate must, strong wants, and wants with others
Central to engineering design process (not held off to last phase)
Actively seek multiple viewpoints from others
Substantially different than “oral and verbal communication”
(Mosborg et al, 2005)

14. Rigor and Pragmatics Aspire to high standards
Be open to surprises / unanticipated findings
Think in phases – small steps
Rigor isn’t a mathematical test but a way of making a convincing argument