Rigor is disciplinary… What words would you use to describe rigor in your discipline? How would you know a study is more rigorous than another? Free recall Quality of Data…

Rigor is…. …a system of inquiry for making persuasive or convincing claims and anticipating the consequences of being wrong …the principles you follow when making study design decisions …influenced by moral, political, ideological, and technical value commitments …a way of communicating the trustworthiness of your research findings As well as … the precision you follow in utilizing techniques

Rigor: System of Inquiry Observation Interpretation Quality of data Evidence Technique Quality of reasoning Other interpretations Research Question Theory Assumptions Lenses Previous work

Rigor: Education Research Principles Pose significant question that can be investigated empirically Link research to relevant theory Use methods that permit direct investigation Provide coherent and explicit chain of reasoning Replicate and generalize across studies Public disclosure – peer critique and evaluation

Rigor: Communicating Quality Issue Quantitative Assess effects of researcher bias “Objectivity” Consistency of findings “Reliability” Quality of procedures “Validity” Relative truth of findings “Internal Validity” Applicability of findings beyond study “External Validity” Sampling Random Qualitative ?

Rigor: Communicating Findings Issue Quantitative Assess effects of researcher bias “Objectivity” Consistency of findings “Reliability” Quality of procedures “Validity” Relative truth of findings “Internal Validity” Applicability of findings beyond study “External Validity” Sampling Random Qualitative “Confirmability” “Dependability” “Verification” “Credibility” “Transferability” Purposeful

Rigor: Precision to a Technique Merriam-Webster 1 : manifesting, exercising, or favoring rigor; very strict 2 : scrupulously accurate :PRECISE MSN Encarta exacting: extremely precise and exacting 2. logic precise: precise and formalized

Pragmatics Trade-off Decisions Available resources vs. Scope “Richness” vs. Precision

Richness vs. Precision Richness Precision (e.g., credibility, internal validity) Precision (e.g., transferability, external validity)

High precision, low richness “Time spent preparing for class” Percent of students reporting From the National Survey of Student Engagement we can learn about what students say about their engineering education. (Some implications) These findings are also interesting in terms of what engineering students say about what helps them learn…maybe turn this slide into a chart where column 1 is what they say, column 2 is a comment for that issue in terms of whether or not it helps them learn. 6 - 1 1 6 - 2 2 6 - 3 1 - 5 1 1 - 1 5 2 1 - 2 5 > 3 Reported hours per week National Survey of Student Engagement – 470 schools (2001)

High richness, Low precision “an ability to communicate effectively” Roger’s view “At least in my world, you don’t define a problem, come up with a solution, and then publish it out and say this is what we’re going to do. Instead, you come up with a potential solution, and you say to people, what do you think of this? Have you got any ideas? Does this work for you? Does this meet your problem statement, which may be significantly different than mine? It may be that they see something from a different viewpoint. It may be from a financial viewpoint or from a liability viewpoint or from an ability to do it smoothly and consistently in a manufacturing environment. All of those are maybe different viewpoints than strictly an engineering viewpoint.” (Mosborg et al, 2005)

“Sort of like the story of the elephant and the blind men “Sort of like the story of the elephant and the blind men . . . [O]ne grabs the trunk and says, oh, an elephant is like a snake. . . . And the fifth one grabs the tail and says, oh, an elephant is like a rope. Well, I mean it’s – they’re all correct. It depends on what hunk of the elephant you grabbed. And it’s the same sort of thing: In order to have successful implementation of a complex project, you can’t look at it from one viewpoint. You have to have multiple viewpoints.”

High richness, Low precision “an ability to communicate effectively” Communicative expertise in design Estimate fluently for others Negotiate must, strong wants, and wants with others Central to engineering design process (not held off to last phase) Actively seek multiple viewpoints from others Substantially different than “oral and verbal communication” (Mosborg et al, 2005)

Rigor and Pragmatics Aspire to high standards Be open to surprises / unanticipated findings Think in phases – small steps Rigor isn’t a mathematical test but a way of making a convincing argument