Teaching sustainable online research practices across the curriculum: The Q6C Solution Sarah Read (English) Kate Deibel (CS / Education) Tim Wright (History) University of Washington

Computers & Writing 2009: Ubiquitous and Sustainable Computing2 Computers & Writing 2009  Goals:  Transcend traditional boundaries  Academic disciplines  K-12 and higher education  Online and offline  Organizers, attendees, and presenters  School, work, and play  Develop a sustainable perspective on lifelong computing and communication

Computers & Writing 2009: Ubiquitous and Sustainable Computing3 Today’s Plan  Framing Q6C Within Conference Intent  A Taste of Application  What is Q6C?  Development  Overview  Using Q6C  For the Teacher  For the Students (and Teacher)  Conclusion  Discussion and workshop

Computers & Writing 2009: Ubiquitous and Sustainable Computing4 The Issue

Computers & Writing 2009: Ubiquitous and Sustainable Computing5 The Research Scenario You are in a college geology course and have been assigned to brief the class on the Crandall Canyon Mine disaster. Answer the question: Was the cause of the mine collapse geological, and what was it?

First Hit: Why is this not a good source ? The Answer Authoritative Source and Author

Computers & Writing 2009: Ubiquitous and Sustainable Computing7 Questions raised by Q6C  How do researchers know when to stop looking for sources?  What do experienced researchers in a field know about when to stop that newcomers don’t know?  How can newcomers to a field be supported to do research with the savvy of experienced researchers?

Computers & Writing 2009: Ubiquitous and Sustainable Computing8 Q6C: Applied to history

Computers & Writing 2009: Ubiquitous and Sustainable Computing9 Symptoms of a Serious Problem In regards to online sources, students...  …stop too soon when evaluating a source  …fail to keep a skeptical frame of mind  …tend to focus on surface/superficial details  …trust the top search results  …reject Wikipedia outright  …accept Wikipedia outright  …fail to transfer critical skills across domains

Computers & Writing 2009: Ubiquitous and Sustainable Computing10 An Information Literacy Skill  The Goal: Teach students to be savvy, critical consumers of Internet sources for both academic and non-academic purposes  Challenges for Teaching  What skills should be taught?  How can those skills best be conveyed?  What is needed to promote skill transfer across disciplines / outside of the classroom?

Computers & Writing 2009: Ubiquitous and Sustainable Computing11 Today’s Plan  Framing Q6C Within Conference Intent  A Taste of Application  What is Q6C?  Development  Overview  Using Q6C  For the Teacher  For the Students (and Teacher)  Conclusion  Discussion and workshop

Computers & Writing 2009: Ubiquitous and Sustainable Computing12 Previous Work and Research Different rules and expectations for student writing exist across disciplines Internet poses challenges and opportunities for student research Students report IT confidence but overestimate source judging abilities Students report difficulties and overload in conducting research Checklists for evaluating reliability of Internet sources McCarthy (1987) Sorapure et al. (1998) Kvavik (2005) Head & Eisenberg (2009) Many authors

Computers & Writing 2009: Ubiquitous and Sustainable Computing13 The Checklist Authorship  Is there an author? You may need to…  Can you tell whether the author is knowledgeable and credible? If the author's qualifications aren't listed… Sponsorship  What does the URL tell you? The URL ending often specifies the type of group hosting the site: commercial (.com), educational (.edu), nonprofit (.org), … Currency  How current is the site?  How current are the site's links? If many of the links no longer work, the site may be too dated for your purposes. Excerpt from Hacker’s A Pocket Manual of Style (2008)

Computers & Writing 2009: Ubiquitous and Sustainable Computing14 Pitfalls of the Checklist Authorship  Is there an author? You may need to…  Can you tell whether the author is knowledgeable and credible? If the author's qualifications aren't listed… Sponsorship  What does the URL tell you? The URL ending often specifies the type of group hosting the site: commercial (.com), educational (.edu), nonprofit (.org)… Currency  How current is the site?  How current are the site's links? If many of the links no longer work, the site may be too dated for your purposes. Excerpt from Hacker’s A Pocket Manual of Style (2008) INACCURATE:.org has never been restricted to only nonprofits

Computers & Writing 2009: Ubiquitous and Sustainable Computing15 Pitfalls of the Checklist Authorship  Is there an author? You may need to…  Can you tell whether the author is knowledgeable and credible? If the author's qualifications aren't listed… Sponsorship  What does the URL tell you? The URL ending often specifies the type of group hosting the site: commercial (.com), educational (.edu), nonprofit (.org)… Currency  How current is the site?  How current are the site's links? If many of the links no longer work, the site may be too dated for your purposes. Excerpt from Hacker’s A Pocket Manual of Style (2008) Not all domains are regulated Domains reflect only general purposes and not specific pages

Computers & Writing 2009: Ubiquitous and Sustainable Computing16 Pitfalls of the Checklist Authorship  Is there an author? You may need to…  Can you tell whether the author is knowledgeable and credible? If the author's qualifications aren't listed… Sponsorship  What does the URL tell you? The URL ending often specifies the type of group hosting the site: commercial (.com), educational (.edu), nonprofit (.org)… Currency  How current is the site?  How current are the site's links? If many of the links no longer work, the site may be too dated for your purposes. Excerpt from Hacker’s A Pocket Manual of Style (2008) Ignores complexity of web authorship Encourages the usage of titles, degrees, and symbols of authority to determine credibility

Computers & Writing 2009: Ubiquitous and Sustainable Computing17 Pitfalls of the Checklist Authorship  Is there an author? You may need to…  Can you tell whether the author is knowledgeable and credible? If the author's qualifications aren't listed… Sponsorship  What does the URL tell you? The URL ending often specifies the type of group hosting the site: commercial (.com), educational (.edu), nonprofit (.org)… Currency  How current is the site?  How current are the site's links? If many of the links no longer work, the site may be too dated for your purposes. Excerpt from Hacker’s A Pocket Manual of Style (2008) Suggests recent data as being more reliable Update frequency will vary by the type of site

Computers & Writing 2009: Ubiquitous and Sustainable Computing18 Criticisms of the Checklist  Inherent problems  Emphasis on surface features over content  Simplistic yes/no questions with no guidance  Erroneous indicators of credibility  Students fail to develop information literacy skills and critical practices  Meola (2004)  Helms-Park & Stapleton (2006)  Need for better evaluative methods to develop sustained, transferable skills  Sidler (2002)

Computers & Writing 2009: Ubiquitous and Sustainable Computing19 Developing Q6C  Intent:  Develop tools to support instructors in teaching web literacy skills  Guiding Principles:  Emphasize the process of source evaluation, not the end product  Recognize disciplinary differences  Promote student metacognition and transfer of skills across and outside academia  Approach:  Discuss source evaluation across the curriculum  Develop a working model of the evaluation process

Computers & Writing 2009: Ubiquitous and Sustainable Computing20 Discussions about Source Evaluation Research Literature  Ayers (2006)  Britt & Aglinskas (2002)  Head & Eisenberg (2009)  Helms-Park & Stapleton (2006)  Hunt & Hunt (2006)  Kvavik (2005)  Lorenzo & Dziuban (2006)  Meola (2004)  Oblinger & Oblinger (2005)  Sidler (2002)  Sorapure et al. (1998)  Sullivan and Porter (1997)  Thompson (2003)  Wineburg (1991,1991,1999) Source Evaluation Tools  Barker & Kupersmith (2009) *  Beck (2009) *  Cohen & Jacobson (2009) *  CARS / CAFÉ – Harris (2007) *  Big6 – Eisenberg & Berkowitz (2001)  Hacker (2008) *  TAP – Johnson & Lamb (2007) *  CSU How to… – Lederer *  Rampolla (2007) *  Smith (1997) * * Checklist variant Cross-disciplinary discussions: Sarah, Kate and Tim

Computers & Writing 2009: Ubiquitous and Sustainable Computing21 Q6C: Modeling Source Evaluation Question C ategorize C ritique Rhetorically C haracterize Authorship C ontextualize C orroborate C onclude

Computers & Writing 2009: Ubiquitous and Sustainable Computing22 Q6C: The Start Question C ategorize C ritique Rhetorically C haracterize Authorship C ontextualize C orroborate C onclude  Maintain a skeptical frame of mind  Ask questions relevant to your research, purpose, and discipline

Computers & Writing 2009: Ubiquitous and Sustainable Computing23 Q6C: The End Q uestion C ategorize C ritique Rhetorically C haracterize Authorship C ontextualize C orroborate Conclude  Is the source credible?  Is the source useful for your research goals?

Computers & Writing 2009: Ubiquitous and Sustainable Computing24 Q6C: The Middle Q uestion Categorize Critique Rhetorically Characterize Authorship Contextualize Corroborate C onclude  Different dialogues to engage in with a potential source  Features and relative importance shaped by research task and discipline  Not necessary to do all components nor in any order

Computers & Writing 2009: Ubiquitous and Sustainable Computing25 Question Q6C: The Cloud Categorize Contextualize Corroborate Conclude Characterize Authorship Critique Rhetorically Repeat as necessary

Computers & Writing 2009: Ubiquitous and Sustainable Computing26 Simple Instantiation of Q6C Categorize Series of posts on Blogger Characterize Authorship Most recent post was 2001 Conclude Too old to be useful Question Topic: Information / opinion on recent U.S. policy towards net neutrality

Computers & Writing 2009: Ubiquitous and Sustainable Computing27 More Complex Instantiation of Q6C Question Conclude Likely to be useful Characterize Authorship He is a Swiss citizen Categorize Many posts on ScienceBlogs Conclude Useful but biases exist Corroborate Provides links to citations Characterize Authorship Part owner of a European ISP Topic: Information / opinion on recent U.S. policy towards net neutrality Characterize Authorship Relevant posts in

Computers & Writing 2009: Ubiquitous and Sustainable Computing28 Summary of Q6C  Working model of source evaluation process  Definite start and end points  Iterative and integrative middle  Emphasizes research as a process  Only concerns one element of all of research  Not specific to particular disciplines  Domain knowledge, heuristics, and criteria need to be supplied by instructors  Provides a common language for discussion  Dirty little secret… not limited to just online sources But how do you use it in teaching?

Computers & Writing 2009: Ubiquitous and Sustainable Computing29 Classroom Applications of Q6C Two approaches for using Q6C: 1. Using Q6C to inform teaching practice and assignment and lesson design (Sarah) 2. Infecting students to the practice and habit of source evaluation via the Q6C model (Tim)

First Hit: Why is this not a good source? The Answer Authoritative Source and Author

Why is this not a best source? Authoritative Source Scientific Source Partial Answer Wrong genre? But current: 07-08

Where to go from here? Purpose tangential to the research question Is there a better source?

Aha! The most credible and the most useful source! Peer Reviewed Directly relevant to research question

How to get a student (or a newcomer) to this “best” source? Finally, a complete answer relevant to the research scenario

Visualizing the Q6C Process Question: Was the cause of the mine collapse geological, and what was it? Scenario: College-level geology course 1st hit: MSHA web page Characterize Authorship Government agency relevant to issue (.gov) Contextualize Not a scientific source Conclude Part of the story, but not citable Corroborate BUT: What sources does it cite? 2 nd hit: Berkeley lab Annual report Characterize Authorship.edu / Scientific lab Categorize Genre: annual report (not peer reviewed) Purpose: tangential to research question Conclude Best source: relevant and credible 3 rd hit: UT seismology scientific report Contextualize

Student Reflections: Corroboration is king (at least in this research scenario)

Q6C as a heuristic for lesson/assignment design 1. Identify a research scenario relevant to course context 2. Identify the tacit research practices of an experienced researcher in that knowledge domain: think Q6C. 3. Construct a scaffolded lesson or assignment to teach explicitly the domain knowledge necessary for critical source evaluation. 4. Plan for reflective writing or discussion to promote metacognition about research practices. OUTCOMES : 1. students locate more useful and credible sources 2. begin to learn how to approach any new research scenario. OUTCOMES : 1. students locate more useful and credible sources 2. begin to learn how to approach any new research scenario.

Other choices when planning a Q6C lesson or assignment  Prepare an archive to anticipate the research process or set students loose?  How are the research process and domain knowledge weighted in assignment evaluation?  Explicitly teach the meaning of Q6C terms, or prompt students to invent their own during a reflective process?

Computers & Writing 2009: Ubiquitous and Sustainable Computing39 Questions raised by Q6C  How do researchers know when to stop looking for sources?  What do experienced researchers in a field know about when to stop that newcomers don’t know?  How can newcomers to a field be supported to do research with the savvy of experienced researchers?

Computers & Writing 2009: Ubiquitous and Sustainable Computing40 Q6C: Applied to history

Computers & Writing 2009: Ubiquitous and Sustainable Computing41 Website analysis 1.0 beta

Computers & Writing 2009: Ubiquitous and Sustainable Computing42

Computers & Writing 2009: Ubiquitous and Sustainable Computing44 Website analysis 1.0

Spanish-American War 1.0

Spanish-American War 2.0

Computers & Writing 2009: Ubiquitous and Sustainable Computing50 Q6C Lessons Learned  Students already do low-level evaluation  Web checklists don’t help much  Thinking contextually with corroboration and purpose in mind does  Students can benefit from using Q6C explicitly

Computers & Writing 2009: Ubiquitous and Sustainable Computing51 Q6C for Students: Next Steps  Design and re-design assignments with Q6C  Use Q6C in course design, implicitly/explicitly  Monitor and analyze results  Tweak  Repeat  Infect others with the Q6C madness

Computers & Writing 2009: Ubiquitous and Sustainable Computing52 Ongoing Work  Continued classroom applications  Modern American Civilization From 1877 (Tim)  The rhetoric of Writing in the Workplace (Sarah)  Further research and development  Refinement of the Q6C working model  Connect to literature on cognitive apprenticeship and metacognition (e.g., Scardamalia & Bereiter (1983))  Dissemination and partnership building  Are you interested in using Q6C in your teaching? Talk to us.

Computers & Writing 2009: Ubiquitous and Sustainable Computing53 Thank You! We wish to acknowledge the following:  Center for Instructional Development and Research  Practical Pedagogy  Our students  Suzzallo Espresso  The Internet  You Q UESTIONS ?

Computers & Writing 2009: Ubiquitous and Sustainable Computing54 Extra Slides  Descriptions of each Q6C Component  Suggestions for Teaching With Q6C  Suggestions for Students Using Q6C

Computers & Writing 2009: Ubiquitous and Sustainable Computing55 Q6C: Question  Maintain a skeptical frame of mind  Ask questions relevant to your research

Computers & Writing 2009: Ubiquitous and Sustainable Computing56 Q6C: Categorize  In the context of your research, is this a primary, secondary, or tertiary source?  What type of site is it (website, blog, wiki, database, etc.)?

Computers & Writing 2009: Ubiquitous and Sustainable Computing57 Q6C: Critique Rhetorically  What do the authors’ choice of words, tone, font, display format, images, genre, and argumentative strategies tell you about the intended audience and the credibility and reliability of this site? (‘Read’ the site.)

Computers & Writing 2009: Ubiquitous and Sustainable Computing58 Q6C: Characterize Authorship  Identify who created the content, when they created it, and for what purpose.  Single or multiple authors? Committee? Institution? Critic? Expert? Unknown? Other?

Computers & Writing 2009: Ubiquitous and Sustainable Computing59 Q6C: Contextualize  Place the information collected in conversation with your existing experience and body of knowledge.  Does it fit? How?

Computers & Writing 2009: Ubiquitous and Sustainable Computing60 Q6C: Corroborate  Assess how the content compares to other sources.  Is the content consistent, complementary, or contradictory?

Computers & Writing 2009: Ubiquitous and Sustainable Computing61 Q6C: Conclude  Is the source credible?  Is the source useful for your research goals?  If not, find a new source, repeat Q6C.

Computers & Writing 2009: Ubiquitous and Sustainable Computing62 Suggestions for Teaching With Q6C Remember that subject-area experts automatically perform the Q6C process, whereas novices need to consciously perform each step when learning how to assess a source's credibility and usefulness.  Identify which components of Q6C you want to emphasize and scaffold the assignment’s research process so that students learn new skills incrementally.  Teach that research is about a process, not about a product.  Construct assignments that engage your students in authentic research practices for your target discipline.

Computers & Writing 2009: Ubiquitous and Sustainable Computing63 Suggestions for Teaching With Q6C Tips for designing assignments and activities:  Make explicit the outcomes of the assignment and encourage reflection to help move students to the meta-cognitive level.  Consider choosing research topics that you are not an expert in so that you can share the discovery process with the students.  Perform the assignment yourself or with a colleague prior to class in order to anticipate student responses.

Computers & Writing 2009: Ubiquitous and Sustainable Computing64 Suggestions for Teaching With Q6C Tips for designing assignments and activities:  Make explicit the outcomes of the assignment and encourage reflection to help move students to the meta-cognitive level.  Consider choosing research topics that you are not an expert in so that you can share the discovery process with the students.  Perform the assignment yourself or with a colleague prior to class in order to anticipate student responses.

Computers & Writing 2009: Ubiquitous and Sustainable Computing65 Suggestions for Students Using Q6C  Q6C is a heuristic and not a checklist. You do not need to answer every question for every source. Some sources will require only asking one component of Q6C; other sources will require repeated application of Q6C.  Remember that there are two questions you should ask for every source: Is it credible? Is it useful? Answers to these questions are not absolute: a source can still be useful but still of dubious reliability.

Computers & Writing 2009: Ubiquitous and Sustainable Computing66 Suggestions for Students Using Q6C  When you think you have found a credible source, perform one last check by applying Q6C a little further. A review of Characterize Authorship, Contextualize, and Corroborate can reveal important, overlooked details.  Practice using Q6C in other classes and daily readings. As you gain experience, you will begin to automatically use Q6C to evaluate all kinds of information.  Remember: Research is a process, not a product.