Creating a Framework Tutorial: A Transformative Process Allison Hosier Information Literacy Librarian, University at Albany LOEX 2016 Conference Pittsburgh, PA May 6, 2016

Agenda Creating a tutorial project using ideas from the Framework Planning the project How learning more about threshold concepts led to revision of plans Tutorial creation process

February 2015 Members of the information literacy department met with the library’s Student Advisory Board to discuss online learning objects. The students at the meeting expressed an interest in a tutorial about understanding and using scholarly articles as a part of research.

February 2015 This meeting took place just after the Framework was finalized. Opportunity to begin applying the Framework as part of instruction

What I thought the project be Choose a frame from the framework Create learning outcomes based on that frame Devise an assessment mechanism Create a tutorial to “teach” the frame

What the project ended up being Basically, a learning journey that involved a lot of self-doubt

What made this so difficult? Key differences between the Standards and the Framework Learning outcomes/assessment and threshold concepts are two different ways of thinking about teaching and learning. Learning outcomes/assessment and threshold concepts are not entirely compatible with each other. Their compatibility with an online learning object such as a tutorial was also in doubt.

Key differences Standards: Prescribed learning outcomes and knowledge practices. Each of the standards could be easily scaled to different instructional situations. Framework: Threshold concepts with recommended knowledge practices and dispositions but not learning outcomes. More difficult to scale.

Learning outcomes and threshold concepts “A Roadmap for Assessing Student Learning Using the New Framework for Information Literacy for Higher Education” by Megan Oakleaf (Journal of Academic Librarianship, 2014)

Two different ways of thinking about teaching and learning Learning outcomes/assessment Pre-determined endpoint for the learning process Learners arrive at the same shift in understanding at the same time Learners demonstrate their learning in the same way Threshold concepts Learning is a highly individual journey Crossing a threshold of understanding is a transformative process Harder to demonstrate

Another point of view Naming What We Know: Threshold Concepts of Writing Studies by Linda Adler-Kassner and Elizabeth Wardle (Eds.) Avoid conflating learning outcomes and threshold concepts. Threshold concepts should be used to influence what you teach, not as the end point.

What I thought the project be Choose a frame from the framework Create learning outcomes based on that frame Devise an assessment mechanism Create a tutorial to “teach” the frame Standards can be taught directly, treated as topics. Threshold concepts not so much.

Further complications If threshold concepts are intended to be “transformative,” could a learning object like a tutorial be enough to help students experience a shift in understanding?

Further complications If threshold concepts are intended to be “transformative,” could a learning object like a tutorial be enough to help students experience a shift in understanding? Probably not. Completing a tutorial is not a transformative experience (no matter how much effort we put into making a great tutorial). So, do tutorials still have a role? Or should threshold concepts only be applied selectively?

The role of tutorials Tutorials or other short instruction opportunities can bring the threshold into view—opens the learner’s eyes to other ways of thinking. They can help the user start to develop a more sophisticated understanding of the topic or create awareness that there is such a thing. This primes the learner to cross the threshold through further instruction or independent work.

Back to the drawing board Choose a frame from the framework Create learning outcomes based on needs expressed by students Revise learning outcomes using chosen frame as an influence Devise an assessment mechanism Create a tutorial that uses the frame as an influence rather than being about the frame

Choosing a frame Many frames had applicable elements but best practice is to focus the content of the tutorial. Scholarship as Conversation was the obvious choice

About the frame vs. influenced by the frame Scholarship as Conversation: The Tutorial vs. Working with Scholarly Articles

Original learning outcomes based on student needs Students will be able to identify the role of a scholarly article in the research process. Students will be able to recognize the function of common elements of a scholarly journal article. Students will be able to distinguish between a scholarly article and a non-scholarly article. Came up with these first to ensure students’ requests were being met. Common learning outcomes for learning the role of scholarly articles in research, but they don’t quite get at the frame.

Students will be able to identify the role of a scholarly article in the research process. Covered in materials previously offered by the library focusing mainly on superficial characteristics of scholarly articles and how they compare to other types of articles. Little to no mention of broader rhetorical context or the place of scholarly information in the larger conversation. Does this approach help students understand what scholarly articles are?

Revised learning outcome Students will be able to recognize the scholarly article as a piece of an ongoing scholarly conversation in which a variety of perspectives may be represented and meaning must be negotiated. Scholarly articles are part of a broader context. Not a source for right answers because they represent multiple perspectives.

Students will be able to recognize the function of common elements of a scholarly journal article. Fulfilled students’ request for help understanding the content of a scholarly article. How to place this in a more rhetorical context? McMillen & Hill: teaching rhetorical aspects of research by framing them as a conversation being conducted in a foreign language

Students will be able to recognize the function of common elements of a scholarly journal article. “Why Teach ‘Research as Conversation’ in Freshman Composition Courses? A Metaphor to help Librarians and Composition Instructors Develop a Shared Model” by McMillen & Hill (Research Strategies, 2004) McMillen & Hill: teaching rhetorical aspects of research by framing them as a conversation being conducted in a foreign language

Revised learning outcome Students will be able to recognize the common building blocks of a scholarly article in order to become an effective and efficient reader of the scholarly conversation. A key to understanding foreign languages is looking for patterns. The common building blocks of a scholarly article are those patterns.

Students will be able to distinguish between a scholarly article and a non-scholarly article. Most closely resembles familiar approaches to teaching students about scholarly articles: using a checklist of superficial clues.

Revised learning outcome Students will be able to distinguish between scholarly articles and other types of contributions to the scholarly conversation in a search for information by using their knowledge of the role scholarly articles play in the scholarly conversation and the common characteristics resulting from that role. Common characteristics as evidence of the niche place scholarly information holds in larger conversation, i.e. article’s title will indicate that the author is an expert speaking to other experts.

Revised learning outcomes Students will be able to recognize the scholarly article as a piece of an ongoing scholarly conversation in which a variety of perspectives may be represented and meaning must be negotiated. Students will be able to recognize the common building blocks of a scholarly article in order to become an effective and efficient reader of the scholarly conversation. Students will be able to distinguish between scholarly articles and other types of contributions to the scholarly conversation in a search for information by using their knowledge of the role scholarly articles play in the scholarly conversation and the common characteristics resulting from that role.

Working with Scholarly Articles Part 1: What is a Scholarly Article? Part 2: Reading a Scholarly Article Part 3: Finding and Identifying Scholarly Articles User can navigate directly to most relevant section.

Assessment Why is assessment of tutorials important? Is it even possible to assess students’ learning as a result of a tutorial?

Why is assessment of tutorials important? Learning goals and assessment go hand-in-hand in instructional design literature. Assessment measures whether students have met learning goals. Can help prove the tutorial’s value in overall instruction program. Assessment may be required by the institution.

Is it even possible to assess student learning as the result of a tutorial? Many studies conducted, usually in controlled environments, to measure whether students can meet learning goals as the result of a tutorial. Examples of more everyday assessment are harder to find. Quizzes are most common. Articulate software allows more elaborate active learning/assessment opportunities, but usually at the expense of accessibility.

Assessment and threshold concepts “In fact, threshold concepts are very well suited to learning outcomes assessment, as long as the assessments permit the use of authentic assessment approaches, provide useful feedback to students to help them over the ‘stuck places,’ emphasize individual variation in the journey that students travel to achieve them, recognize that learners may redefine their sense of self, link learning and grading in meaningful ways, organize programmatic assessment around transformational ideas, and support metacognition.” (Oakleaf, 2014)

Assessment in scholarly articles tutorial Check Your Knowledge questions at the end of each section. The user answers a multiple choice question related to the content of that section. A correct answer allows them to move on to the next section or exit the tutorial. An incorrect answer takes them through a brief review with an option for more in-depth review before trying the question again. Allows students with adequate understanding of the material to move on.

Assessment Check Your Knowledge questions at the end of each section. Set up using buttons and hot spots in Camtasia rather than the built-in quiz feature to allow more options to create different paths. Unfortunately, this means there’s no way to keep track of users’ answers, so the use of these questions as an assessment strategy is limited (or non- existent).

Assessment Form at the end of the tutorial, intended to be used by students who are completing the tutorial as part of a course assignment. Students are asked to describe one thing they have learned and one thing they still have a question about.

Model answer “I learned that scholarly articles are a piece of the scholarly conversation with a set of identifiable characteristics, including that they are written by experts with a variety of perspectives that may change over time. Identifying the sections of the article that are most relevant to my research goals will help me become a more efficient and effective reader who is able to negotiate meaning from the perspectives I encounter as I develop my own ideas on the topic.” Not realistic, but student responses taken in aggregate can help measure whether the tutorial is meeting the learning goals.

Limitations of assessment Land & Meyer caution against forms of assessment that encourage mimicry, since it’s impossible to tell if the student have really grasped the concept or are repeating what they think the instructor wants to hear while retaining misconceptions If you can figure out how to do this in a tutorial, let me know. Also, tutorial is not a transformative experience. It’s an experience on the way to a transformative experience.

Since then Working with Scholarly Articles tutorial became available shortly before Fall 2015 semester: http://library.albany.edu/infolit/scholarly1 Promoted to faculty who use tutorials and on the library’s website Usage in the first year was consistent but not high enough to draw conclusions from Initial analysis of the information gathered will be conducted Summer 2016. Further promotion will need to be done to increase usage.

References Adler-Kassner, L., & Wardle, E. (2015) Naming what we know: The project of this book. In L. Adler- Kassner & E. Wardle (Eds.), Naming what we know: Threshold concepts of writing studies (pp. 1- 11). Boulder: University Press of Colorado. McMillen, P. S., & Hill, E. (2004). Why teach “research as a conversation” in freshman composition courses? A metaphor to help librarians and composition instructors develop a shared model. Research Strategies, 203-22. doi:10.1016/j.resstr.2005.07.005 Oakleaf, M. (2014). A roadmap for assessing student learning using the new Framework for Information Literacy for Higher Education. Journal Of Academic Librarianship, 40(5), 510-514. doi:10.1016/j.acalib.2014.08.001

Creating a Framework Tutorial: A Transformative Process Allison Hosier Information Literacy Librarian, University at Albany ahosier@albany.edu