Photo Journaling in the Evaluation Context David Urias, Ph.D. Drexel University – Evaluation Research Network Olga Pierrakos, Ph.D James Madison University AEA Coffee Break Webinar January 27, 2011

Overview The Basics Examples of Coded Entries The Process Product Analysis Examples of Coded Entries Barriers to Effective Photo Journaling: Lessons Learned Versatility of Photo Journaling Final Thoughts Q & A Photo journaling enables participants to express, capture, and chronicle their emotions through photography and journaling. Working independently in this venue allows participants to tell their personal story in a powerful and reflective way to share insights, thoughts, and fears about their overseas and overall experience. The process of photo journaling can be a powerful qualitative evaluation tool for many types of learning experiences, whether those experiences are classroom-based or extra-curricular. So, the purpose of this webinar is to show how to treat photos as data, how to analyze their content, and how to use both the product and process to improve not only an evaluation, but also stakeholder learning and investment.

The Basics Experience + Reflection Growth In short, photo journaling is a tool to enable the capture of a particular moment’s essence in a photo, and then reflect upon that moment from a later—and so different—perspective. This process leads to personal growth. Why? Because an unexamined experience leads to stagnation. Times change, perspectives change, and creating a blog or other similarly required “reflective” task often amounts to a cursory recounting of the week’s events, like a news report. Photo journaling, in contrast, is a more structured method of self-reflection, one that requires an earnest effort and pure intention. It allows one to understand one’s self, his/her relationships, and the fundamental nature of existence. While a chosen photo is a snapshot in time, the photo journaling process broadens one’s view of reality. It is as if, standing on top of a mountain, a shift from a zoom lens to a wide-angle lens occurs. One can then appreciate the broader panorama - the former perspective still included, but accompanied by much that had been hidden. And that which was hidden makes the view extraordinary. “…meaningfully, enriching experience on an academic as well as a personal level.” ~ undergrad program participant

The Process Specifically Describe Analyze Interpret Note questions arising from experience Step 2 Reflect on that moment in the form of accompanying text – useful to provide guided questions to focus photo selection and reflection Access a particular moment in the form of a photo To complete the two step process, one will need to a) access the particular moment in the form of a photo, and b) reflect on that moment in the form of an accompanying text. As part of photo journaling tasks, participants are provided guided questions to assist in describing the setting while avoiding inferences about how people felt or what they thought; Analyze the event; provide an Interpretation of what feelings and thoughts may have caused the event to occur, why it was significant, what questions were raised, and what one thinks was learned from them. Additionally, participants are asked to figure out what they accomplished, identify problems that emerged and how they plan to follow up, and distill from the event what was learned, while also relating the experience to academic concepts, course work, etc. The take-away here is that the format of photo journaling requires participants to distinguish between description and analysis, each with its own section. When students begin journaling, they typically have difficulty distinguishing between these two sections, which run the risk of undermining their use for reflection. Step 1 Requires participants to distinguish between description and analysis, each with its own section

Product Analysis Level 1: Descriptive Level 2: Empathic Level 3: Participants provide evidence that new knowledge is obtained which allows him/her to make sense of new experiences or make links between old and new knowledge (what one used to think/did vs. what is learned and how it affects him/her). Level 2: Empathic Participants express thoughts or emotions about others and self. S/he reflects their experience (emotions, attitudes, beliefs) onto how future participants may feel or react to the experience. Participant empathizes with those around him/her. Level 3: Analytic Participants demonstrate the application of learning to a broader context of personal and professional life. Photo journal entry provides evidence of learning/ growth in order to contrast, compare or plan for new actions or responses. Participants also notice unexpected positive or negative outcomes related to the project. Level 4: Metacognitive Participants demonstrate an examination of the learning process: showing what learning occurred, how learning occurred, and how newly acquired knowledge or learning altered existing knowledge. Participants plan to change future behavior based on the project experience and its outcome(s) on his/her life. As a form of qualitative data, photo journals are analyzed thematically; specifically, writing is examined for wording which describes the skills learned and insights gained from the program, along with the ability to use such skills and insights during their experience. Initial analysis should identify the level of reflection. Entries should be coded independently by at least two individuals unfamiliar with the task, and then compared for differences which are to be resolved by consensus. Based on the work of Chabon & Lee-Wilkerson (2006), the level of reflection is coded as one of four levels. This slide is designed to give you a sense of the big picture, but let’s take a quick look at each of the levels.

Examples of Product Analysis/ Coded Entries Level 1: Descriptive Participants provide evidence that new knowledge is obtained which allows him/her to make sense of new experiences or make links between old and new knowledge (what one used to think/did vs. what is learned and how it affects him/her). Being an outsider is a physical attribute I am pretty sure our preferential treatment is due to our Americanness  Level 1: Descriptive The participant provided evidence that new knowledge was obtained which allowed him/her to make sense of new experiences or make links between old and new knowledge (what one used to think/did vs. what was learned and how it affected him/her). Level 1- Descriptive   Being an outsider [in the Korean context] is a physical attribute. No matter how much I may become mentally qualified to join the club I will never [be able to] shed myself of the label outsider. I am pretty sure our preferential treatment is due to our Americanness in particular as opposed to just any foreigners. But this has led me to question what I have been defining an outsider as. I will always be American and I will always look different and therefore always get this treatment but will I always be an outsider? What really is the difference between an outsider and a foreigner? So is he an outsider or not? The reflection doesn’t go into detail or attempt to address this and thus, is characterized as a superficial reflection or level 1-descriptive.

Examples of Product Analysis/ Coded Entries Level 2: Empathic Participants express thoughts or emotions about others and self. S/he reflects their experience (emotions, attitudes, beliefs) onto how future participants may feel or react to the experience. Participant empathizes with those around him/her. Men and women find different topics of conversation and comments on people’s appearances acceptable. Koreans discuss things that would sometimes be considered rude and offensive in America Level 2 Rubric: Empathic The participant expressed thoughts or emotions about others and self. S/he reflected their experience (emotions, attitudes, beliefs) onto how future participants may feel or react to the experience. Participant empathizes with those around him/her. Level 2- Empathic While walking along the street in Seole, men and women find different topics of conversation and comments on people’s appearances acceptable. Koreans discuss things that would sometimes be considered rude and offensive in America.

Examples of Product Analysis/ Coded Entries Level 3: Analytic Participants demonstrate the application of learning to a broader context of personal and professional life. Photo journal entry provides evidence of learning/ growth in order to contrast, compare or plan for new actions or responses. Participants also notice unexpected positive or negative outcomes related to the project. Exploring the culture has reinforced something I don’t think most American’s really ponder or internalize – The United States is not the center of the world. There’s a lot for us to learn from other countries and cultures and if we don’t reach out on levels beyond government and commerce, we’re likely to miss it Level 3 Rubric: Analytic The participant demonstrated the application of learning to a broader context of personal and professional life. Photo journal entry provided evidence of learning/growth in order to contrast, compare or plan for new actions or responses. Participant also noticed unexpected positive or negative outcomes related to the project. Level 3- Analytic Exploring the culture has reinforced something I don’t think most American’s really ponder or internalize – The United States is not the center of the world. There’s a lot for us to learn from other countries and cultures and if we don’t reach out on levels beyond government and commerce, we’re likely to miss it.

Examples of Product Analysis/ Coded Entries Level 4: Metacognitive Participants demonstrate an examination of the learning process: showing what learning occurred, how learning occurred, and how newly acquired knowledge or learning altered existing knowledge. Participants plan to change future behavior based on the project experience and its outcome(s) on his/her life. It is important that we be globally engaged. Globalization is more about the increased communication and cooperation worldwide. My success [in the program] facilitates the preparation for future student participation in international research collaboration Level 4 Rubric: Metacognitive Participant demonstrated examination of the learning process, showing what learning occurred, how learning occurred, and how newly acquired knowledge or learning altered existing knowledge. Participant plans to change future behavior based on the project experience and its outcome(s) on his/her life. Level 4- Metacognitive It is important that we be globally engaged due to the norm for advancements to be a result of international work. Globalization is not only about fast food chains in third world countries but the increased communication and cooperation worldwide. My success facilitates the preparation for and participation in international research collaboration for students.

Barriers & Lessons Learned Too much description Neglect critical analysis Process needs to be continually reinforced and monitored regularly Unexamined assumptions No critical reflection on those assumptions Monitor reflective process regularly The first issue that occurred in our test case of using photo journaling was that participants added so much description to their episode section that they neglected an analysis of the episode. Information about the process needs to be continually reinforced and monitored regularly. The second issue to arise was that participants typically made unexamined assumptions in the episode section without giving themselves the opportunity to reflect on those assumptions critically. This issue can be avoided in the future, again, if the reflective process is regularly monitored. The limitation of using photo journaling as a single, qualitative evaluation tool is that the best that can be accomplished is a general understanding of the program outcomes and outcome achievement; it is not appropriate to expect a definitive statement about the success of the program or about what work has yet to be done - it is a given that challenges remain ahead for the program. However, a positive implication of using photo journaling in the evaluation process rests in the fact it is a primary source of data and one of several other sources. It is important to note that multiple sources of data used in a program evaluation reduces the potential for bias and provides greater opportunity for a balanced picture of the program. An evaluation team can take steps to reduce bias by balancing stakeholder information with additional program documentation and input from other primary sources wherever possible. This strategy has been documented as a means to enhance validity. Not appropriate to expect a definitive statement about the success of the program or about what work has yet to be done

Photo Journaling’s Versatility Like the cartoon on this slide, photo journaling is versatile. As a method, it can be modified to meet specific program goals and objectives.  For example, for a program that has a goal on diversity, the photo journaling process can be structured in a way to extract from students their reflections on certain aspects of diversity.  So, in a sense, the photo journaling process can be made more structured for programs that have specific and defined goals and more open-ended for those programs whose goals are less defined. Also, there are multiple ways one could code photo journaling entries.  As was shown, we used one way, but depending on what the measurable goals are, one could certainly use other rubrics for coding. Photo journaling as a method can be modified to meet the specific program goals and objectives

Final Thoughts Conclusion However Method enables one to express, capture, and chronicle emotions via photography and journaling Offers a fuller “picture” of a program, assists in making compelling statements of program results, adds vitality to program reports, and enhances the meaning behind participants’ reflective statements However Assessing student outcomes is a work in progress Application of photo journaling without distinguishing between description and analysis results in undermining reflection; but Intends to identify issues that could be used by program stakeholders to improve program delivery, scope, outreach, and impact When evidence from photo journaling indicates that students are having difficulty demonstrating a desired outcome at the appropriate level, improvements can be made in the processes designed to promote the outcome and, if appropriate, new assessments made. Outcomes targeted for program improvement can be gauged more frequently than those which students consistently meet. The design and use of photo journaling can automate the search and sampling process; but to be done properly, assessment in general takes time, money, and skill. Although the use of photo journals may not be the most expedient way to collect assessment data, there is a consistently inverse relationship between the quality of measurement methods and their expediency. In short, work in this area is intended to identify issues that could be used by program stakeholders to improve program delivery, scope, outreach, and impact. It is hoped that an active and ongoing dialogue on quality assessment techniques occurs in order to strengthen the academic grounding of the evaluation profession, while assisting stakeholders in evaluating their programs.

Q & A Thinking outside the box