1. Multimedia Advances in Evaluation: The Use of Skype, Elluminate, and 3D Virtual World Technologies in Conducting Focus Group Interviews Evaluation, Assessment, & Policy Connections (EvAP) Dr. Rita O'Sullivan, Director School of Education, University of North Carolina-Chapel Hill 919/843-7878 Johnavae Campbell, Corliss Brown, Jessica Milton, & Lauren Kendall AEA November 2010

2. Demonstration Agenda I. Introduction II. Web-based technology III. Choosing a medium IV. Preparation- evaluator/participants V. Demonstration: Elluminate, Teleplace, & Skype VI. Summary AEA November 2010

3. Evaluation, Assessment and Policy Connections (EvAP) EvAP builds evaluation capacity, effectiveness of organizations, and sustainability of successful programs 1. Conduct evaluations of educational, community, and service organizations 2. Provide training and technical assistance in evaluation, assessment, and strategic planning 3. Offer Evaluation Institutes for building understanding in systematic, collaborative, program evaluation

4. Introduction Advantages of web-based technologies for focus group evaluation research include: Reduced costs – logistical and travel expenses Accessibility and flexibility Increased recruitment opportunities of participants Innovative ways for participants to articulate their experiences and opinions Multiple methods of communicability Increased engagement and participation Provides relative anonymity and can minimize the perception of power differentials Expedites respondent communications and document exchange AEA November 2010

5. Technology Components Each program has at least five basic components:  Real-time discussion capacity  Chat/Instant messaging capabilities  Document sharing  Free trial services  Online tutorials

6. AEA November 2010 Choosing a Medium Start with your evaluation plan. Ask yourself the following questions to help determine if one of these options may be right for your team. What do you hope to accomplish? Remember, not all topics are appropriate for web-based focus groups. If you are interested in sharing things like photos, documents, or stories this is a great tool. Who is your audience? Are you working with students, teachers or a demographic who may have access to computers or a lab? Are they comfortable using a computer to share information? If not, this may not be right for you. Where are your participants located? Are you interested in working with participants who are not in the same geographic location? This may be a great option!

7. AEA November 2010 Preparation- for evaluator Learn Program  Are there training sessions offered? Have you used the program before? Create Focus Group Protocol  The focus group protocol should be created based on the evaluation plan. As always, the questions may change depending on the responses and dynamics of the group. Plan Virtual Focus Group  Decide how you will use the technology. Will participants use audio, chat, visual, or a combination? Do you need to show the participants documents or applications? How will you ensure all attending are engaged and able to participate? How much time will you need? Will you do a pilot test?

8. Preparation- for participants Invitation to focus group.  Who is participating?  What preparation/material/equipment is needed?  What program is sponsoring the focus group?  When is the focus group (include log-in time)?  Where can participants gain access (internet link)? Introduction to program  Training session. In-depth instruction on features of the program and how-to access program  Brief Introduction. Overview of features of program and procedures for focus group AEA November 2010

9. What is Elluminate? Elluminate Live!® enables instructors and students to interact and collaborate in real time to combine blended online/onsite learning activities. Elluminate refers to it as Edu 2.0, similar to Web 2.0, allowing for dynamic, shared content building in real-time. AEA November 2010

10. Why use Elluminate? UNC-BEST Program Goals of Program  place teachers in high-needs schools  attract minority teachers to math and science  provide high quality research and laboratory experiences As part of the evaluation focus groups were conducted with the instructors, seniors, and graduates from the previous year. Graduates were teaching in various parts of North Carolina AEA November 2010

11. An Elluminate Focus Group Question-based discussion  Chat  Audio  Polling/Emoticons Document-based discussion  Microsoft Office document sharing  Live website sharing Data Analysis  Recording AEA November 2010

12. Elluminate Demonstration Moderator Link Participant Link Recording Link AEA November 2010

13. Teleplace: A 3D virtual meeting space Program Background  Evaluated an NSF funded grant that used 3D immersive virtual world technology to;  Increase STEM interest among participating middle school students  Increase 3D technology integration among participating teachers AEA November 2010

14. Teleplace: A 3D virtual meeting space Evaluation Questions  What was the perceived quality of delivery of the summer workshop?  Did students teach teachers 3D technological skills?  Did students adequately prepare to teach teachers?  How did students collaborate in the virtual world during the summer workshops?  How have students collaborated since the summer?  How have teachers engaged students in the use of 3D world tools to explore solutions to real world problems?  Are middle school students more motivated to pursue ICT careers after participation in the STEM-ICT 3D program? AEA November 2010

15. Teleplace: A 3D virtual meeting space Virtual interviews were intended to;  Find out more about their experiences in the summer workshop (including the worlds they created) and since the summer workshop  Assess students’ interest in STEM and STEM ICT careers  Engage students in using the technology AEA November 2010

16. Teleplace: A 3D virtual meeting space Decision based on following factors;  Students spread out geographically  Needed access to students during the school day  Student-created worlds in Teleplace  High-level of engagement with technology AEA November 2010

17. Skype Demonstration Program Background Burroughs Wellcome Fund Career Award for Science and Mathematics Teachers (BWF) program goals: Increase pedagogical and content knowledge of science and mathematics Increase professional development and collaboration with other master science and mathematics teachers Develop teachers as leaders in the field AEA November 2010

18. Skype Demonstration Virtual interview was intended to:  Provide technical assistance in developing Evaluation Logic Models  Frame program activities  Determine desired program outcomes  Identify data collection strategies  Plan for process and outcome evaluation of program activities AEA November 2010

19. Skype Demonstration Decision based on following factors:  Geographic barriers  Convenience of choosing location  Expedite respondent communications and document sharing  Innovative way to communicate Features utilized during the session:  Document Sharing  Audio and Visual Communication  Audio Recording AEA November 2010

20. Skype Welcome Page AEA November 2010

21. Skype Calling AEA November 2010

22. Skype Talking AEA November 2010

23. Skype Conferencing AEA November 2010

24. Skype Group Talk AEA November 2010

25. Skype Document Sharing AEA November 2010

26. Synopsis Considerations  Cost: What will it cost the evaluator and participants?  Access: Do participants have access to computers with Internet?  Participant Engagement: Will participants be comfortable using this medium?  Communicability: How do you want participants to communicate (text, talk, etc.)? AEA November 2010

27. Synopsis cont. Refer to your evaluation plan Choose the appropriate medium  What, if anything, does an internet-based focus group offer or allow for that a face-to-face focus group does not? Prepare as the evaluator  Consider additional technology-related factors (connectivity, availability, accessibility)  Determine how you will utilize the technology and practice using it (responsibilities of moderator/assistant moderator)  Prepare protocol, including specific measures/instructions for participants  Pilot-test Prepare participants AEA November 2010

28. Contact Information EvAP: Find us on the web at: http://www.unc.edu/depts/ed/evap/ http://www.unc.edu/depts/ed/evap/ (919) 843-7878 Rita O’Sullivan, Director: ritao@unc.eduritao@unc.edu Johnavae Campbell: johnavae@email.unc.edujohnavae@email.unc.edu Corliss Brown: ccbrown@email.unc.educcbrown@email.unc.edu Jessica Milton: jrmilton@email.unc.edujrmilton@email.unc.edu Lauren Kendall: lkendall@unc.edulkendall@unc.edu AEA November 2010