1. www.cirtl.net CIRTL Network Meeting www.cirtl.net/networkgroup Thursday, July 17, 2014 11:00-12:30 ET/ 10:00-11:30 CT/ 9:00-10:30 MT / 8:00-9:30 PT This meeting will be recorded Begin by running the Audio Setup Wizard Select : Tools>Audio> Audio Setup Wizard or select the audio wizard icon Backup Call-in: Call-in Number: 1 (571) 392-7703 Passcode: 931 749 952 188

2. www.cirtl.net Announcements Reminder: CIRTL Membership Dues – August 1, 2014 Fall In-Person Meeting RSVP / Hotel (Don’t Wait!) Institutional Portraits SOW Update Cross-Network Registration opens 7/21/2014 Agenda Topic 1: Local Evaluations University of Massachusetts at Amherst University of Georgia University of Rochester

3. www.cirtl.net IP & SOW Update Institutional Portraits (cirtl.net/2014ip) 18 IPs submitted, 18 partially processed & posted, 1 pending revision 4 not submitted Submissions to date will be used for NSF annual report SOWs 18 submitted, 13 reviewed/ready for RSP, 3 required revisions, 1 pending IP 1 to be reviewed

4. www.cirtl.net Course Registration Monday July 21 – Wed Aug 27 Full Courses Teaching-as-Research: Scholarship of Teaching and Learning I Diversity in the College Classroom Short Courses Student Centered Course Design The Active Classroom: Successfully Implementing Small-Group Learning in STEM Courses CIRTL Reads Journal Club (available as drop-in also) An Introduction to Evidence-Based Undergraduate STEM Teaching: A CIRTL College Classroom MOOC (registration Coursera, not yet open) http://www.cirtl.net/courses/se mesterofferingscurrent Register at:

5. www.cirtl.net CIRTLCast Series 2014-2015 Offer speakers from your institution September Promoting Underrepresented Minority Student Success in STEM Higher Education October Teaching and Learning in the American System: International Graduate Students and Post-Docs November The Academic Job Search January & February Educational Innovations/The Active Classroom March Surviving the First Year as Faculty Member April Personal Financial Management Contact: Don Gillian-Daniel, dldaniel@wisc.edu Contact: Stephen M. Roth, sroth1@umd.edu Contact: Judy Milton, jmilton@uga.edu Contact: Rob Linsenmeier, r-linsenmeier@northwestern.edu Contact: Judy Milton, jmilton@uga.edu Contact: Colleen McLinn cmm252@cornell.edu

6. CIRTL Network Meeting Presentation Local Evaluation Efforts July 17, 2014 Elizabeth Jakob, Faculty Co-Leader and Professor of Psychology Tilman Wolf, Faculty Co-Leader, Professor of Electrical and Computer Engineering, and Associate Dean of the College of Engineering Jung Yun, Director of New Faculty Initiatives, Center for Teaching and Faculty Development The University of Massachusetts Amherst

7. UMass B.C. (before CIRTL)  TA orientation workshop  Skills for college teachers course  25-30 people First year of CIRTL expanded offerings  Workshops for Associates  Seminar for Practitioners  Scholars just starting CIRTL “community”  1784 STEM grad students and post-docs  Receive emails about events Introduction of CIRTL to UMass

8. Workshop offerings  Six short workshops (90 min.)  One long workshop (1 ½ days)  Felder and Brent: “Effective College Teaching” Practitioner learning community  Started Spring 2014 Upcoming CIRTL seminar  July 31: “Syllabus Construction” Upcoming CIRTL short course  Fall 2014: “Student Centered Course Design” Year One Activities

9. Short workshops (subject to assessment)  Teaching at different institutional types  Top ten “teaching mistakes”  Preparing a teaching statement  Team-based learning  Large lecture strategies  Dealing with difficult students Long workshop (not assessed)  Effective college teaching Workshop Topics

10. Workshop assessment is focus of remaining presentation  Assessment for practitioners exists, but not discussed here Implementation of assessment process  Separation of programming and evaluation  Separate assessment conducted by Jung Yun, Center for Teaching and Faculty Development  Assessment results are guiding programming  Some changes to next year’s offerings Workshop Assessment

11.  Did the events provide practical information that participants could use in their own teaching?  Did the speakers stimulate the participants’ interest in the subject matter?  Did the variety of event formats (e.g., panel, workshop, discussion) allow sufficient time to ask questions and raise issues?  Did the participants find the events worthwhile?  Which events were most/least effective and why?  Who participated, and from what schools/colleges?  How often did participants return? CIRTL Workshop Evaluation Questions

12.  Online registration forms with demographic information  Post-event evaluation forms (online and print) with 5-point scale and open-ended questions  MS Access participant database Analysis Tools

13. 178 participants in 6 workshops over the academic year (105 unique individuals) Findings: Participants

14. Unique individuals comprised of 82 graduate students, 21 post-docs, and 2 new faculty members Findings: Career Stage

15. Of the 105 unique individuals, 42 (40%) participated in multiple workshops. Findings: Repeat Participation

16. Findings: Satisfaction CIRTL Workshops 2013-14 Provided Practical Information Stimulated Interest in Subject Matter Time for Questions and Issues Overall Satisfaction with Event Institutional Types4.694.004.554.66 Top Ten Teaching Mistakes4.764.854.624.85 Teaching Statement4.824.864.824.86 Team Based Learning3.50 3.693.44 Large Lecture Strategies4.70 4.604.80 Difficult Students4.694.724.564.69 AVERAGES4.534.444.474.55 Note: 79% post-event survey response rate

17. Planned Changes to Evaluation Efforts, Year Two Expand analysis areas Current:  Participation  Engagement  Potential Application  Satisfaction Future:  Participation  Engagement  Potential Application  Satisfaction  Learning  Application

18. Planned Changes to Evaluation Efforts, Year Two Future:  Online registration forms with demographic information  Pre- and post-event evaluation forms with 5-point and open-ended questions  MS Access participant database  Follow-up survey Expand analysis tools Current:  Online registration forms with demographic information  Post-event evaluation forms (online and print) with 5- point and open-ended questions  MS Access participant database

19. More coherent selection of workshop topics  First year was driven by available presenters Reach more participants, more often  Email GPDs  Hired someone to quickly update website  Any suggestions? Improve assessment of practitioners Gain experience with first scholars Plans for Year Two

20. Questions? Comments? Suggestions? Thank you! UMass Amherst CIRTL The University of Massachusetts Amherst

21. CIRTL@UGA Evaluation of Graduate Teaching Seminar CIRTL Network Meeting July 17, 2014

22. Graduate Teaching Seminar (GRSC 7770) Integrated Course Design College Teaching & Student Learning Using Technology in the College Classroom Service Learning Course Design Teaching as Research Learning Community SoTL Seminar (spring 2015) Teaching Portfolio Interdisciplinary Certificate in University Teaching Teaching and Lab Assistant Orientation CIRTL Online Courses

23. Graduate Teaching Seminar (GRSC 7770) Integrated Course Design College Teaching & Student Learning Using Technology in the College Classroom Service Learning Course Design Teaching as Research Learning Community SoTL Seminar (spring 2015) Teaching Portfolio Interdisciplinary Certificate in University Teaching Teaching and Lab Assistant Orientation CIRTL Online Courses

24. Graduate Teaching Seminar A course required for all graduate students with instructional duties – Multiple sections taught across campus in the fall semester, including both discipline specific and interdisciplinary sections – Provides graduate teaching assistants with knowledge of pedagogical approaches and available support systems. Special sections are taught for international students Only graduate level course for which a graduate student can be a co-instructor of record with a faculty member

25. Understanding the Basics Conducted a survey with all current TAs and LAs about their perceptions of GRSC 7770 What we learned: – Overall ratings ranged from “great” to “waste of time” – Interaction with instructors and peers was rated very important to “shape understanding of effective teaching” – Learning a range of strategies for student engagement was very important – Course design and lesson planning was rated important to very important

26. GRSC 7770 Course learning goals in the biological sciences typically include: the breadth and depth of the term “inquiry” as it pertains to teaching science in multiple learning environments different ways that adults learn writing as a mode of teaching and learning peer teaching observations and subsequent meaningful peer discussions with a culture of respect and responsibility active and passive teaching methods the importance of reflective discourse in your teaching how to encourage discussion in the classroom how to teach diverse populations of students how to write and use passive and active learning assessments how to manage disruptive student situations in the classroom how to manage potential issues of academic dishonesty how to manage potential issues of student emotional problems how to begin to build your teaching portfolio

27. Evaluation of GRSC 7770 I can use different teaching strategies to engage students I can adapt my teaching strategies to deliver specific content I can develop assessments that match the learning goals I can set clear learning goals to guide my teaching I can adapt my teaching strategies for diverse students in my class I can develop a structured lesson plan I can facilitate group discussions I can manage a classroom to maximize student learning I can access campus resources for specific purposes to assist students and myself I can use evidence-based research to improve my teaching I can give students feedback to improve learning

28. Evaluation Challenges Students in 10 sections were asked to complete the surveys – Each course was unique with different learning goals – Uneven response rate across sections – Higher (80%) response to pre-survey than post survey (47%)

29. Our Way Forward Challenge: aligning department needs with university’s goals to ensure purpose of GRSC is achieved – Offer summer workshops for faculty and graduate students teaching GRSC 7770 fall courses; required for departments which received an assistantship – Establish set of common core learning outcomes which can be used in all disciplines, based on CIRTL learning outcomes 2.0 – Revision of current self-efficacy instrument to align with CIRTL learning outcomes 2.0

30. Questions

31. CIRTL TAR Fellow Program: Evaluation and Assessment at the University of Rochester Barbara Masi Director of Education Innovation and Assessment Initiatives Arts, Science and Engineering 31

32. TAR Program Goals Goals 1. Engage TAR Fellows who are passionate about teaching. 2. Develop TAR Fellow teaching and education research competencies so that they are able to use education research methods to investigate and improve the teaching/ learning experiences of students and their own teaching. 3. Equip TAR Fellows with professional competencies so that they are able to shape their careers as teachers and continue to use their new skills as education researchers. 4. Pilot and deliver program elements that effectively develop TAR fellow competencies. 32

33. TAR Fellow Program Learning Outcomes Outcomes aligned with mix of CIRTL TAR Practitioner and Scholar (our Fellows) program outcomes in 3 areas: Teaching as research: TAR Learning community: LC Learning through diversity: LD Designed program elements to emphasize development of teaching and education research competencies: Topic seminars, Felllow update meetings, research projects, mentoring Some emphasis on professional competency definition and Fellow development (eg. learning community building, oral and written communication, professional networking): Formal presentation of project results Sharing ideas, helping peers with project hurdles Development of teaching portfolio 33

34. TAR Program Element: Application Process TAR Fellow applicants must submit education research project idea with application. Application proposal review: Ensures that chosen fellows have invested time in thinking about teaching as research. Are ready to engage in teaching as research training and projects that may be tangential to their graduate program research. Project ideas submitted by applications can be revised and refined. (however, all of our fellows submitted great project ideas!) 8 TAR Fellows in pilot year (began February 2014) 34

35. TAR Program Element: Education Research Project Fellows assigned mentor with experience in education research and topic/ discipline of research. Eg. Student studying communication skill development in psychology course assigned mentor from Writing Program Fellows and mentors meet separately and work together to refine/ revise project so that it is manageable yet compelling. Students are guided in formulating clear research questions and hypotheses, choose and design appropriate experimental methods. Students are asked to explore education literature for similar studies, understand the research “conversation” and methods used in those studies. Students must also make contact with faculty in courses or programs of interest, ensure faculty are on board with method implementation. Project presentation event at end of year. 35

36. TAR Program Element: Training Seminars, Update Meetings, Travel Small group seminars to introduce key topics or practices in education research. IRB proposal Education research databases Writing surveys Statistics More coming, Fellows have input on what’s next Monthly Fellow Community meeting: CIRTL team, fellows and mentors meet together to share project updates, help one another in solving project challenges. Fellows can also attend CIRTL Central workshops. Fellows given small stipend to support conference attendance. 36

37. Evaluation Choose TAR Fellows with articulated interest in teaching Rate entry application with extended questions on pertinent academic and personal background, reasons for desiring participation, and proposed project goals Program Element Quality Mid-year and end-of-year program evaluation surveys to provide feedback on TAR program seminars, community meetings, mentoring, suggestions for improvement. 37

38. Assessment Develop TAR Fellow Competencies Pre-TAR program survey that includes a battery of self- effiicacy questions directly related to TAR learning outcomes, and reflection questions on previous teaching and learning experiences and goals and learning style. Post-TAR program survey that parallels the pre-TAR survey questions and asks students to reflect on goals achieved during the year and plans for next steps. A mid-year reflection written and oral exercise in which TAR Fellows and mentors reflect on student progress. A jury of faculty and TAR staff will rate final student presentations of project work at the end of the year. A rubric will be developed for jury rating student work and for providing feedback to Fellows on their projects. 38

39. TAR Fellow Teaching Portfolios An essential element of the suite is the use of teaching as research portfolios so that our Fellows are engaged in reflection on goals, projects, and competency development. Fellows portfolio will include: Teaching experience Statement of teaching goals Pre/post self-efficacy survey results Open-ended reflections on progress (completed as part of program elements or surveys) Project results, jury review of project Also: notes on education research literature, research methods, conferences attended, network of contacts Will emphasize awareness of teaching portfolios as professional development. Portions can be used in job applications! 39

40. Pre/Post Program Self-Efficacy Survey Drew on CIRTL TAR Central learning outcome list to develop self-efficacy questions First asked Fellows if ever completed list of tasks. Essential reality questions! Used 0-100% confidence scale for survey questions. Students rated confidence to complete task at time of survey Will have students compare confidence to complete teaching and teaching as research tasks at beginning and end of program 40

41. Self-Efficacy Question Examples Teaching as Research: TAR Develop realistic, well-defined, achievable, measurable, student-centered learning goals for my teaching and learning project Learning Community: LC Contribute to professionally-focused learning communities associated with teaching and learning Learning through Diversity: LD Determine diverse backgrounds among a group of students, and consider opportunities and challenges of findings on each student’s learning 41

42. Pre/Post Program Reflection Questions Pre-program survey also includes reflection questions Self-reflection: understand self as learner Pedagogical approach: learning activity design, investigating and solving student comprehension difficulty Post-program survey will ask Fellows to return to pre- program questions. In ideal, pedagogical approaches will demonstrate movement toward mastery. Fellows will be able to use pre/post responses as part of their portfolio of progress. No time on task is wasted! 42

43. Program findings to date Still in first ½ of program year TAR Fellow project ideas were ambitious AND compelling! Fellows now in midst of learning what it takes to refine and complete a doable projects with valid results. In midst of realization of how much they need to learn to master teaching as research. Using TAR Fellow expertise to teach peers, build learning community (eg. stats, survey tools, building datasets) TAR Fellow projects benefit university since most projects are related to UG education concerns. 43

44. Questions? 44