From innovation to implementation A systematic approach to faculty development evaluation D. Christopher Brooks, Research Fellow, OIT-DMC Lauren Marsh, Senior Educational Technology Consultant, OIT-DMC Kimerly J. Wilcox, Senior Educational Technology Consultant, OIT-DMC

Overview Background on Digital Media Center & its programs Instrument/item construction [Very] preliminary results Q & A

Digital Media Center, OIT Investigate emerging academic technologies; Provide evaluation and research services; Offer technology training and usability services; Provide consultation services and faculty development programs. 3

Digital Media Center Development Programs TA Web Certification Digital Teaching Workshop Technology Enhanced Learning (TEL) Grants Faculty Fellowship Program 4

DMC, OIT Faculty Fellowship Program Established in 2000; Purpose: To cultivate faculty leadership in the area of technology-enhanced learning (TEL). http://www.oit.umn.edu/ http://dmc.umn.edu/fellowship/ 5

DMC, OIT Faculty Fellowship Program Small, cross-disciplinary community of instructors; Work individually and collaboratively; Scholarly investigation of how to enhance student learning through the thoughtful integration of technology into teaching and learning; Share their findings with the larger University community. 6

FFP Phase 1 (2000 - 2004) Up to one year (academic year plus summer); Bi-weekly seminar-style meetings; Individual TEL projects; Project show-and-tell at end. 7

FFP Phase 2 (2005 - 2007) Changes: Refocus on instructional design for TEL project; Prototype development; Assessment and evaluation plan. 8

FFP Phase 3 (2008 - 2009) Changes: 3-day workshop plus 10 meetings over 18 months; Theme for individual TEL projects; Collaborative project; Partnerships; Program evaluation - Pilot program. 9

Program Evaluation Overview of procedural elements Discussion of preliminary results

Process: Identify levels of analysis Institutional level Departmental level Program level Participant level

Process: Identify dimensions of interest Develop initial set of dimensions Brainstorm common dimensions with program leaders and research team Reconfigure common dimensions Refine common and brainstorm unique items with program leaders Draft final set of dimensions

Faculty Fellowship Program: Common dimensions Skill development Fostering interdisciplinary community Promoting Technology Enhanced Learning (TEL) Promoting Student Learning Outcomes (SLOs)

Faculty Fellowship Program: Unique dimensions Cultivating leadership Promoting the Scholarship of Teaching and Learning (SoTL) Transforming the instructional culture Generating organizational awareness

Process: Constructing items Operationalize definitions of dimensions Draw upon existing theories and literature Draft individual items that tap dimensions from multiple angles

Example: Leadership Modeled influence Inspirational motivation Intellectual stimulation Institutional engagement

Example: Leadership I discuss technology enhanced learning issues with my departmental colleagues. (Modeled influence) I serve as an advisory resource for colleagues on issues related to teaching and technology. (Inspirational motivation)

Example: Leadership 3) I participate in public conversations about teaching with technology. (Intellectual stimulation) 4) I am comfortable approaching administrators with my ideas about university programs and initiatives regarding technology and teaching. (Institutional engagement)

Process: Instrument construction Draft items for all dimensions Circulate and collect feedback Edit and finalize items Randomize item sequence Choose and create measurement scale Produce final document

Implementation Distribute evaluation prior to onset of program (July 2008) Distribute evaluation at six-month intervals during program (January & July 2009) Distribute evaluation at conclusion of program (January 2010)

Table 1. FFP Evaluation Questionnaire: Items with Statistically Significant Differences at p < .10 Goal Type Dimension Item Pretest Score Change Common Interdisciplinary Community I think the costs of collaboration outweigh the benefits. 1.00 0.75 Technology Enhanced Learning I use technology in teaching to communicate with students. 3.20 0.60 Unique Leadership I discuss technology enhanced learning issues with departmental colleagues. 2.60

Goal Type Dimension Pretest Mean Change Gain Score Table 2. Faculty Fellowship Program Evaluation Questionnaire: Change for Common and Unique Goals Dimensions Goal Type Dimension Pretest Mean Change Gain Score Common Interdisciplinary Community 3.25 -0.06 2.7% Skill Development 3.15 -0.15 7.0% Student Learning Outcomes 3.40 -0.28 11.7% Technology Enhanced Learning 0.24 40.0% Unique Instructional Culture 3.55 0.05 11.1% Leadership 2.80 0.25 20.8% Organizational Awareness 2.50 0.10 6.7% Scholarship of Teaching and Learning 3.16 0.16 19.0%

Questions?

Thank you! Kimerly Wilcox, Ph.D., Senior Educational Technology Consultant, OIT (wilco001@umn.edu) D. Christopher Brooks, Ph.D., Research Fellow, OIT (dcbrooks@umn.edu) Lauren Marsh, Ph.D., Senior Educational Technology Consultant, OIT (lauren@umn.edu)

What did you think? Your input is important to us! Click on “Evaluate This Session” on the Midwest Regional program page.

The University of Minnesota is an equal opportunity educator and employer. This PowerPoint is available in alternative formats upon request. Direct requests to Kimerly J. Wilcox, Digital Media Center, Office of Information Technology, 212 Walter Library, 117 Pleasant Street SE, University of Minnesota, Minneapolis, MN 55455, 612-624-3528, wilco001@umn.edu. www.umn.edu/oit