1. From innovation to implementation
A systematic approach to faculty development evaluation
D. Christopher Brooks, Research Fellow, OIT-DMC
Lauren Marsh, Senior Educational Technology Consultant, OIT-DMC
Kimerly J. Wilcox, Senior Educational Technology Consultant, OIT-DMC

2. Overview Background on Digital Media Center & its programs
Instrument/item construction
[Very] preliminary results
Q & A

3. Digital Media Center, OIT
Investigate emerging academic technologies;
Provide evaluation and research services;
Offer technology training and usability services;
Provide consultation services and faculty development programs. 3

4. Digital Media Center Development Programs
TA Web Certification
Digital Teaching Workshop
Technology Enhanced Learning (TEL) Grants
Faculty Fellowship Program 4

5. DMC, OIT Faculty Fellowship Program
Established in 2000;
Purpose: To cultivate faculty leadership in the area of technology-enhanced learning (TEL). 5

6. DMC, OIT Faculty Fellowship Program
Small, cross-disciplinary community of instructors;
Work individually and collaboratively;
Scholarly investigation of how to enhance student learning through the thoughtful integration of technology into teaching and learning;
Share their findings with the larger University community. 6

7. FFP Phase 1 (2000 - 2004) Up to one year (academic year plus summer);
Bi-weekly seminar-style meetings;
Individual TEL projects;
Project show-and-tell at end. 7

8. FFP Phase 2 ( )
Changes:
Refocus on instructional design for TEL project;
Prototype development;
Assessment and evaluation plan. 8

9. FFP Phase 3 ( )
Changes:
3-day workshop plus 10 meetings over 18 months;
Theme for individual TEL projects;
Collaborative project;
Partnerships;
Program evaluation - Pilot program. 9

10. Program Evaluation Overview of procedural elements
Discussion of preliminary results

11. Process: Identify levels of analysis
Institutional level
Departmental level
Program level
Participant level

12. Process: Identify dimensions of interest
Develop initial set of dimensions
Brainstorm common dimensions with program leaders and research team
Reconfigure common dimensions
Refine common and brainstorm unique items with program leaders
Draft final set of dimensions

13. Faculty Fellowship Program: Common dimensions
Skill development
Fostering interdisciplinary community
Promoting Technology Enhanced Learning (TEL)
Promoting Student Learning Outcomes (SLOs)

14. Faculty Fellowship Program: Unique dimensions
Cultivating leadership
Promoting the Scholarship of Teaching and Learning (SoTL)
Transforming the instructional culture
Generating organizational awareness

15. Process: Constructing items
Operationalize definitions of dimensions
Draw upon existing theories and literature
Draft individual items that tap dimensions from multiple angles

16. Example: Leadership Modeled influence Inspirational motivation
Intellectual stimulation
Institutional engagement

17. Example: Leadership
I discuss technology enhanced learning issues with my departmental colleagues.
(Modeled influence)
I serve as an advisory resource for colleagues on issues related to teaching and technology.
(Inspirational motivation)

18. Example: Leadership
3) I participate in public conversations about teaching with technology.
(Intellectual stimulation)
4) I am comfortable approaching administrators with my ideas about university programs and initiatives regarding technology and teaching.
(Institutional engagement)

19. Process: Instrument construction
Draft items for all dimensions
Circulate and collect feedback
Edit and finalize items
Randomize item sequence
Choose and create measurement scale
Produce final document

20. Implementation
Distribute evaluation prior to onset of program (July 2008)
Distribute evaluation at six-month intervals during program (January & July 2009)
Distribute evaluation at conclusion of program (January 2010)

21. Table 1. FFP Evaluation Questionnaire: Items with Statistically Significant Differences at p < .10
Goal Type
Dimension
Item
Pretest Score
Change
Common
Interdisciplinary Community
I think the costs of collaboration outweigh the benefits.
1.00
0.75
Technology Enhanced Learning
I use technology in teaching to communicate with students.
3.20
0.60
Unique
Leadership
I discuss technology enhanced learning issues with departmental colleagues.
2.60

22. Goal Type Dimension Pretest Mean Change Gain Score
Table 2. Faculty Fellowship Program Evaluation Questionnaire: Change for Common and Unique Goals Dimensions
Goal Type
Dimension
Pretest Mean
Change
Gain
Score
Common
Interdisciplinary Community
3.25
-0.06
2.7%
Skill Development
3.15
-0.15
7.0%
Student Learning Outcomes
3.40
-0.28
11.7%
Technology Enhanced Learning
0.24
40.0%
Unique
Instructional Culture
3.55
0.05
11.1%
Leadership
2.80
0.25
20.8%
Organizational Awareness
2.50
0.10
6.7%
Scholarship of Teaching and Learning
3.16
0.16
19.0%

26. Questions?

27. Thank you!
Kimerly Wilcox, Ph.D., Senior Educational Technology Consultant, OIT
D. Christopher Brooks, Ph.D., Research Fellow, OIT
Lauren Marsh, Ph.D., Senior Educational Technology Consultant, OIT

28. What did you think? Your input is important to us!
Click on “Evaluate This Session” on the Midwest Regional program page.

29. The University of Minnesota is an equal opportunity educator and employer. This PowerPoint is available in alternative formats upon request. Direct requests to Kimerly J. Wilcox, Digital Media Center, Office of Information Technology, 212 Walter Library, 117 Pleasant Street SE, University of Minnesota, Minneapolis, MN 55455, ,