1. Quality: Becoming what you measure Chère Campbell Gibson Presented at the 17 th Annual TelecoopTechnology Conference

2. The inspiration Dunning, B., Van Kekerix, M., & Zaborowski, L. (1993). Reaching learners through tele- communications. San Francisco, CA: Jossey- Bass Publishers.

3. Our focus Quality standards over time Core quality criteria for e-learning Benchmarking quality criteria Ethics and values in quality assessment

4. Old measures of quality “The historic pattern in post-secondary education has been to weigh the quality of non- traditional education by how closely it mirrors the format and content of education in a traditional classroom.” Reaching learners through telecommunications, (1993), (p. 187)

5. Later assessments of quality Enrollment - number of learners, often with an emphasis on the dollars generated Attrition – number of students who drop out of the program they enrolled in leading to research on learner characteristics and later to program characteristics or organizational obligations

6. Organizational obligations Organizational obligations, i.e., program features deemed important for program quality emerged in the 1990s and addressed distance education in general.

7. Organizations involved WICHE American Council on Education Pew Foundation Middle States Association of Schools and Colleges American Distance Education Consortium

8. Common features Curriculum Faculty support Learner support Technology support & infrastructure Institutional support Evaluation Cost

9. Later developments Began to focus more specifically on the online environment including efforts of such groups as: WCET Sloan Foundation Institute for Higher Education Policy (IHEP)

10. Common features Curriculum/Course development Teaching/Learning (more overt emphasis) Faculty support Learner support Course structure Institutional support Evaluation/Assessment Cost

11. A question emerges But one might ask, What are the hallmarks of quality in e-learning?

12. Core quality criteria emerge Two studies in the last five years explored core quality criteria for online learning Both began with literature reviews, adding Case studies – 6 institutions (IHEP) Delphi - 54 distance education experts (Mariasingam) Both studies resulted in 24 core criteria – 75% in common

13. Institutional support A documented technology plan that includes electronic security measures is in place and operational. The reliability of the technology delivery system is as failsafe as possible. A centralized system provides support for building and maintaining the distance education infrastructure.

14. Course development Instructional materials are reviewed periodically to ensure they meet program standards. Courses are designed to require students to engage in analysis, synthesis, and evaluation as part if their course and program requirements.

15. Teaching/learning Student interaction with faculty and other students in an essential characteristic and is facilitated in a variety of ways, including voice mail and/or email. Feedback to students on assignments and questions is constructive and provided in a timely manner.

16. Course structure Before starting an online program, students are advised about the program to determine if they possess the self-motivation and commitment to learn at a distance, and If they have access to the minimal technology required by the course design.

17. Course structure con’t Students are provided with supplemental course information that outlines course objectives, key concepts and ideas, learning outcomes summarized in a clearly written statement. Students have access to sufficient library resources that may include a virtual library…

18. Student support Students receive information about programs including admission requirements, tuition and fees, books and supplies, technical and proctoring requirements and student support services. Throughout the duration of the course or program, students have access to technical assistance including detailed instruction…

19. Student support, con’t Questions directed to student service personnel are answered accurately and quickly with a structured system in place to address complaints.

20. Faculty support Technical assistance in course development is available & faculty are encouraged to use it Faculty members are assisted in the transition from classroom teaching to online instruction and are assessed during the process. Instructor training/assistance …continues throughout the progression of the course.

21. Evaluation and assessment The program’s educational effectiveness and teaching/learning processes are assessed through an evaluation process that uses several methods and applies specific standards. Data on enrollment, costs and successful/ innovative uses of technology are used to evaluation program effectiveness.

22. Evaluation & assessment con’t Intended learning outcomes are reviewed regularly to ensure clarity, utility and appropriateness.

23. Areas of disagreement Prescriptive teaching practices, both in terms of process and content Supportive environment, including faculty incentives, teaching awards, etc. Ongoing quality assurance/TQM Involvement of stakeholders, especially employers, in assessment.

24. Missing feature Benchmarks - A reference point serving as a standard for comparing or judging other things New International Webster’s Concise Dictionary of the English Language (2000)

25. Examples A core criterion: Before beginning an online program students are advised about the program to determine if they possess the self-motivation and commitment to learn at a distance.

26. Benchmarks Students are provided written documents that help them determine their suitability, including motivation and commitment - MINIMUM Students are required to conduct an online self-analysis and discuss results in a personal advisory session with student counselor or academic advisor - BEST

27. Next steps We need to: Determine which quality criteria we feel are appropriate for our education and training programs and Establish benchmarks for those criteria

28. Our successes As a field we have: Moved beyond acceptance of marginal standards; Identified core quality criteria; Recognized the absence of quality benchmarks in the truest sense of the word; BUT

29. A question Do the criteria found in the literature and presented by a variety of agencies and organizations fit the values of your institution?

30. Beyond functional quality Ethical outcomes as a measure of quality Equity Climate Autonomy Relationships Others that fit your organizational values and purposes

31. An easy example “Before beginning an online program students are advised about the program to determine if they …have access to the minimal technology required by the course design.” What it doesn’t add is “If not, the institution will ensure access to appropriate technology …”

32. Equity Distance education provides access but for whom? Communities of color Lower income individuals Persons with disabilities Geographically distant learners Speakers of English as a second language Those with limited educational background

33. Climate Are we creating a positive environment for learning? Are the interactions among and between the teachers, learners, content, the organization, the technology that unites all conducive to learning?

34. Autonomy “Another potential measure of quality is the degree to which programming supports and advances differences and self-directedness among learners.” Reaching learners through telecommunications (1993), p. 202

35. Relationships Learner as customer – the buyer-seller relationship versus Relationships built on meeting unique needs and mutual concerns and desires, recognizing neglected constituencies and striving to overcome barriers to participation

36. A truism What a manager chooses NOT to measure indicates which outcomes are less valued. And you become what you measure!

37. And so you must ask What have I failed to value through my quality assessment? It says a lot to others, including your learners!

38. Thank you Chère Campbell Gibson ccgibson@wisc.edu