1. “When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations on the assessment of online learners.”
Oosterhof, et al. (2008) Assessing Learners Online

2. Donald Staub, Title III Director Carteret Community College
Assessing Distance Learning: Are they learning what we think we’re teaching? ? ? ? ? ? ?
Donald Staub, Title III Director Carteret Community College

4. The Lineup DL @ CCC Why Worry about DL Assessment? How We’re Doing It
The DL Program
DL in Programs & Courses
Possibilities & Challenges

5. DL @ CCC 33 programs @ CCC Enrollment ~1800 DL is 10+ years old
Spring 2011
399 total sections
261 seated sections
90 Internet (23%)
40 Hybrid (10%)
8 Web-based 5

6. Why worry about assessing DL?
@ the College & Program-Level:
Student Success
Persistence
(btw: success vs. retention vs. persistence)
“Good assessment guides good PD/ID”

7. Generalizability @ the Course-Level:
(c.f. Oosterhof, et al, 2008)
“…the degree to which learner performances that are observed generalize to performances that are not observed.”
[i.e. are they learning what we think we’re teaching?]

8. So, how are we doing this? Caveat: We’re not saying this is right or wrong…it’s just the way that we’re currently approaching it.

9. “Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”
(CS 3.3.1)
COC Distance Education Policy Statement

10. The (former) Guidelines

12. Q1: Is the distance education program reflected in the institution’s strategic (or master) plan?
Strategic Plan
We will create a culture where all institutional actions are focused on improving student learning and success and leading to student goal completion.
PM 16—Success and Withdrawal Rates by Instructional modalities

13. Q2: Is there evidence that outcomes for the program have been identified?
DL-AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.
DL-AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.

14. Assessment: The number of internet/hybrid sections offered.
Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?
AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.
Assessment: Quality Assessment Plan, Success/ Persistence Rates, Student Surveys/Evaluations.
AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.
Assessment: The number of internet/hybrid sections offered.

15. QAP

16. The QAP (Quality Assessment Plan)
Developed in-house (using models)
Peer-reviewed by content specialist
Stipends (T3)
All current online courses have been reviewed
All new online courses must be reviewed/ certified

17. The QAP Score Card
points is the total possible a teacher can earn - believe its 91
minimum is what they need to pass - I am happy if they get 63 or above (hybrids may do feedback in class or individually so we learned the way the QAP is now hybrid instructors may get lower scores due to no feedback to students even through they may do it during or after class - so we are modifying this for hybrids next go around
earned is what they actually got so Laurie Freshwater may have earned a 70 and Mary Walton a 67 - below 60 we have them come back to Pre-Ah to do a 2 hour meeting to fix what needs to be addressed.

18. The QAP Score Card: Learning

19. QAP Success/Persistence

20. seated: 75% success (persist: 87.1%)
Success - Spring 2010
75 sections
seated: 75% success (persist: 87.1%)
internet: 66% success (persist: 81.7%)
hybrid: 73% success (persist: 87.8%)
FA09 internet: 62 sections; 68% success
FA08 internet: 61 sections; 67% success
FA07 internet: 59 sections; 66% success
FA06 internet: 48 sections; 66% success 20

21. QAP Success/Persistence Program/Course Evaluations

22. Surveys & Evaluations: DL Program

23. Surveys & Evaluations: DL Program23

24. Surveys & Evaluations: Course-Level

25. ILLOs (Gen Ed outcomes)
Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?
Modality Comparison
Grade Distributions
Program Reviews
ILLOs (Gen Ed outcomes)

26. Modality Comparison

27. Modality Comparison
Institutional-level

28. Modality Comparison
Program-level

29. Modality Comparison
Course-level

30. Modality Comparison Grade Distributions

31. Grade Distributions institution-level

32. Grade Distributions course-level

33. Modality Comparison Grade Distributions Program Reviews

34. Evaluate Instructional Modalities

35. Modality Comparison Grade Distributions Program Reviews ILLOs

36. ILLOs (Gen Ed outcomes)
Communication
Computer Literacy
Critical Thinking
Humanities & Fine Arts
Information Literacy
Personal Growth & Responsibility

37. Computer Literacy - CIS 110
Outcome: Students demonstrate proficiency in the use of computer technology including application software, system software, and internet browsers.
Assessment: Final Exam in CIS 110.

38. Computer Literacy - CIS 110 Section analysis

39. Computer Literacy - CIS 110
Same instructor teaching one section of each

40. Q4: Has the institution developed student learning competencies for the courses/programs offered by distance education? If these are the same competencies for courses/programs offered by “traditional” methodologies, is assessment identified for distance learning students separate from students taking courses by “traditional” methodologies?

41. So, are they learning what we think we’re teaching?
“Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”
So, are they learning what we think we’re teaching?

42. And the answer is…
Yes! … No?…maybe??
PLLOs
CLLOs

43. PLLOs

44. PLLOs (SLOs)
Same instructor teaching one section of each (common assignment from each section)
Business Administration
ACC 120
BUS 110
ECO 251/252

45. PLLO: preparation of financial statements as part of mid-term Assessed in: ACC 12045

46. BUS 110
ECO 250

47. PLLOs CLLOs

48. Course-level assessment - CLLOs
ACC 120
ART 111
EDU 119
ENG 111
HIS 111
PSY 150 & 241
RCP 114
SOC 210 48

49. Course-level assessment ~ CLLOs
BUS 110 … Students will be able to:
A: Explain Business Ethics and Social Responsibility
B: Describe the major concepts of business, such as Entrepreneurship, Management, Marketing, Human Resource Management, and Financing
C: Describe Business Globalization

51. Use of Results: “The on-line section needs some extra material to ‘reinforce’ or explain globalization.”

52. “Students placed more value on the CMS as an effective learning tool when they perceived the CMS as supporting interactivity among student, instructor, peers, content.” (Chou, Peng, Chang, 2010)

53. ART 111

54. A - ( ): - You posted 3 or more times and you completed the Assignment with a well thought through response. (The more you post MEANINGFUL responses over 3 the higher your grade. Though welcome and expected, please understand that a One or Two sentence "feel good" post such as (... Great response Michaela!!) or (... I love how you put the meaning and then went down and talked about them. it made it easy to understand!!) will count for very little towards your grade. 54

55. HIS 111
Essays (online) vs. Research Paper (seated) … same rubric

56. RCP 114

57. Other Possibilities
Downstream assessment
E-ports 57

58. Challenges

59. “I have to pull teeth to get them to post even when they get a score”
“I got rid of [the discussion board]…it’s a pain in the ass.”
“I have to pull teeth to get them to post even when they get a score”
“Value of discussion board is suspect at best.”

60. “I don’t know” [what texts and assessments the other instructors are using in this multi-section course]
“My first goal is to get the instructors to agree on common CLLOs. Then we’ll work on common assessments.”
Outcome-Assessment alignment

61. Title III Director Carteret Community College staubd@carteret.edu
Donald Staub
Title III Director Carteret Community College