Cluster 5 Spring 2005 Assessment Results Sociocultural Domain

Cluster 5 Course Requirement  Complete 1 course in each domain Wellness Domain GHTH 100. Personal Wellness GKIN 100. Lifetime Fitness and Wellness GEIC 101. Individuals in the Human Community: The Individual Perspective Sociocultural Domain GPSYC 101. General Psychology GPSYC 160. Life Span Human Development GSOCI 240. Individual in Society GEIC 102. Individuals in the Human Community: The Community Perspective

Assessment Day Data Collection Scheme: Pre/Post COHORT 1 COHORT 2 COHORT 3 Students in each cohort are tested twice on the same instrument – once as incoming freshmen (0 credit hours) and again in after they’ve completed credit hours. Cohort 2: First time for Cluster 5 to have pre/post data since new objectives/instruments.

Sociocultural Domain Instruments  Sociocultural Domain Assessment (SDA) –3 different versions of test  Scenario-based multiple-choice exam created by faculty Administration error led to SDA1 & SDA2 being administered in Spring 2005 – because of this only 27 common items on both forms used to create total score

Results  First: examine results for nd semester sophomores in Spring 2005 –Overall results –By JMU vs. Transfer/AP –By domain “completers” vs. “non-completers” –By which course completed –Rel. between course grade and SDA score  Second: examine pre/post results for the 264 students who took the SDA in both Fall 2003 and Spring 2005 –Overall pre/post difference –pre/post difference by domain “completers” vs. “non- completers” –pre/post difference by which course completed –Pre/post difference by item for “completers”

Overall SDA Results: 2 nd Semester Sophomores (N = 398) On average, students are answering almost 70% of the items correctly. Most scores are between 17 & 23.

JMU (n = 369) vs. Transfer/AP Students (n = 29)  Are students who obtained transfer or AP credit for their courses scoring differently than students who completed their courses at JMU? –No.  JMU students: M = 18.63, SD = 3.83  AP/transfer students: M = 18.55, SD = 3.68  Difference was not statistically, t(335)=.11, p =.92, nor practically significant (d =.02).

Domain “Completers” vs. “Non-Completers” More courses completed, the higher the mean. Differences among the means, however, are of small practical significance (d’s between )

SDA Means by Course Completed “Completers” Only There were minor differences among the SDA means (d’s ~ ) GEIC101 not reported due to small N

Relationship between SDA and Course Grade r =.31r =.23 r =.69r =.36 w/o outlier Positive and moderate relationships …as anticipated.

Pre/Post Results (N = 264)  Spring 2005: M = 18.78, SD = 3.66  Fall 2003: M = 16.62, SD = 3.38  The difference between these means is of medium to large practical significance (d =.64) Average increase of 8% in sociocultural domain knowledge

Pre/Post for “Non-Completers” (n = 33) vs. “Completers” (n = 199) Non-completersCompleters “Completers” start off with higher means than “Non- completers” – but they both make similar gains over time (~2 pt. increase, d ~.55) Larger gain (~3 pts, d = 1.02) for the 32 students who completed 2 – 3 sociocultural domain courses.

Pre/Post by Sociocultural Course Completed Spring 2005 average for GSOCI240 students may seem superior, keep in mind that these students were more knowledgeable coming into the course and that the averages for GSOCI240 are based on only 10 students. The magnitude of the change in knowledge is very similar for the three different courses (d ~.75). N = 153 N = 98N = 10

Pre/Post Difference in Items for “Completers” 7 items in Fall 2003 where less than half of the students are obtained the correct answer; there are only 2 such items in Spring 2005 Most of the gains are made for items of moderate to large difficulty.

Items with Large Pre/Post Gains 7% or more students answered these items correctly in Spring 2005 than in Fall 2003 Majority of these items (67%) were written to assess goals 2 and 3. Goal 2:Identify implications of taking action regarding social/behavioral issue Goal 3:Use evidence to develop and evaluate positions regarding social/behavioral issue

Items with Small (or No) Pre/Post Gains Items with no change, or change in the wrong direction are shown below for items that are of medium to extreme difficulty. 67% of these items are assessing either goals 1 and 4, which indicates that: 1) either more focus should be placed these goals in sociocultural domain courses or 2) that these items are of poor quality and either need to be revised or replaced Goal 1: Make plausible interpretations about behavior in social contexts Goal 4: Discriminate between ethical and nonethical practices in the social/behavioral sciences

Conclusions 2 nd Semester Sophomore Data  2 nd semester sophomores scoring on average ~70%  AP/Transfer students are scoring no differently than JMU students  “Non-completers” of the sociocultural domain requirement are scoring no differently than “Completers”  Same average score regardless of what course student took to complete sociocultural domain requirement  There is a relationship between course grade and score, indicating that test content and course content are related

Conclusions Pre/Post Data(1)  Average increase of 8% in sociocultural domain knowledge between 0 credit hours and credit hours (large effect) –62%  70%  “Completers” and “Non-completers” both make similar gains over time (~2 pt. increase, d ~.55) –Might be evidence that score gains are a result of maturation, not a result of taking sociocultural course –Note that “completers” come in with more knowledge. –Larger gains (~3 pts, d = 1.02)are seen for students completing 2 – 3 courses. –May need to collect larger sample of “non-completers” in future assessments

Conclusions Pre/Post Data(2)  Same gains are being made on the test for students who fulfilled their requirement by taking different sociocultural courses –Students who take SOCI240 may be more knowledge coming into the course than students who choose to take other courses  7 items where <50% of students obtaining correct response in Fall 2003 compared to 2 in Spring 2005  Not seeing gains on easy items  Most gains are for items assessing Goals 2 & 3  No gains (or gains in the wrong direction) for items assessing Goals 1 & 4

Next Steps  Take a closer look at change in % of students correctly answering items over time –Use information to make inferences about program OR possibly to revise items  Set standard score on test in order to obtain richer information

Setting Standards  May want to use standard setting procedures to establish “cut-offs” for proficiency on the test  Could be that students are gaining knowledge/skills over time, but are they gaining enough?