General Education Science Assessment Leadership Team Assessment Day May 2017 General Education Science Assessment Leadership Team Victor Bondzie, Timothy Barnett, and Rita Luther

This assessment day focuses on at-risk students

Graduation Rates (%) 6.2% 10.7% 9.2% 3.4% Other? SE Asian Is the data interpreted per time point as 19.4%, 32.2%, and 36.6% of AA, H, C students, respectively graduate in five years? C and H students show 10.7% and ~9.2%,resp increase over the 8 year period in 5 yr grad rate, while AA student show only ~6.2% increase. 6.2% 10.7% 9.2% 3.4%

Five Year Graduation Rate By Success in First Five Courses Graduation Rates (%) Five Year Graduation Rate By Success in First Five Courses Are the five course any course or Gen Ed courses only? Any course If successful in specified #classes then that is your rate of graduate 8.9% 7.7% 4.2% 0.8%

Success Rates by Gender/Ethnicity All Fall 2016 Courses – College-Wide Ethnicity Gender Is success rate for the first five courses and are students FTIC? How well they do in their classes (A, B or C) Failure rate D, W, F, I

Four At-Risk Student Populations Identified: From Achieving the Dream to the Academic Initiative Review at Valencia College Four At-Risk Student Populations Identified: Students not successful in any of their first five courses or do not enroll twice in their first four terms are not as likely to graduate. New students attending college for the first time (FTIC) “Student success is largely determined by a student’s experience during his or her first-year in college” (Upcraft, et al. 2005) Students taking math and identified as in need of support (developmental) 2015 Strategic Report found that 46% of student success in developmental math from Fall 2011 to Summer 2013 semesters. Students who are among those that show declining rates of persistence, such as African American males.

Fall 2016 Survey of At-Risk Students Indicated: Some students are using and trying to follow advice given by support services. Students want more feedback and in some cases more direction from faculty. They want to learn strategies for engaging faculty, learning support and advising at Valencia College. Developmental Education Research 2016-17 English courses: Survey of students (n=1,387 13% response rate), 1 focus group (n=7), and 7 student interviews Math courses: Survey of students (n=1,226 12% response rate), 3 focus groups (n=23), and 5 student interviews Nichole Jackson, Assistant Director of Learning Assessment, Valencia Institutional Assessment

Students who are at risk tell us they need: Support transitioning from high school to college Strategies making the most of services available Preparation to achieve in courses higher than their current ability level Encouragement, incentives, or requirements in order to seek help A connection to support services: that will deepen their knowledge be available for them to use repeatedly and regularly in person and online

Agenda Items Review Learning Outcomes Review 2017 Spring Data Discuss 2016-17 Outcomes Resulting From Data Discuss Future Directions of the Science Gen. Ed. Work

REVIEW OF LEARNING OUTCOMES Outcome #1: Critical Thinking- Effectively analyze, evaluate, synthesize, and apply information and ideas from diverse sources and disciplines. Faculty developed (2014-2015) and employed Biology, Chemistry, and Physics scenarios to address Observation, Hypothesis, Data and Conclusions. Outcome #2: Quantitative and Scientific Reasoning- Students should be able to use processes, procedures, data, or evidence to solve problems and make effective decisions - Not yet attempted.

REVIEW OF 2017 SPRING DATA

Current Expectations The students should be able to apply the scientific method. Expectations assessed through student analysis of the scientific method in the context of a specific scientific problem. We expect that at least 3000 student participants. We also expect that 70% or more of our students will earn a grade of 75% or higher.

Summary of Spring 2017 Data Overall Responses Fall 2015 Fall 2016 Spring 2016 Spring 2017 Students Invited 11442 12448 11296 11862 Total Responses 3179 3222 2406 2892 % Responded 27.78 25.88 21.30 24.38 Minimum One Question Completed Taking ≥ 4 Minutes Fall 2015/Spring 2016-Chemistry Fall 2016/Spring 2017-Physics Fall 2015 Fall 2016 Spring 2016 Spring 2017 Total Responses 2230 2300 1590 1973 % Responses 19.49 18.48 14.08 16.63 Mean Score 1.76 2.33 2.00 2.34 No. Scores ≥ 75% (% Total) 587 (26.32%) 1051 (45.70%) 544 (34.21%) 905 (45.87%)

Science Gen Ed Assessment “Passing” Rates % Students Scoring ≥ 75% Biology Assessment Chemistry Assessment Physics Assessment

Distribution of Scores

Percent Correct Per Question % Students Not all questions correct only a single question correct Individual question basis

Participation By Campus

Faculty Participation 33% 30% 20% 16%

QUESTION BREAKDOWN

Question #1: Which of the following observations support conditions for the existence of water? Observations Question #2: A scientist graphed the event of a single temporary drop in the brightness of a star that was observed during an observation period of 1,500 days this way: The scientist created 4 additional graphs to show the timing of brightness changes in four other stars that were also observed for 1,500 days each. If each planet is described by a distinct drop, which of the following graphs would be most consistent with the presence of two planets circling a star, rather than just 1 planet? Hypothesis/Prediction/Data Analysis Question #3: Which of the following statements is true from the experimental data obtained by the Kepler team? Data Analysis Question #4: Which of the following statements can reasonably be concluded? Conclusions

Question #1: Which of the following observations support conditions for the existence of water? Fall 2016 Spring 2017 Answer # Responses % Responses A) The core of these new planet is similar to our own Earth 191 8% 162 B) The majority of the planets are smaller than four times the size of the Earth 121 5% 94 C) All of the recent discovered worlds are located in multi-planet systems 145 6% 149 D) Four of the newly discovered planets orbit in their sun’s habitable temperature zone 1843 80% 1568 79% Total 2300 100% 1973

Question #2: A scientist graphed the event of a single temporary drop in the brightness of a star that was observed during an observation period of 1,500 days this way: The scientist created 4 additional graphs to show the timing of brightness changes in four other stars that were also observed for 1,500 days each. If each planet is described by a distinct drop, which of the following graphs would be most consistent with the presence of two planets circling a star, rather than just 1 planet? Fall 2016 Spring 2017 Answer # Responses % Responses A) 391 17% 329 B) 562 24% 471 C) Correct Answer 1215 53% 1021 52% D) 127 6% 149 8% Total 2300 100% 1973

Question #3: Which of the following statements is true from the experimental data obtained by the Kepler team? Fall 2016 Spring 2017 Answer # Responses % Responses A) Approximately 35 of the 715 new planets are bigger than the size of Neptune 964 42% 854 43% B) Approximately 7% of the 715 new planets are less than two-and-a-half times the size of Earth 288 13% 267 14% C) Approximately 500 of the 715 new planets are smaller than the size of Neptune 332 266 D) Approximately 95% of the 715 new planets are located in multi-planet systems 716 31% 586 30% Total 2300 100% 1973

Question #4- Which of the following statements can reasonably be concluded? Fall 2016 Spring 2017 Answer # Responses % Responses A) There is a high probability that if two cats are seen that they could be two lions 464 20% 374 19% B) There is a high probability that if more than two cats are seen there would be a lion and multiple lionesses 1331 58% 1172 59% C) There is a high probability that if two cats are seen they could be two lionesses 111 5% 97 D) There is an equal probability that if two cats are seen that they could be two lions or two lionesses 394 17% 330 Total 2300 100% 1973

Spring 2017: Question #1 Disaggregated Data By Ethnicity and Gender # Responses (%) Males # Responses (%) Females Answer African American Hispanic Caucasian A) The core of these new planet is similar to our own Earth 4 (6%) 16 (7%) 9 (5%) 25 (12%) 40 (9%) 27 (8%) B) The majority of the planets are smaller than four times the size of the Earth 5 (7%) 8 (3%) 7 (4%) 18 (9%) 25 (5%) 16 (5%) C) All of the recent discovered worlds are located in multi-planet systems 12 (5%) 29 (14%) 44 (10%) 25 (7%) D) Four of the newly discovered planets orbit in their sun’s habitable temperature zone 57 (80%) 205 (85%) 175 (88%) 133 (65%) 346 (76%) 281 (81%) Total 71 (100%) 241 (100%) 200 (100%) 205 (100%) 455 (100%) 349 (100%)

Spring 2017: Question #2 Disaggregated Data By Ethnicity and Gender # Responses (%) Males # Responses (%) Females Answer African American Hispanic Caucasian A) 10 (14%) 31 (13%) 26 (13%) 30 (15%) 68 (15%) 65 (19%) B) 16 (23%) 50 (21%) 40 (20%) 61 (30%) 127 (28%) 81 (23%) C) Correct Answer 40 (57%) 142 (59%) 121 (61%) 90 (44%) 216 (48%) 181 (52%) D) 4 (6%) 18 (7%) 13 (7%) 24 (12%) 42 (9%) 22 (6%) Total 70 (100%) 241 (100%) 200 (100%) 205 (100%) 455 (100%) 349 (100%)

Spring 2017: Question #3 Disaggregated Data By Ethnicity and Gender # Responses (%) Males # Responses (%) Females Answer African American Hispanic Caucasian A) Approximately 35 of the 715 new planets are bigger than the size of Neptune 24 (34%) 114 (47%) 110 (55%) 57 (28%) 161 (35%) 163 (47%) B) Approximately 7% of the 715 new planets are less than two-and-a-half times the size of Earth 8 (11%) 23 (10%) 20 (10%) 40 (20%) 81 (18%) 37 (11%) C) Approximately 500 of the 715 new planets are smaller than the size of Neptune 33 (14%) 18 (9%) 25 (12%) 79 (17%) 56 (16%) D) Approximately 95% of the 715 new planets are located in multi-planet systems 31 (44%) 71 (29%) 52 (26%) 83 (40%) 134 (29%) 93 (27%) Total 71 (100%) 241 (100%) 200 (100%) 205 (100%) 455 (100%) 349 (100%)

Spring 2017: Question #4 Disaggregated Data By Ethnicity and Gender # Responses (%) Males # Responses (%) Females Answer African American Hispanic Caucasian A) There is a high probability that if two cats are seen that they could be two lions 11 (15%) 41 (17%) 22 (11%) 54 (26%) 99 (22%) 60 (17%) B) There is a high probability that if more than two cats are seen there would be a lion and multiple lionesses 40 (56%) 158 (66%) 146 (73%) 101 (49%) 258 (57%) 208 (60%) C) There is a high probability that if two cats are seen they could be two lionesses 2 (3%) 12 (5%) 6 (3%) 19 (4%) 15 (4%) D) There is an equal probability that if two cats are seen that they could be two lions or two lionesses 18 (25%) 30 (12%) 26 (13%) 28 (14%) 79 (17%) 66 (19%) Total 71 (100%) 241 (100%) 200 (100%) 205 (100%) 455 (100%) 349 (100%)

Fall 2016: Question #1-4 Disaggregated Data By Grade (% Percent Correct) Spring 2017 Grade Q #1 Q #2 Q #3 Q #4 A 641 (86%) 451 (60%) 391 (52%) 483 (64%) 584 (84%) 407 (58%) 384 (55%) 445 (64%) B 633 (80%) 413 (52%) 341 (43%) 446 (56%) 525 (80%) 328 (50%) 283 (43%) 385 (59%) C 483 (79%) 296 (49%) 221 (36%) 343 (56%) 339 (77%) 204 (47%) 163 (37%) 253 (58%) D 114 (70%) 77 (48%) 45 (28%) 84 (52%) 89(71%) 62 (50%) 38 (30%) 68 (54%) F 39 (70%) 20 (36%) 17 (30%) 26 (46%) 42 (66%) 28 (44%) 12 (19%) 33 (52%)

STUDENT SCORING BY SUNDRY VARIABLES

Student Scoring by Sundry Variables Format with titles

Student Scoring by Sundry Variables Scores by Number of Science Courses Completed Prior to Fall 2016

Student Scoring by Sundry Variables Scores by Discipline

SUMMARY/DISCUSSION OF SPRING 2017 DATA Student responses and distribution of answers overall were similar between Fall 2016 and Spring 2017 Sci Gen Ed Assessments Data indicated a consistent discord between ethnic groups within males and female student populations for correct answer selections in the Sci Gen Ed Assessment with african american students showing the lowest percentage of students selecting correct answers. This was followed by hispanic students and finally caucasian students showing the highest percentage of students selecting correct answers. Data also indicated gaps between male and female students in question performance for african american and hispanic female students. Caucasian female students had a higher percentage of students selecting correct answers among females and in some cases overlapping with the performance of some male populations.

SUMMARY/DISCUSSION OF SPRING 2017 DATA Trends shown in proportion of student with correct answers based on course grades Sundry variables: Comparable frequencies of students completed the tutorial between Fall 2016 and Spring 2017 Mean scores of those completing the tutorial are slightly higher, but not a striking difference (too early) Slightly higher mean scores for students that completed more science courses, but not a striking difference. Physics, chemistry, biology, earth science, horticulture (Score performance differences from higher to lower)

FUTURE DIRECTIONS OF SCI GEN ED WORK How best do we assist at-risk students? (Mentoring, tutoring, NSE, other ideas) Where do we go from here… The fall and spring data is almost identical. Can we run one assessment each fall and know that it will be representative of students overall? (This would give you much more time to move from results in fall to action plan development in spring towards assessment day). Should we run biology assessment (edited) in Fall 2017? How do we get full adjunct faculty buy-in to assessment to increase student participation? We need new volunteers to serve on the Assessment team with Rita for two years Further discussions on assessment day. Run in Fall only since data between semesters is similar and do some action thing in spring Bob-Instead of multiple choice, do a lab report, random sampling to grade. Full vs sampling data If lab report, may be more motivated to complete. Provide uniform rubric and sample lab report (good example)

Thank you!