1. Preparation “A Glass Half Full” Please organize yourselves into groups of 5-6. Please organize yourselves into groups of 5-6. Note cards for comments will be distributed prior to the beginning of the session and are on your tables. Note cards for comments will be distributed prior to the beginning of the session and are on your tables. Please select a different recorder for each group activity. Please select a different recorder for each group activity.

2. A Glass Half Full: CETP Core Evaluation Frances Lawrenz Karen Appeldoorn University of Minnesota

3. What are the various sources of Core data and what are their strengths and weaknesses? Briefly discuss the above question. Briefly discuss the above question. Record your responses on note cards. Record your responses on note cards. We will pick them up from each group. We will pick them up from each group.

4. Data Sources Used by the Core Higher Education: PIs or Evaluators, Deans or Department Chairs, STEM and STEM Education Faculty, Students, NSF Student Scholars Higher Education: PIs or Evaluators, Deans or Department Chairs, STEM and STEM Education Faculty, Students, NSF Student Scholars K-12: Principals, CETP and NonCETP Teachers, CETP and NonCETP Students, NSF Graduated Scholars K-12: Principals, CETP and NonCETP Teachers, CETP and NonCETP Students, NSF Graduated Scholars Both: Classroom Observers, Classroom Activities, Student Assessments Both: Classroom Observers, Classroom Activities, Student Assessments

5. Make and record predictions about the following – we will return to these at the end of the presentation: How often did HE faculty think students worked on real-world issues? How often did HE faculty think students worked on real-world issues? How helpful did HE students find doing data collection and analysis? How helpful did HE students find doing data collection and analysis? How often did K-12 teachers believe students had enough time to learn what was required? How often did K-12 teachers believe students had enough time to learn what was required? How helpful did K-12 students find completing portfolios? How helpful did K-12 students find completing portfolios? What % of Deans do you think said their faculty had formalized interaction with K-12? What % of Deans do you think said their faculty had formalized interaction with K-12? What did PIs say was the most important outcome? What did PIs say was the most important outcome?

6. What is your definition of standards-based reform? Determine a definition in your group. Determine a definition in your group. Write your definition on a note card. Write your definition on a note card. We will collect the cards and ask for a show of hands on selected items. We will collect the cards and ask for a show of hands on selected items.

7. What is standards-based reform? Standards-based reform: identifying challenging academic standards for what all students should know and be able to do and aligning other aspects of the system, such as testing, accountability, teacher certification, and professional development, with the new standards Standards-based reform: identifying challenging academic standards for what all students should know and be able to do and aligning other aspects of the system, such as testing, accountability, teacher certification, and professional development, with the new standards

8. National Teaching Standards Science and Mathematics Science and Mathematics SCIENCE AND MATH STANDARDS: Inquiry- based program: Select content to meet interests, knowledge, understanding, abilities and experience of students SCIENCE AND MATH STANDARDS: Inquiry- based program: Select content to meet interests, knowledge, understanding, abilities and experience of students Student-centered rather than teacher-centered teaching strategies are implicit in the National Teaching Standards Student-centered rather than teacher-centered teaching strategies are implicit in the National Teaching Standards

9. National Teaching Standards SCIENCE AND MATH STANDARDS: Engage in ongoing assessment of teaching and of student learning: Use multiple methods and systematically gather data about student understanding and ability SCIENCE AND MATH STANDARDS: Engage in ongoing assessment of teaching and of student learning: Use multiple methods and systematically gather data about student understanding and ability

10. Improving Teaching 86% Education faculty said they observed colleagues teaching 86% Education faculty said they observed colleagues teaching 67% STEM faculty reported observing colleagues teaching 67% STEM faculty reported observing colleagues teaching Most often, observations were done because of interest in improving one’s own teaching Most often, observations were done because of interest in improving one’s own teaching 49% say their courses influenced changes in other courses 49% say their courses influenced changes in other courses About 50% of the faculty felt that half of their colleagues were engaged in reform About 50% of the faculty felt that half of their colleagues were engaged in reform

11. Comments about Instruction “I believe my dept has crossed an important threshold thanks in part to CETP. Reformed instruction is now the norm and we have reached a critical number of faculty who use reformed instruction. We hire new faculty based in part on their understanding of reforms and their willingness to teach science in a reformed manner.” Dean/Dept Chair “I believe my dept has crossed an important threshold thanks in part to CETP. Reformed instruction is now the norm and we have reached a critical number of faculty who use reformed instruction. We hire new faculty based in part on their understanding of reforms and their willingness to teach science in a reformed manner.” Dean/Dept Chair “There is more disposition on the part of the faculty to use constructivist approaches in their teaching, as well as to integrate information technologies in their courses.” Dean/Dept Chair “There is more disposition on the part of the faculty to use constructivist approaches in their teaching, as well as to integrate information technologies in their courses.” Dean/Dept Chair

12. HE Teaching Activities Most activities were used between Seldom and Occasionally Most activities were used between Seldom and Occasionally Most common activities were real world problems, writing descriptions of reasoning, making models and using technology to communicate Most common activities were real world problems, writing descriptions of reasoning, making models and using technology to communicate Students thought most helpful activities were students working on real-world problems, students using or making models, and using technology to understand concepts taught in class. Students thought most helpful activities were students working on real-world problems, students using or making models, and using technology to understand concepts taught in class.

13. Time Spent and Helpfulness of Standards- Based HE Teaching Practices

14. Survey ItemFrequency Faculty Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Students worked on problems related to real-world or practical issues. 2.693.152.61 Students wrote descriptions of their reasoning. 2.673.022.42 Students used or made models.2.642.952.61 Students made connections to other fields. (STEM and non- STEM) 2.572.852.45 Students evaluated the extent of their own learning. 2.322.882.40

15. Survey ItemFrequency Faculty Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Students performed investigative activities that included data collection and analysis. 2.572.902.44 Students designed and made presentations that helped them learn class concepts. 2.572.642.46 Students participated in whole- class discussions during which the instructor talked less than the students. 2.462.632.31 Students worked with other students where the whole group got one grade. 2.372.432.31

16. Survey ItemFrequency Faculty Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Student assessment results were used to modify what was taught and how. 2.762.782.39 New information was based on what students already knew about the topic. 2.672.912.44 Students had a voice in decisions about course activities. 2.142.742.43

17. Survey ItemFrequency Faculty Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Students completed assessments/assignments that included problems with complex solutions. 2.533.042.31 Students completed assessments/assignments that included multiple choice/short answer items. 2.372.652.29 Students completed assessments/assignments that included portfolios. 2.332.192.28 Students completed assessments/assignments that included full-length papers/reports. 2.282.412.15

18. Survey ItemFrequency Faculty Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Students used technology as a tool to communicate with their instructor. 2.722.472.51 Students used technology to understand or explore in more depth concepts taught in class. 2.592.772.53 Students used technology as a tool in investigations to gather and analyze scientific or mathematical data. 2.482.742.50 Students used technology as a tool for assessment. 2.262.582.40

19. Capsule Description of the Quality of the Lesson (HE) Level: Description of Level CETP N% Level 1: Ineffective Instruction25.3 Level 2: Elements of Effective Instruction37.9 Level 3: Beginning Stages of Effective Instruction1436.8 1. Low535.7 2. Solid321.4 3. High642.9 Level 4: Accomplished, Effective Instruction1334.2 Level 5: Exemplary Instruction615.8

20. Most Important Outcomes (PI) 38% said increased collaboration among higher education institutions 38% said increased collaboration among higher education institutions 31% reported improved interaction between STEM and education departments 31% reported improved interaction between STEM and education departments 31% saw improved interaction between higher education and K-12 schools 31% saw improved interaction between higher education and K-12 schools

21. Faculty Reporting 73% said more interaction with faculty from other institutions 73% said more interaction with faculty from other institutions 72% STEM faculty reported more STEM and education dept interaction 72% STEM faculty reported more STEM and education dept interaction 83% Education faculty found more STEM and education dept interaction 83% Education faculty found more STEM and education dept interaction 46% faculty said they have formalized interaction with K-12 schools 46% faculty said they have formalized interaction with K-12 schools 32% reported field based experiences in K-12 32% reported field based experiences in K-12

22. To what extent has CETP Improved (4=substantial): STEM faculty interactions about instruction, mean=3.9 STEM faculty interactions about instruction, mean=3.9 STEM and Education faculty interactions, mean=3.8 STEM and Education faculty interactions, mean=3.8 2yr and 4 yr college interactions, mean=2.6 2yr and 4 yr college interactions, mean=2.6 Faculty and k-12 interactions, mean=3.7 Faculty and k-12 interactions, mean=3.7

23. Comments about Interaction “The funding has tremendously encouraged collaboration. I will never go back to working alone.” Faculty Member “The funding has tremendously encouraged collaboration. I will never go back to working alone.” Faculty Member “We have much greater interaction with mathematics and science faculty from other institutions. CETP is the cause of it.” Faculty Member “We have much greater interaction with mathematics and science faculty from other institutions. CETP is the cause of it.” Faculty Member “We visit colleagues in similar programs at other institutions and we communicate more, in person and by email, with colleagues in other departments at my home institution. This is a direct consequence of the CETP.” Faculty Member “We visit colleagues in similar programs at other institutions and we communicate more, in person and by email, with colleagues in other departments at my home institution. This is a direct consequence of the CETP.” Faculty Member

24. Deans/Department Chairs 53% reported changes in interaction with faculty in other institutions 53% reported changes in interaction with faculty in other institutions 86% said their faculty had formalized interaction with K-12 86% said their faculty had formalized interaction with K-12 77% said classes in their depts. had field based experiences 77% said classes in their depts. had field based experiences 42% said their depts. offered special programs to increase diversity 42% said their depts. offered special programs to increase diversity

25. Comments about Interaction “The two year colleges were virtually unknown prior to CETP. Now they are a part of the mix when discussions, roundtables, seminars or professional development workshops are organized.” Dean/Dept Chair “The two year colleges were virtually unknown prior to CETP. Now they are a part of the mix when discussions, roundtables, seminars or professional development workshops are organized.” Dean/Dept Chair

26. Percent of Time Spent on Teaching and/or Curriculum Reform Two-YearFour-Year STEMEducationSTEMEducation N%N%N%N% 1 (<10%)13.8 2 (10-19%)13.8 3 (20-29%)22.7 4 (30-39%)56.813.8 5 (40-49%)16.379.513.8 6 (50-59%)68.1415.4 7 (60-69%)16.3150.01114.927.7 8 (70-79%)16.31925.7934.6 9 (80-89%)531.3150.079.5519.2 10 (90-100%)850.01723.027.7 Weighted Mean/SD 9.06/1.398.50/1.117.65/1.967.27/2.07

27. Value of Teaching 84% dept chairs rated teaching as very important in hiring decisions 84% dept chairs rated teaching as very important in hiring decisions 89% dept chairs said promotion and tenure included work on instructional improvement 89% dept chairs said promotion and tenure included work on instructional improvement 34% 4yr STEM faculty found teaching more valued than research 34% 4yr STEM faculty found teaching more valued than research 59% 4 yr Education faculty reported teaching more valued than research 59% 4 yr Education faculty reported teaching more valued than research

28. Comments about K-12 Interaction “At the same time new courses were developed and implemented, teachers from the School District served as ‘critical friends’ in observing college teaching.” Dean/Dept Chair “At the same time new courses were developed and implemented, teachers from the School District served as ‘critical friends’ in observing college teaching.” Dean/Dept Chair “STEM faculty are working between disciplines to garner funds and develop innovative preservice programming. They are creating linkages with K-12 schools as never before.” Dean/Dept Chair “STEM faculty are working between disciplines to garner funds and develop innovative preservice programming. They are creating linkages with K-12 schools as never before.” Dean/Dept Chair

29. Relationships with Higher Education 74% of the principals said their schools have ongoing relationships with institutions of higher education 74% of the principals said their schools have ongoing relationships with institutions of higher education 56% found these relationships have not changed but all but one comment about them was positive 56% found these relationships have not changed but all but one comment about them was positive 47% of teachers reported having relationships with higher education 47% of teachers reported having relationships with higher education CETP teachers reported more relationships CETP teachers reported more relationships

30. Teacher Preparation Programs: Principal View 90% of principals believed teacher preparation programs make a difference 90% of principals believed teacher preparation programs make a difference Components of high quality programs were: Components of high quality programs were: standards-based practices, standards-based practices, strong content knowledge base, strong content knowledge base, understanding of child development and behavior understanding of child development and behavior

31. Barriers to Excellence 62% of principals reported barriers to excellence 62% of principals reported barriers to excellence 58% of teachers reported barriers 58% of teachers reported barriers Lack of resources was the major barrier Lack of resources was the major barrier CETP teachers perceived more barriers than nonCETP—perhaps they have higher expectations CETP teachers perceived more barriers than nonCETP—perhaps they have higher expectations

32. Principal Ratings of Importance of Teaching Strategies

33. N% Students writing lengthy descriptions of their reasoning. 1. Not very important1.8 2. Somewhat important32.4 3. Important3125.2 4. Very important6149.6 5. Extremely important2722.0 Weighted Mean/SD3.93/.76 Investigative activities that include data collection and analysis. 1. Not very important 2. Somewhat important1.8 3. Important1411.4 4. Very important5343.1 5. Extremely important5544.7 Weighted Mean/SD4.32/.71 Whole-class discussion during which the teacher talks less than the students. 1. Not very important 2. Somewhat important32.4 3. Important3427.4 4. Very important5443.6 5. Extremely important3326.6 Weighted Mean/SD3.90/.81

34. N% Students working in groups; receiving one grade per group. 1. Not very important119.0 2. Somewhat important2016.4 3. Important4940.2 4. Very important3125.4 5. Extremely important119.0 Weighted Mean/SD3.08/1.04 Presentation of new information that is based on students' prior knowledge. 1. Not very important 2. Somewhat important21.6 3. Important2016.3 4. Very important5746.3 5. Extremely important4435.8 Weighted Mean/SD4.19/.80 Students gathering information to answer their own questions. 1. Not very important 2. Somewhat important21.6 3. Important1512.2 4. Very important5242.3 5. Extremely important5443.9 Weighted Mean/SD4.26/.87

35. N% Using computers to support deep conceptual understanding. 1. Not very important21.6 2. Somewhat important108.1 3. Important4234.1 4. Very important5242.3 5. Extremely important1713.8 Weighted Mean/SD3.70/.80 Using a variety of assessment techniques: multiple choice tests, portfolios, projects, etc. 1. Not very important 2. Somewhat important 3. Important1512.2 4. Very important5343.1 5. Extremely important5544.7 Weighted Mean/SD4.39/.68 Use of national science or mathematics standards. 1. Not very important21.6 2. Somewhat important32.3 3. Important2922.7 4. Very important4535.2 5. Extremely important4434.4 Weighted Mean/SD4.15/.88

36. CETP vs. NonCETP K-12 Teachers CETP teachers rated their preparation programs higher than nonCETP teachers CETP teachers rated their preparation programs higher than nonCETP teachers CETP teachers were more likely to say they had received instruction on standards CETP teachers were more likely to say they had received instruction on standards CETP teachers attended more professional meetings CETP teachers attended more professional meetings CETP teachers were more likely to view themselves as facilitators than nonCETP CETP teachers were more likely to view themselves as facilitators than nonCETP

37. K-12 Teaching Activities Most used between seldom and occasionally Most used between seldom and occasionally Most common were activities that include data collection and analysis and that the students have enough time Most common were activities that include data collection and analysis and that the students have enough time CETP teachers and students reported using the techniques more than nonCETP CETP teachers and students reported using the techniques more than nonCETP NonCETP students rated techniques as more helpful than CETP students NonCETP students rated techniques as more helpful than CETP students

38. Time Spent and Helpfulness of Standards Based K-12 Teaching Practices

39. Survey ItemFrequency Teacher Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Students worked on problems related to real-world or practical issues. 3.292.751.89* - Students determined how much they knew about something. 3.182.971.94* - Students made connections to other fields. (STEM and non- STEM) 3.032.361.42* - Students used or made models, e.g., physical, conceptual or mathematical. 2.98*2.43*1.78* - Students wrote about why they thought something. 2.76*2.45*1.52* -

40. Survey ItemFrequency Teacher Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Students did activities that included data collection and analysis. 2.97*3.01*2.06 Students participated in whole- class discussions during which the teacher talked less than the students. 2.61*2.341.39 Students designed and made presentations that helped them learn class concepts. 2.53*2.40*1.63* - Students worked with other students where the whole group got one grade. 2.41*2.501.75

41. Survey ItemFrequency Teacher Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Student assessment results were used to modify what was taught and how. 3.61 Not a student item Students had enough time to learn what was required. 3.443.192.15* - New information was based on what students already knew about the topic. 3.362.852.01* - Students had a voice in decisions about class activities. 2.832.53*1.70

42. Survey ItemFrequency Teacher Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Students completed assessments/assignments that included multiple choice/short answer items. 3.222.96*2.06* - Students completed assessments/assignments that included problems with complex solutions. 3.13* - 2.971.89* - Students completed assessments/assignments that included full-length papers/reports. 2.21*2.33*1.38* - Students completed assessments/assignments that included portfolios. 1.92*1.99*1.06* -

43. Survey ItemFrequency Teacher Weighted Mean (1-4) Frequency Student Weighted Mean (1-4) Helpfulness Student Weighted Mean (0-3) Students used technology to understand or explore in more depth concepts already taught in class. 2.842.651.97* - Students used technology as a tool for checking understanding (assessment). 2.75* - 2.60*1.85* - Students used technology as a tool in investigations to gather and organize information. 2.57*2.72*1.97 Students used technology as a tool to communicate with their instructor or other students. 2.15*2.001.32* -

44. Capsule Description of the Quality of the Lesson (K-12) Level: Description of Level CETPNonCETP N%N% Level 1: Ineffective Instruction1013.3611.8 Level 2: Elements of Effective Instruction1317.31223.5 Level 3: Beginning Stages of Effective Instruction 2634.72039.2 1. Low1029.4939.1 2. Solid1132.4834.8 3. High1338.2626.1 Level 4: Accomplished, Effective Instruction1418.71121.6 Level 5: Exemplary Instruction1216.023.9

45. How important would the data sources rate basing new information on what students already know about a topic?

46. New information is based on what students already know about a topic: The majority of Principals rated this as Very Important The majority of Principals rated this as Very Important The majority of K-12 Teachers said it happens Regularly The majority of K-12 Teachers said it happens Regularly The majority of 6-12 Students said it happens Occasionally The majority of 6-12 Students said it happens Occasionally The majority of Faculty say it happens Regularly The majority of Faculty say it happens Regularly The majority of College Students say it happens Occasionally The majority of College Students say it happens Occasionally

47. Multiple Assessments Principals think the use of multiple assessments (such as portfolios and multiple choice items) is Very Important Principals think the use of multiple assessments (such as portfolios and multiple choice items) is Very Important However, in regards to portfolios: However, in regards to portfolios: Faculty – Seldom Faculty – Seldom College Students – Never College Students – Never K-12 Teachers – Never K-12 Teachers – Never 6-12 Students – Never 6-12 Students – Never

48. Understanding science/math requires special skills that only some possess. (Scaled: 1-4, 1-->Strongly Disagree) Higher Ed. (Weighted Mean/SD) Four-Year STEMEducation Faculty on Science 1.51/.641.25/.49 Faculty on Math 1.66/.671.26/.49 Students on Science 2.39/.832.00/.79 Students on Math 2.44/.852.03/.82 K-12 (Weighted Mean/SD) CETPNonCETP K-12 Teachers on Science* 1.71/.641.90/.61 K-12 Teachers on Math* 1.78/.611.97/.62 6-12 Students on Science 2.36/.902.29/.91 6-12 Students on Math 2.51/.932.43/.95

49. What Students Thought About Science and Mathematics CETPNonCETP N%N% Mathematics is boring. 1. Disagree78860.337861.5 2. Agree51939.723738.5 Mathematics is important to everyone's life. 1. Disagree20515.510216.7 2. Agree111584.551083.3 Science is boring.* 1. Disagree92070.739665.0 2. Agree38229.321335.0 Science is important to everyone's life. 1. Disagree29422.414724.1 2. Agree102177.646275.9

50. How accurate were your initial predications? Go back to the note cards you made earlier. Go back to the note cards you made earlier. How accurate were your initial predications? How accurate were your initial predications?

51. Scholars 85 % felt they benefited from the scholarship 85 % felt they benefited from the scholarship 45% felt the scholarship was not influential in their decision (they wanted to be teachers even without the scholarship) 45% felt the scholarship was not influential in their decision (they wanted to be teachers even without the scholarship) Scholars report they had good opportunities and had positive impact on their institutions Scholars report they had good opportunities and had positive impact on their institutions 27% of the PIs report funding to continue the scholarships. 27% of the PIs report funding to continue the scholarships.

52. Comments about Scholars “A student congress of continuing and graduated students was formed by a CETP scholar. This Congress offered the opportunity for experienced teachers, novice teachers, and undergraduates to exchange ideas and to discuss issues related to teaching and the profession.” Dean/Department Chair “A student congress of continuing and graduated students was formed by a CETP scholar. This Congress offered the opportunity for experienced teachers, novice teachers, and undergraduates to exchange ideas and to discuss issues related to teaching and the profession.” Dean/Department Chair

53. Comments from Scholars “I am proud to be a member of this wonderful community of dedicated professionals. I love knowing that I have such a strong system of support behind me.” “I am proud to be a member of this wonderful community of dedicated professionals. I love knowing that I have such a strong system of support behind me.” “Very, very positive – I know I have a life long connection to a professor who is willing to help me any way he can when I begin my teaching career.” “Very, very positive – I know I have a life long connection to a professor who is willing to help me any way he can when I begin my teaching career.” “As a refugee with low income, I couldn’t at all afford the tuition for (recertification)…But this scholarship gave me back hope.” “As a refugee with low income, I couldn’t at all afford the tuition for (recertification)…But this scholarship gave me back hope.”

54. Concerns/Questions about Core Data Collection What concerns or questions do you have about collecting Core data? What concerns or questions do you have about collecting Core data? Record these on a note card and we will pick them up. Record these on a note card and we will pick them up.