Building Assessment Skills and Competencies

Science Education Assessment (SEA) Scholars: A local, authentic approach to teaching graduate students about assessment Katie Kearns and Lisa Wiltbank Representing Indiana University Bloomington

Assessment of information literacy skills Undergraduate achievement Graduate student development of assessment skills Faculty culture around course and curriculum improvement KDK

Authentic context: Assessing undergraduate information literacy skills SEA Scholars (grad-faculty partnerships) College-level assessment for accreditation Biology undergraduate major program goals Information literacy among biology majors KDK

Graduate student learning outcomes: Authentic methods - performance of assessment Wrote learning outcomes for information literacy across the curriculum Program Developed meaningful, assessable assignments Wrote rubrics Collected, analyzed, interpreted evidence of student learning from courses Course Program Synthesized assessments data Reported on assessments at curriculum level Shared findings and recommendations with program faculty LBW

Graduate student learning outcomes: Additional assessment methods Pre- and post-program assessment 15 words you associate with “assessment” For what reasons might an instructor find assessment of student learning valuable? What methods might instructors use to assess student learning? Regular reflection and feedback on SEA Scholar meetings KDK

Graduate student learning outcomes: Cognitive assessments – knowing about assessment First list was synonyms Second list of nouns was longer, course-embedded, and multi-modal: learning outcomes, quantitative, qualitative, essay, exam, quiz What nouns do you associate with assessment? First list was outcome-oriented: effectiveness, evidence, failure, success Second list was process-oriented: improvement, feedback, progress, accountability What outcomes do you associate with assessment? KDK

Reflections on learning Graduate student learning outcomes: Open-ended assessment – reflecting on identity “We have been exposed to the biology undergraduate program in a broader and deeper way than we could otherwise.” “Graduate students must think about improving student learning both on the course level and at the larger scale of the entire curriculum, which is a way of thinking that may be entirely new to the graduate students.” Reflections on learning Course AND curriculum view of assessment Role negotiation Developing an expert identity with faculty Learning how to ask for things LBW

Graduate student learning outcomes: Beyond the formal assessments – authentic products Dissertation chapters about course and curriculum assessment projects Book chapter Manuscript Career choices LBW

Lessons and recommendations for teaching future faculty about assessment Instruction process Engage in authentic work Encourage interprofessional collaborations – faculty, graduate students, staff Ongoing “tasting the soup” Assessment methods Ask open-ended questions Look long-range for outcomes Provide opportunities for multiple audiences for reflection Include those being assessed in what they want to be assessed on The flip side: the authentic assessment project ended, as did the SEA Scholar partnerships. KDK

Building Assessment Skills and Competencies – Perspectives from Cornell University Colleen McLinn Director, CIRTL at Cornell Cornell University Graduate School & Derina Samuel Associate Director for Graduate Student Teaching Development Center for Teaching Innovation Speakers represent the two main collaborating units, but also worked with the Knight Institute for Writing in the Disciplines, the Office of Undergraduate Biology, the Department of Physics, and others. Dean Barbara Knuth of the Graduate School was the PI for this project.

Key Activities PREPARE Workshops (17) Visiting Speakers CIRTL Webinars University-Wide Conversations First-Year Writing Seminar Assessment Fellows (16) Biology and Physics TAs Disciplinary Skills Development 17 future faculty earned Prepare certificates 16 FWS assessment fellows conducted TAR projects University-wide and beyond: Create new opportunities for grad students and postdocs to learn about defining / assessing undergraduate learning outcomes PREPARE workshop series and certificates Visiting speakers and symposia CIRTL Network webinar series Disciplinary: Develop assessment and action research skills First-Year Writing Seminar Assessment Fellows designed, conducted, presented Teaching as Research projects Customized Biology and Physics TA workshops Flipping the classroom Clickers, concept inventories, etc. Project website - http://blogs.cornell.edu/prepare

Evidence of Effectiveness Workshop session evaluations Artifacts from fellows’ Teaching as Research projects (posters, papers) Pre-/post-surveys of fellows Topics: How effective was your university? Were they prepared? At the beginning, did focus groups for needs assessment

Specific Competencies and Themes Identity and interactions Content and opportunities Varied sources, including qualitative Use findings to improve In-and out-of- class time Active and student-centered Goals for students Skills needed in the discipline Think big Use time Consider diversity Collect evidence 1. Thinking big about goals for students, and skills needed in the discipline 2. Thoughtful use of in- and out-of-class time 3. Active and student-centered pedagogies 4. Inclusive Teaching: Attending to learner identity and interactions and instructor identity. Varied sources of content and opportunities to demonstrate learning. 5. Identifying possible sources of evidence for student learning, including qualitative measures 6. Using your findings to improve teaching

Successes Student perspectives Framing assessment as integral to future faculty preparation Necessary skills for designing courses Especially beneficial for humanities students Creating learning communities focused on assessment Benefits of Teaching as Research projects Future faculty as project coordinators Frame assessment as integral to future faculty preparation: Workshops Consultations on writing competitive job applications Student perspectives Explicit focus on learning outcomes and assessment is logical step in designing courses Seek to assess own teaching effectiveness Surveys, rubrics, formative assessment Humanities students more directly involved with course design, already oriented to careers in the professoriate Cohort-based programming and discipline-specific examples more coherent Benefits of students working in groups to conduct Teaching as Research (TAR) projects Cost-effective, shorter-term Raised disciplinary faculty awareness of and interest in TAR Faculty partner engagement in TAR Identify candidate students Generate research questions 4 – both success and challenge Targeted outreach Develop and facilitate workshops Coordinate certificate programs (STEM, Humanities) Near-peer mentoring Realistic examples of assignments and questions

Challenges Future faculty coordinators time- constrained STEM TAs early in graduate school, lots to learn STEM faculty not initially eager to involve TA viewpoints in course overhaul discussions Insufficient resources to develop humanities certificate

In Hindsight, and for the Future… More direct observations and longitudinal studies of future faculty Develop a day-long or multi-day institute in course design and student learning assessment More discussions bringing together faculty and teaching assistants

Thank you! Questions?

Building Assessment Skills and Competencies Think/Pair/Share In groups of two 2 minutes: Think about what is the most important skill or mindset that future faculty need for assessing the learning of undergraduates? Share your response with your partner

Building Assessment Skills and Competencies Think/Pair/Share In groups of two 2 minutes: Think about what was the most successful strategy you used for helping graduate students assess student learning? Share your response with your partner