SEI Case Brief: E-Portfolios at Cornell university Clare van den Blink, Cornell University Joan Getman, University of Southern California Thursday April 14

Learning Objectives Understand how to develop a pilot and project design plan. Create an assessment strategy to inform technology and implementation decisions incorporating both quantitative and qualitative measures. Develop technology assessment criteria tied to project assumptions and strategies. JOAN: One way to capture your thinking and take-aways from this session: respond or take note of the questions that need to be answered in designing a pilot and evaluation plan. These are the learning objectives we will try to meet in describing our pilot which spanned 3 semesters at Cornell. We involved ?? Faculty and ?? Students in these classes.

What’s driving your interest in ePortfolios? Mandate from institutional leaders to recommend an ePortfolio solution? Faculty inquiries? Faculty practices? Student interest? Institutional accreditation/assessment? Due diligence/Scanning the environment? JOAN POLL HERE. Why are you interested? Who is sponsoring the pilot? What is driving your project will inform motivation for faculty and students to participate Drivers, expectations and sponsors will influence the design and articulate the goals of your pilot. If you are feeling like the ePortfolios will be searching for an audience…turn stakeholder/sponsor attention to questions of value and provide information that establishes portfolio practice as distinct from portfolio technologies.

Questions Who is sponsoring your pilot? To whom are you making a recommendation? Are there multiple decision-makers? Co-sponsored with the Center for Teaching Excellence; Not driven by IT. Interest corresponded in time with accreditation prep.

[Value?  Key technology features?  Service feasibility?] from Project to Pilot Different phases | different goals | different evaluation plans Faculty Project: Evaluation led to the first pilot Phase I: Early explorations | Small pilot Goal: Assess value and critical features (Sakai OSP) Phase II: Campus pilot Goal: Identify central tool for campus (Chalk & Wire) Value to teaching & learning Campus readiness/Need Technical review Focus: Viability of scaling and centrally supporting an ePortfolio service. [Value?  Key technology features?  Service feasibility?] CLARE Refer to the project plans and FIT proposals B.) Create an assessment strategy to inform technology & implementation decisions incorporating both quantitative and qualitative measures. Examples will be shared of how the pilot’s varied evaluation approaches and how the results informed institutional decisions about adopting e-Portfolio technology. (or not)

Pilot Evaluation Strategy What are the KEY questions? What are you trying to evaluate? Define research questions from pilot goals Literature search |Landscape view Focus the evaluation questions Determine data collection methods Analysis - Reporting JOAN Be clear about what you are trying to find out. Background: lit search– Different key research questions in our different pilot phases. Case study: value proposition/faculty and student interest - faculty mostly— Pre-Pilot: identifying key functionality and ongoing assessment of value— Pilot: campus readiness and need - viability, scalability and tech requirements - service dimensions

Context “New  ways  of  operating   economically,  socially,  and  intellectually  necessarily  shift  focuses  in  education.  In  this  fast-­‐changing   world  we  must  educate  students  to  know  how  to  learn,  how  to  turn  the  information  that  is  now  so   accessible  and  ubiquitous  into  knowledge,  and  how  to  document  and  analyze  their  own  learning.” Barbara Cambridge “Learning,  Knowing,  and  Reflecting:   Literacies  for  the  21st  Century” JOAN

Literature and Landscape Scenarios for ePortfolio use at different levels Course | Program |Institution Portfolio philosophy: Who owns and controls access to content in a portfolio? Students | Instructors and Program Administrators Technologies: Comparison of Solutions Static to Dynamic | Features for assessment and reporting JOAN Start by taking a look at the external and internal landscape regarding assessment. “Selection  of  the  appropriate  electronic   portfolio  solution  and  effectiveness  of  the  implementation  depends  in  large  part  on  clarity   about  the  reasons  for  introducing  electronic  portfolios.”

Your Campus Landscape Are portfolios being used on your campus today? If so, can you describe how faculty and students use e/Portfolios? To the group: Does your community speak in terms of assessment, outcomes and rubrics?

Evaluation Plan Articulate goals and research questions… This ePortfolio pilot will evaluate the extent to which…. ePortfolios can support faculty and student needs (e.g. with respect to assessment and representing individual student learning); ePortfolios can represent learning outcomes at the program, departmental or school level; Technology meets the needs of faculty and students; and Technology has a relatively “easy” learning curve. CLARE

Pilot Participants Who is already interested? Who to invite based on current practices and discipline? Art, service learning and writing programs Representative of campus demographics Have you engaged all the key stakeholders and pilot partners? CLARE How participants were engaged in different phases (FIT grant/use cases/pilot phase I/pilot phase II) Who is going to help implement the pilot?

Choosing a Technology Criteria Teaching & Learning Features? Usability: how ‘easy’ is it to use the tool? Faculty & student perspectives How well can the tool be supported? Pilot Technical Approach Vendor hosted solution – to focus on the features & value not the technology backend CLARE See details in report….. Focus on Phase 2 of the pilot. Hosted Solution in phase 1 and phase 2.

PILOT technology REVIEW: The short list Chalk and Wire TaskStream and Learning Achievement tools Pebble Pad Waypoint Outcomes Digital Measures:Activity Insight Epsilen ePortfolio Innovative Learning Assessment Technologies Pass-Port CE LiveText Nuventie:iWebfolio TK20: Higher Ed Mahara Blogs CLARE

Pilot Start-Up | Support for Piloteers Faculty: Training & documentation Students: In-class orientations Staff: Hands-on time Record data about staff time & need for faculty/student support. Data informs the campus service decisions. CLARE In  August  2009,  pilot  participants  attended  an  orientation  at  which  Center  for  Teaching   Excellence  staff  did  presented  on  “Reflection  and  Portfolio  Based  Learning.”    Faculty   followed  up  by  identifying  evidence  of  learning  in  their  courses  and  watching  a   demonstration  of  Sakai  OSP  by  CIT  staff.  

EVALUATION: Data Collection Phase 1: Small group exploration Case study approach. Faculty interviews, pre and post-pilot surveys /Student surveys Phase 2: Campus Pilot Mixed-mode approach: qualitative & quantitative. Faculty pre-pilot survey’s and focus groups Student surveys Phone check-ins with faculty during pilot Interviews with service providers; review support time reports Ongoing feedback to vendor; vendor review Technology assessment checklists-detailed criteria. JOAN August 2009: A  pre-­‐pilot  survey  revealed  that  faculty  had  the   following  goals  for  using  ePortfolios: Baseline questions about usage; focus group methods that structured the conversation; Structuring criteria for technology selection - how C&W was selected

Question What is the primary reason for wanting students to have portfolios? Create a collection of ‘learning artifacts’ Assess learning Showcase or highlight specific work None of the above All of the above JOAN POLL question.

Repository Assessment Showcase Where’s the focus? Repository Assessment Showcase JOAN Finding: tools put the emphasis in different places. Faculty also put their emphasis in different places.

Challenges New vocabulary: artifacts New pedagogical practices: reflection New assessment approach: learning outcomes and rubrics So many tools! ePortfolios = another place where students upload assignments. JOAN If you are going to have an ePortfolio pilot?

Student centered v. course centered Challenges New frame of reference: Student centered v. course centered Value of ePortfolios emerges over time and across courses… JOAN Also switching from Sakai to Chalk and Wire for Phase 2: Different ways of thinking and framing rubrics

[Value?  Key technology features?  Service feasibility?] Pilot Outcomes Has the data enabled us to answer the key questions? YES! Did we make a recommendation? JOAN [Value?  Key technology features?  Service feasibility?]

EVALUATION AREAS & Analysis Technology review Vendor relationship Business model Student perspective Faculty perspective Support feedback Institutional readiness CLARE: took all these factors into accountbefore making our recommendation.

Final Report “Pilot  outcomes  indicate  that  while  Sakai’s  Open  Source  Portfolio  (OSP)  and  Chalk  and  Wire,  have   beneficial  features,  neither  of  these  ePortfolio  solutions  emerged  as  best  able  to  support   teaching,  learning  and  assessment  at  Cornell.” CALRE Excerpt from executive summary

[Value?  Key technology features?  Service feasibility?] Final Recommendation “The pilot team’s recommendation is for Cornell not to invest in a single, enterprise solution for ePortfolios at this time. This recommendation is based on the widely varying needs of the pilot participants, cost and licensing factors, high support requirements, and campus readiness.” CLARE Recommendation not to invest [Value?  Key technology features?  Service feasibility?]

Benefits of the pilot “The  pilot  successfully   introduced  ePortfolios,  and  at  this  point,  the  Cornell  community  may  benefit  from  further   discussion  about  portfolio  practices,  the  benefits  of  adopting  portfolios  at  different  levels   (i.e.  course,  program  or  department)  and  institutional  goals  for  using  portfolios.”   CLARE Positive steps we recommend to keep the dialogue going since there was perceived value in the pilots.

Next phase: Supporting Portfolio Practices Faculty Support Services at Cornell will: Consult on portfolio practices and available technologies. Explore technologies with individual faculty as time allows. Monitor community interest. Review emerging ePortfolio solutions. Provide references and support ongoing dialogue about assessment and ePortfolio practices. CLARE

Telling the Story Communicate recommendations and report outcomes to campus Written report Online resources Faculty events Transition to end pilot Export of faculty and student content Support pilot participants in transition to other tools, i.e. blog service Ongoing support for e-Portfolio’s as a consultative service JOAN We included what we learned to reach our recommendation We described benefits of the pilot – success was being able to make an informed recommendation. Transition is often very difficult…Pilots that won’t end. Faculty who invested resent it being closed.

Was the pilot a success? YES! Success: Being able to make an informed recommendation based on data and starting a campus conversation about learning outcomes and assessment strategies. JOAN

Clare van den Blink, cv36@cornell.edu Joan Getman, jgetman@usc.edu Questions? Clare van den Blink, cv36@cornell.edu Joan Getman, jgetman@usc.edu