1. SEI Case Brief: E-Portfolios at Cornell university
Clare van den Blink, Cornell University
Joan Getman, University of Southern California
Thursday April 14

2. Learning Objectives
Understand how to develop a pilot and project design plan.
Create an assessment strategy to inform technology and implementation decisions incorporating both quantitative and qualitative measures.
Develop technology assessment criteria tied to project assumptions and strategies.
JOAN:
One way to capture your thinking and take-aways from this session: respond or take note of the questions that need to be answered in designing a pilot and evaluation plan.
These are the learning objectives we will try to meet in describing our pilot which spanned 3 semesters at Cornell.
We involved ?? Faculty and ?? Students in these classes.

3. What’s driving your interest in ePortfolios?
Mandate from institutional leaders to recommend an ePortfolio solution?
Faculty inquiries? Faculty practices?
Student interest?
Institutional accreditation/assessment?
Due diligence/Scanning the environment?
JOAN
POLL HERE.
Why are you interested?
Who is sponsoring the pilot?
What is driving your project will inform motivation for faculty and students to participate
Drivers, expectations and sponsors will influence the design and articulate the goals of your pilot.
If you are feeling like the ePortfolios will be searching for an audience…turn stakeholder/sponsor attention to questions of value and provide information that establishes portfolio practice as distinct from portfolio technologies.

4. Questions Who is sponsoring your pilot?
To whom are you making a recommendation?
Are there multiple decision-makers?
Co-sponsored with the Center for Teaching Excellence; Not driven by IT.
Interest corresponded in time with accreditation prep.

5. [Value?  Key technology features?  Service feasibility?]
from Project to Pilot
Different phases | different goals | different evaluation plans
Faculty Project: Evaluation led to the first pilot
Phase I: Early explorations | Small pilot
Goal: Assess value and critical features (Sakai OSP)
Phase II: Campus pilot
Goal: Identify central tool for campus (Chalk & Wire)
Value to teaching & learning
Campus readiness/Need
Technical review
Focus: Viability of scaling and centrally supporting an ePortfolio service.
[Value?  Key technology features?  Service feasibility?]
CLARE
Refer to the project plans and FIT proposals
B.) Create an assessment strategy to inform technology & implementation decisions incorporating both quantitative and qualitative measures.
Examples will be shared of how the pilot’s varied evaluation approaches and how the results informed institutional decisions about adopting e-Portfolio technology. (or not)

6. Pilot Evaluation Strategy
What are the KEY questions?
What are you trying to evaluate?
Define research questions from pilot goals
Literature search |Landscape view
Focus the evaluation questions
Determine data collection methods
Analysis - Reporting
JOAN
Be clear about what you are trying to find out.
Background: lit search–
Different key research questions in our different pilot phases.
Case study: value proposition/faculty and student interest - faculty mostly—
Pre-Pilot: identifying key functionality and ongoing assessment of value—
Pilot: campus readiness and need - viability, scalability and tech requirements - service dimensions

7. Context
“New  ways  of  operating   economically,  socially,  and  intellectually  necessarily  shift  focuses  in  education.  In  this  fast-­‐changing   world  we  must  educate  students  to  know  how  to  learn,  how  to  turn  the  information  that  is  now  so   accessible  and  ubiquitous  into  knowledge,  and  how  to  document  and  analyze  their  own  learning.”
Barbara Cambridge
“Learning,  Knowing,  and  Reflecting:  
Literacies  for  the  21st  Century”
JOAN

8. Literature and Landscape
Scenarios for ePortfolio use at different levels
Course | Program |Institution
Portfolio philosophy: Who owns and controls access to content in a portfolio?
Students | Instructors and Program Administrators
Technologies: Comparison of Solutions
Static to Dynamic | Features for assessment and reporting
JOAN
Start by taking a look at the external and internal landscape regarding assessment.
“Selection  of  the  appropriate  electronic   portfolio  solution  and  effectiveness  of  the  implementation  depends  in  large  part  on  clarity   about  the  reasons  for  introducing  electronic  portfolios.”

9. Your Campus Landscape
Are portfolios being used on your campus today? If so, can you describe how faculty and students use e/Portfolios?
To the group:
Does your community speak in terms of assessment, outcomes and rubrics?

10. Evaluation Plan Articulate goals and research questions…
This ePortfolio pilot will evaluate the extent to which….
ePortfolios can support faculty and student needs
(e.g. with respect to assessment and representing individual student learning);
ePortfolios can represent learning outcomes at the program, departmental or school level;
Technology meets the needs of faculty and students; and
Technology has a relatively “easy” learning curve.
CLARE

11. Pilot Participants Who is already interested?
Who to invite based on current practices and discipline?
Art, service learning and writing programs
Representative of campus demographics
Have you engaged all the key stakeholders and pilot partners?
CLARE
How participants were engaged in different phases (FIT grant/use cases/pilot phase I/pilot phase II)
Who is going to help implement the pilot?

12. Choosing a Technology Criteria Teaching & Learning Features?
Usability: how ‘easy’ is it to use the tool?
Faculty & student perspectives
How well can the tool be supported?
Pilot Technical Approach
Vendor hosted solution – to focus on the features & value not the technology backend
CLARE
See details in report…..
Focus on Phase 2 of the pilot.
Hosted Solution in phase 1 and phase 2.

13. PILOT technology REVIEW: The short list
Chalk and Wire
TaskStream and Learning Achievement tools
Pebble Pad
Waypoint Outcomes
Digital Measures:Activity Insight
Epsilen ePortfolio
Innovative Learning Assessment Technologies Pass-Port CE
LiveText
Nuventie:iWebfolio
TK20: Higher Ed
Mahara
Blogs
CLARE

14. Pilot Start-Up | Support for Piloteers
Faculty: Training & documentation
Students: In-class orientations
Staff: Hands-on time
Record data about staff time
& need for faculty/student support.
Data informs the campus service decisions.
CLARE
In  August  2009,  pilot  participants  attended  an  orientation  at  which  Center  for  Teaching   Excellence  staff  did  presented  on  “Reflection  and  Portfolio  Based  Learning.”    Faculty   followed  up  by  identifying  evidence  of  learning  in  their  courses  and  watching  a   demonstration  of  Sakai  OSP  by  CIT  staff.  

15. EVALUATION: Data Collection
Phase 1: Small group exploration
Case study approach.
Faculty interviews, pre and post-pilot surveys /Student surveys
Phase 2: Campus Pilot
Mixed-mode approach: qualitative & quantitative.
Faculty pre-pilot survey’s and focus groups
Student surveys
Phone check-ins with faculty during pilot
Interviews with service providers; review support time reports
Ongoing feedback to vendor; vendor review
Technology assessment checklists-detailed criteria.
JOAN
August 2009: A  pre-­‐pilot  survey  revealed  that  faculty  had  the   following  goals  for  using  ePortfolios:
Baseline questions about usage; focus group methods that structured the conversation;
Structuring criteria for technology selection - how C&W was selected

16. Question
What is the primary reason for wanting students to have portfolios?
Create a collection of ‘learning artifacts’
Assess learning
Showcase or highlight specific work
None of the above
All of the above
JOAN
POLL question.

17. Repository Assessment Showcase
Where’s the focus?
Repository Assessment Showcase
JOAN
Finding: tools put the emphasis in different places.
Faculty also put their emphasis in different places.

18. Challenges New vocabulary: artifacts
New pedagogical practices: reflection
New assessment approach: learning outcomes and rubrics
So many tools!
ePortfolios = another place where students upload assignments.
JOAN
If you are going to have an ePortfolio pilot?

19. Student centered v. course centered
Challenges
New frame of reference:
Student centered v. course centered
Value of ePortfolios emerges over time and across courses…
JOAN
Also switching from Sakai to Chalk and Wire for Phase 2: Different ways of thinking and framing rubrics

20. [Value?  Key technology features?  Service feasibility?]
Pilot Outcomes
Has the data enabled us to answer the key questions?
YES!
Did we make a recommendation?
JOAN
[Value?  Key technology features?  Service feasibility?]

21. EVALUATION AREAS & Analysis
Technology review
Vendor relationship
Business model
Student perspective
Faculty perspective
Support feedback
Institutional readiness
CLARE: took all these factors into accountbefore making our recommendation.

22. Final Report
“Pilot  outcomes  indicate  that  while  Sakai’s  Open  Source  Portfolio  (OSP)  and  Chalk  and  Wire,  have   beneficial  features,  neither  of  these  ePortfolio  solutions  emerged  as  best  able  to  support   teaching,  learning  and  assessment  at  Cornell.”
CALRE
Excerpt from executive summary

23. [Value?  Key technology features?  Service feasibility?]
Final Recommendation
“The pilot team’s recommendation is for Cornell not to invest in a single, enterprise solution for ePortfolios at this time. This recommendation is based on the widely varying needs of the pilot participants, cost and licensing factors, high support requirements, and campus readiness.”
CLARE
Recommendation not to invest
[Value?  Key technology features?  Service feasibility?]

24. Benefits of the pilot
“The  pilot  successfully   introduced  ePortfolios,  and  at  this  point,  the  Cornell  community  may  benefit  from  further   discussion  about  portfolio  practices,  the  benefits  of  adopting  portfolios  at  different  levels   (i.e.  course,  program  or  department)  and  institutional  goals  for  using  portfolios.”  
CLARE
Positive steps we recommend to keep the dialogue going since there was perceived value in the pilots.

25. Next phase: Supporting Portfolio Practices
Faculty Support Services at Cornell will:
Consult on portfolio practices and available technologies.
Explore technologies with individual faculty as time allows.
Monitor community interest.
Review emerging ePortfolio solutions.
Provide references and support ongoing dialogue about assessment and ePortfolio practices.
CLARE

26. Telling the Story
Communicate recommendations and report outcomes to campus
Written report
Online resources
Faculty events
Transition to end pilot
Export of faculty and student content
Support pilot participants in transition to other tools, i.e. blog service
Ongoing support for e-Portfolio’s as a consultative service
JOAN
We included what we learned to reach our recommendation
We described benefits of the pilot – success was being able to make an informed recommendation.
Transition is often very difficult…Pilots that won’t end.
Faculty who invested resent it being closed.

27. Was the pilot a success?
YES! Success: Being able to make an informed recommendation based on data and starting a campus conversation about learning outcomes and assessment strategies.
JOAN

28. Clare van den Blink, cv36@cornell.edu Joan Getman, jgetman@usc.edu
Questions?
Clare van den Blink,
Joan Getman,