1. Advancing by the Numbers
Jim Farmer
As presented at the Portal2008 “Measurement & Assessment” Conference
4 June 2008 | Gettysburg College, Gettysburg PA USA

2. Assessment defined
“Assessment is the process of documenting, usually in measurable terms, knowledge, skills, attitudes and beliefs. “

3. Types of assessment
In an educational setting, summative assessments are typically used to assign students a course grade.
Assessment of learning
Formative assessment, also referred to as educative assessment, is used to aid learning.
Assessment for learning
Wikipedia

4. Assessment with feedback
Process
improvement
Execution
“Continuous improvement”

5. Applying assessment
“Engineer” learning based on feedback from the use of eLearning systems.
Assess prescribe deliver

6. Assessment as a management style
Open availability of data
Open discussion of the alternatives
“Explaining” the solution
Within the organization
Outside the organization
To the public
Expect fair and unfair criticism!

7. The Commission wrote
“We recommend that America’s colleges and universities embrace a culture of continuous innovation and quality improvement. We urge these institutions to develop new pedagogies, curricula and technologies to improve learning, particularly in the areas of science and mathematics.”
Spelling Commission, U.S. Department of Education

8. Two perspectives of eLearning
Graduate programs and undergraduate tutorials and independent study tend to:
Focus on collaboration among students and faculty
Use resources typical of a cross-institutional research effort
Undergraduate programs, especially first and second year tend to:
Focus on online tutorials, “drill and practice,” and assessments for learning
Use faculty-recommended resources and Internet services

9. Some US/UK differences
Thomas Weko, Higher Education Policy Institute, Oxford, March 2004

10. Unprepared (reading)
43.9% did not have the “ability to make relatively simple inferences beyond the author ’s main thought and/or understand and evaluate relatively abstract concepts.”
And additional 39.5% did not have “the ability to make complex inferences or evaluative judgments that require piecing together multiple sources of information from the passage (Rock, Pollack and Quinn 1995).

11. Unprepared (mathematics)
29.7% could not solve simple operations with decimals, fractions, powers, roots or solve simple problems requiring the understanding of low-level mathematical concepts. Many could not do decimal arithmetic.
An additional 40% could not understand intermediate-level mathematical concepts and could not formulate multistep solutions to word problems.

12. Use of supplementary material

13. Staffing instructional technology

14. Assessments Assessing “tools” Assessing students
Capabilities [Preparation, learning style]
Learning [Added knowledge]
Assessing teaching and learning
Teaching methods
Learning opportunities

15. Success in math courses

16. Assessing remedial education
The results suggest that students in remediation are more likely to persist in college in comparison to students with similar test scores and backgrounds who were not required to take the courses. They are also more likely to transfer to a higher-level college and to complete a bachelor's degree. Bettinger 2005

17. Assessing faculty
Whether an instructor teaches full-time or part-time, does research, has tenure, or is highly paid has no influence on a college student's grade [0.5%], likelihood of dropping a course [1.3%] or taking more subsequent courses in the same subject [4.0%]. Hoffman 2006

18. Assessing faculty
We have shown that student evaluations [of faculty] differ from the ideal construct, because they do not appear to reflect learning, but are sensitive to grading leniency. Weinberg 2007

19. A portal is the toolbox of the knowledge worker

21. Suppes at Stanford University
First Year Russian 1972

22. Suppes at Stanford University
Learning Trajectories

23. Learning trajectories
Content Mastery
Course Grade
Time
End of Scheduled Term
ABCDF
Add “supplementary materials” to the red line
Add “more time” to the purple line
Supplementary mastery vs boredom
Additional time for those who need it.
Based on the work of Pat Suppes at Stanford University

24. Assessing portals
At one university 85% of the students personalized their portal.
At most colleges and universities less than 10% personal their portal.
uPortal 2005
Why?

25. Assessing portals “Time on task” Focus Purpose
Learning style, Warning to faculty and advisors
Suppes
Focus
Portlet/channel use
AICC 2001
Purpose
Portlet/channel use by type
University of Delaware 2004

26. Assessing VRE portals
Data [from the archeology site] is available in 7 weeks instead of one year.
Data from several projects is consolidated and has a single presentation increasing the quality of research.
Others use the data and contribute.
Mark Baker 2008 [unpublished]

27. Impact on classroom activities

28. The student’s perspective
“The explosion of knowledge”
Collateralized Debt Obligation
Structured Investment Vehicle

29. “Portals are a journey of increasing functionality for expanding communities."

30. The end jxf@immagic.com jxf@Georgetown.edu

31. What about Web 2.0? What is the educational purpose?
Is the college willing to take responsibility for content?
Is the content to be shared?
Who is responsible for content?
Who “owns” the content?