Advancing by the Numbers Jim Farmer As presented at the Portal2008 “Measurement & Assessment” Conference 4 June 2008 | Gettysburg College, Gettysburg PA USA

Assessment defined “Assessment is the process of documenting, usually in measurable terms, knowledge, skills, attitudes and beliefs. “

Types of assessment In an educational setting, summative assessments are typically used to assign students a course grade. Assessment of learning Formative assessment, also referred to as educative assessment, is used to aid learning. Assessment for learning Wikipedia

Assessment with feedback Process improvement Execution “Continuous improvement”

Applying assessment “Engineer” learning based on feedback from the use of eLearning systems. Assess prescribe deliver

Assessment as a management style Open availability of data Open discussion of the alternatives “Explaining” the solution Within the organization Outside the organization To the public Expect fair and unfair criticism!

The Commission wrote “We recommend that America’s colleges and universities embrace a culture of continuous innovation and quality improvement. We urge these institutions to develop new pedagogies, curricula and technologies to improve learning, particularly in the areas of science and mathematics.” Spelling Commission, U.S. Department of Education

Two perspectives of eLearning Graduate programs and undergraduate tutorials and independent study tend to: Focus on collaboration among students and faculty Use resources typical of a cross-institutional research effort Undergraduate programs, especially first and second year tend to: Focus on online tutorials, “drill and practice,” and assessments for learning Use faculty-recommended resources and Internet services

Some US/UK differences Thomas Weko, Higher Education Policy Institute, Oxford, March 2004

Unprepared (reading) 43.9% did not have the “ability to make relatively simple inferences beyond the author ’s main thought and/or understand and evaluate relatively abstract concepts.” And additional 39.5% did not have “the ability to make complex inferences or evaluative judgments that require piecing together multiple sources of information from the passage (Rock, Pollack and Quinn 1995).

Unprepared (mathematics) 29.7% could not solve simple operations with decimals, fractions, powers, roots or solve simple problems requiring the understanding of low-level mathematical concepts. Many could not do decimal arithmetic. An additional 40% could not understand intermediate-level mathematical concepts and could not formulate multistep solutions to word problems.

Use of supplementary material

Staffing instructional technology

Assessments Assessing “tools” Assessing students Capabilities [Preparation, learning style] Learning [Added knowledge] Assessing teaching and learning Teaching methods Learning opportunities

Success in math courses

Assessing remedial education The results suggest that students in remediation are more likely to persist in college in comparison to students with similar test scores and backgrounds who were not required to take the courses. They are also more likely to transfer to a higher-level college and to complete a bachelor's degree. Bettinger 2005

Assessing faculty Whether an instructor teaches full-time or part-time, does research, has tenure, or is highly paid has no influence on a college student's grade [0.5%], likelihood of dropping a course [1.3%] or taking more subsequent courses in the same subject [4.0%]. Hoffman 2006

Assessing faculty We have shown that student evaluations [of faculty] differ from the ideal construct, because they do not appear to reflect learning, but are sensitive to grading leniency. Weinberg 2007

A portal is the toolbox of the knowledge worker

Suppes at Stanford University First Year Russian 1972

Suppes at Stanford University Learning Trajectories

Learning trajectories Content Mastery Course Grade Time End of Scheduled Term ABCDF Add “supplementary materials” to the red line Add “more time” to the purple line Supplementary mastery vs boredom Additional time for those who need it. Based on the work of Pat Suppes at Stanford University

Assessing portals At one university 85% of the students personalized their portal. At most colleges and universities less than 10% personal their portal. uPortal 2005 Why?

Assessing portals “Time on task” Focus Purpose Learning style, Warning to faculty and advisors Suppes 1971-1972 Focus Portlet/channel use AICC 2001 Purpose Portlet/channel use by type University of Delaware 2004

Assessing VRE portals Data [from the archeology site] is available in 7 weeks instead of one year. Data from several projects is consolidated and has a single presentation increasing the quality of research. Others use the data and contribute. Mark Baker 2008 [unpublished]

Impact on classroom activities

The student’s perspective “The explosion of knowledge” Collateralized Debt Obligation Structured Investment Vehicle

“Portals are a journey of increasing functionality for expanding communities."

The end jxf@immagic.com jxf@Georgetown.edu

What about Web 2.0? What is the educational purpose? Is the college willing to take responsibility for content? Is the content to be shared? Who is responsible for content? Who “owns” the content?