Glenn Ledder Department of Mathematics University of Nebraska-Lincoln Assessment Using Online Assignment Management Systems

1. Defining the terms 2. OAM advantages and disadvantages 3. Low-maintenance administration 4. Accurate and reproducible results 5. Getting meaningful results Assessment Using Online Assignment Management Systems

Defining the Terms

“Assessment” Conflicting meanings

“Assessment” Conflicting meanings –General meaning: measuring academic success in any context

“Assessment” Conflicting meanings –General meaning: measuring academic success in any context –Specific meaning: measuring academic success in a broad context

“Assessment” Conflicting meanings –General meaning: measuring academic success in any context –Specific meaning: measuring academic success in a broad context (larger than a single course)

Dictionary definition: –“assess”: to determine the importance, size, or value of –“evaluate”: to determine or fix the value of Assessment vs Evaluation

Dictionary definition: –“assess”: to determine the importance, size, or value of –“evaluate”: to determine or fix the value of Connotation: –“assessment” usually emphasizes the making of summary judgments –“evaluation” usually emphasizes the measurement of quantitative value Assessment vs Evaluation

Assessment: –measuring knowledge and understanding Evaluation: –measuring achievement

Assessment vs Evaluation Assessment: –measuring knowledge and understanding –used for placement Evaluation: –measuring achievement –used to determine course grades

Assessment vs Evaluation Assessment: –measuring knowledge and understanding –used for placement Evaluation: –measuring achievement –used to determine course grades Test: an instrument used for assessment, evaluation, or both

OAM Advantages and Disadvantages

OAM Advantages/Disadvantages + Immediate grading (no wait for grading or machine scoring)

OAM Advantages/Disadvantages + Immediate grading + General purpose testing center (no need for direct faculty involvement)

OAM Advantages/Disadvantages + Immediate grading + General purpose testing center – Administrative difficulties (How do you maintain the database?)

OAM Advantages/Disadvantages + Immediate grading + General purpose testing center – Administrative difficulties + Algorithmic Q’s and symbolic A’s (individual answers, no prompts)

OAM Advantages/Disadvantages + Immediate grading + General purpose testing center – Administrative difficulties + Algorithmic Q’s and symbolic A’s – Difficulties in writing questions (How to avoid tricky, picky, sticky?)

Low-Maintenance Administration

Lots of instructors and proctors –Should require minimal knowledge

Low-Maintenance Administration Lots of instructors and proctors –Should require minimal knowledge One or two faculty manage the system –Should require minimal time

Low-Maintenance Administration Lots of instructors and proctors –Should require minimal knowledge One or two faculty manage the system –Should require minimal time Questions are “permanent.” Student records are “transient.”

106 COURSE Question banks Gateway exam Practice assignments The 106 COURSE data is “permanent.” The only regular changes are to the Gateway exam dates. Maintained by faculty manager. The Math 106 EDU folder structure

106 COURSE Question banks Gateway exam Practice assignments 106 CLASS Question banks Gateway exam Practice assignments Student records The 106 COURSE data is “permanent.” The only regular changes are to the Gateway exam dates. Maintained by faculty manager. Students register for their own 106 CLASS. Tests are inherited. Student records are local. Maintained by individual instructors/proctors.

Accurate and Reproducible Results

Reproducibility Tests should measure something. –The test has to yield consistent results in multiple administrations for a single student.

Reproducibility Tests should measure something. –The test has to yield consistent results in multiple administrations for a single student. But questions must change when a test is given at different times. –This is especially important for evaluations.

Reproducibility Use template problems to get a great variety of answers. Use template problems to get uniformity of content and difficulty. Group problems into categories that are consistent in content and difficulty.

The Math 106 Gateway Exam 1.Elementary functions: x n, sin( ax ), cos( ax ), tan( ax ), e ax, ln x, n x 2. Products 3. Quotients4. Compositions 5. Compositions of compositions 6. Products with a composite factor 7. Compositions of products 8. Quotients with an embedded composition 9. Quotients with an embedded product 10. Functions defined by equations 10 questions, 8 correct to pass

Category 4 - Compositions A, C, N > 0; B ≠ 0; K ≠ 0, 1; The algorithm chooses 5 random integers satisfying certain requirements.

Category 4 - Compositions A, C, N > 0; B ≠ 0; K ≠ 0, 1; X = t, u, v, w, x, y, z ; The algorithm randomly chooses an independent variable.

Category 4 - Compositions X = t, u, v, w, x, y, z ; A, C, N >0; B ≠0; K ≠0,1 P = X N +B, X N +BX Q = AX N +B, AX N +BX, sqrt (X)+B S = sin AX, cos AX, tan AX T = e -CX +B, e KX +BX U = Ae -CX +B, Ae KX +BX, A ln X, AN X The algorithm randomly creates functions P, Q, S, T, U, using some of the chosen integers.

Category 4 - Compositions X = t, u, v, w, x, y, z ; A, C, N >0; B ≠0; K ≠0,1 P = X N +B, X N +BX Q = AX N +B, AX N +BX, sqrt (X)+B S = sin AX, cos AX, tan AX T = e -CX +B, e KX +BX U = Ae -CX +B, Ae KX +BX, A ln X, AN X F 1 = sqrt( P ), sqrt( S ), sqrt( T ) F 2 = S N, T N F 3 = ln Q, ln CS F 4 = e Q, e CS F 5 = sin Q, cos Q, sin U, cos U The algorithm chooses one of 5 functions based on prior elements. There are 38 templates, each with 7 independent variables and at least one parameter

Reproducibility Tests should measure something. –The test has to yield consistent results in multiple administrations for a single student.

Accuracy Tests should measure what they were intended to measure.

Accuracy Tests should measure what they were intended to measure. –Questions should have to be done right to be counted right.

Accuracy Tests should measure what they were intended to measure. –Questions should have to be done right to be counted right. –Students who can do what the test is supposed to measure should get a good score.

Counted right → Done right Savvy test takers can do well on multiple choice tests even when they don’t understand the material.

Counted right → Done right Savvy test takers can do well on multiple choice tests even when they don’t understand the material. For calculations, students should have to type in the answer.

Counted right → Done right Savvy test takers can do well on multiple choice tests even when they don’t understand the material. For calculations, students should have to type in the answer. For conceptual questions, use multiple selection. (Which of the following…)

Can do → Good score Tricky: Misinterpretation or small mistakes account for many wrong answers.

Can do → Good score Tricky: Misinterpretation or small mistakes account for many wrong answers. Find the derivative of cos e 2x and Find the derivative of e -2 cos x are tricky

Can do → Good score Tricky: Misinterpretation or small mistakes account for many wrong answers. Find the derivative of cos e 2x and Find the derivative of e -2 cos x are tricky, compared to Find the derivative of e 2 cos x

Can do → Good score Tricky: Misinterpretation or small mistakes account for many wrong answers. Find the derivative of cos e 2x and Find the derivative of e -2 cos x are tricky, compared to Find the derivative of e 2 cos x The problem is meant to test the chain rule, not elementary algebra.

Can do → Good score Picky: Too many details connect minor errors with wrong answers. Find the derivative of cos 2 (2x+3)+ 4 sin x. Find the derivative of 4 x 5 -2x cos ( e x 2 ).

Can do → Good score Picky: Too many details connect minor errors with wrong answers. Find the derivative of cos 2 ( 2x+3 ) + 4 sin x. Find the derivative of cos 2 ( 2x+3 ). Find the derivative of 4 x 5 -2x cos ( e x 2 ). Find the derivative of e x 5 cos ( e x 2 ).

Can do → Good score Picky: Too many details connect minor errors with wrong answers. Find the derivative of cos 2 ( 2x+3 ) + 4 sin x. Find the derivative of cos 2 ( 2x+3 ). Find the derivative of 4 x 5 -2x cos ( e x 2 ). Find the derivative of e x 5 cos ( e x 2 ). The simpler problems are hard enough.

Can do → Good score Sticky: Difficulties in transferring answer from paper to computer. Find the derivative of —–. ———— 2x 2 x+3 4x(x+3)-2x 2 (x+3) 2

Can do → Good score Sticky: Difficulties in transferring answer from paper to computer. Find the derivative of —–. ———— Find the derivative of —– at x = 2. — 2x 2 x+3 4x(x+3)-2x 2 (x+3) 2 2x 2 x

Can do → Good score Sticky: Difficulties in transferring answer from paper to computer. Find the derivative of —–. ———— Find the derivative of —– at x = 2. — 2x 2 x+3 4x(x+3)-2x 2 (x+3) 2 2x 2 x Type 4x(x+3)-2x 2 / (x+3) 2, get 2x 2 (x+3) 2 4x(x+3) - ———

Getting Meaningful Results

Accuracy Tests should measure what they were intended to measure.

Getting Meaningful Results Tests should measure what they were intended to measure. What should OAM evaluations and assessments be intended to measure?

Don’t give up paper evaluations

Use paper exams for questions that demand partial credit and questions where the answer is an integral, a graph, or an explanation.

Don’t give up paper evaluations Use paper exams for questions that demand partial credit and questions where the answer is an integral, a graph, or an explanation. Use OAM evaluation for routine computations and basic concepts.

Don’t give up paper evaluations Use paper exams for questions that demand partial credit and questions where the answer is an integral, a graph, or an explanation. Use OAM evaluation for routine computations and basic concepts. Testing takes time and effort. Decide what is important and then create the test to match. Set high standards and allow retakes.

Do give up paper assessments

Assess routine computations and basic concepts.

Do give up paper assessments Assess routine computations and basic concepts. Expect very few retakes. Qualified students should pass on the first try. Provide necessary formulas online or as a printed supplement. Test conceptual understanding and knowledge, not memory for trivia.