Assessment Seminar IMPLEMENTING ACTUAL AND PERCEIVED LEARNING ASSESSMENT TOOLS Victor Mbarika, Ph.D. Southern University

Background Undergraduate education in the US has been repeatedly criticized for failing to develop students’ higher-order thinking and problem-solving skills needed in the new information and technology-based economy. Higher-order cognitive skills reflect a set of skills an individual acquired over time for decision making under various conditions of certainty and time limit. It implies a learned ability to identify, integrate, evaluate, and inter- relate concepts within a given problem domain, thereby making the appropriate decisions to address the problem. In recent years research to improve students’ higher order cognitive skills has received little research attention. (e.g. AACSB, 2005; Landauer 1995; Dillon & Gabbard 1999; Mbarika 2003)

Assessment Objectives To identify, modify and validate two classes of assessment matrices: (1) Perceived Learning Matrix; and (2) Actual Learning Matrix to assess the impact of teaching business- related courses, especially on the development of higher- order cognitive skills. The three main constructs and criteria that will be used to assess student perceived learning include: Learning-Driven factor; Content-Driven factor; and Higher-Order Cognitive Skills factor (Goodhue & Thompson 1995; Hingorani et al. 1998).

Constructs used for the Perceived Learning Matrix (measures as reported by the learners/students)

Constructs and items used to measure learning-driven and content-driven factors ConstructItems Self Reported Learning (3 items)Improved my understanding of basic concepts, learned new concepts, learned to identify central management and technical issues. (Hingorani et al, 1998, Mbarika et al., 2001) Learning Interest (3 items)Discussed technical and managerial issues outside of class, did additional reading on technical and managerial issues, did some thinking for myself about technical and managerial issues. (Hingorani et al, 1998, Mbarika et al., 2001) Learned from Others (2 items)Learned to value other students’ point of view, learned to inter- relate important topics and ideas. (Hingorani et al, 1998, Mbarika et al., 2001) Challenging (4 items)Successful at bringing real life problems to the classroom, challenging, helpful in learning difficult topics, helpful in transferring theory to practice. (Hingorani et al, 1998, Mbarika et al., 2001) Timeliness (2 items)Task completed on time, case study reports delivered on time. (Goodhue and Thompson, 1995, Mbarika et al., 2001) Ease of use (3 items)Easy to learn, easy to use, had enough training to use the case study. (Goodhue and Thompson, 1995, Mbarika et al., 2001) Quality (6 items)Current, up to date, needed data, useful, appropriate level of detail, sufficiently detailed. (Goodhue and Thompson, 1995) Locatability (4 items)Easy to find, easy to locate, obvious, exact definitions of terms were available. (Goodhue and Thompson, 1995)

Higher-Order Cognitive Skills The factor, Higher-Order Cognitive Skills, represents how an individual has acquired an adequate portfolio of skills that could be used to make decisions within a specified period of time. It implies an improved ability to identify, integrate, evaluate, and inter-relate concepts within the (given) study, and thus make the appropriate decision in a given problem-solving situation. Sample of Pilot Instrument Already Developed It is strongly encouraged to give the strudents some bonus points to complete the survey (made available online eventually).

Tools used for the Actual Learning Matrix (measures as reported by the researchers)

Pre- and Post- Test instrument The pre- and post- test instrument is used to assess learning before the beginning of instruction. The same set of questions used for pre- and post- test Students not made aware of pre-test. –Pre-test could be administered as a pop quiz at the beginning of the semester. –Post-test administered as an announced quiz towards the end of the semester, but students must not be hinted that it will look like the pre-test Sample pre- and post- test instrument

Where you are needed !!! Working with a graduate assistant, we will collaborate with 3 or 4 volunteers to test these tools in some classes. Assistance from your end will be reword the perceived assessment instrument as well as the pre- and post- test instrument, to reflect the course you test– simply filling in the blanks…

Pilot Test Results I recently tested the perceived instruments in my e-business classes. Cronbach alphas were computed for each construct. The alphas were 0.96 for quality, 0.89 for locatability, 0.78 for timeliness, 0.66 for ease of use, 0.78 for Self reported learning, 0.83 for learning interest, 0.81 for challenging, and 0.71 for learned from others. The alpha for higher order cognitive skills was Although further analyses are need, current descriptive stats shows positive learning areas as well as areas that need improvement.

THANK YOU Questions???