1. Meaningful Assessment and the Impact of Technology Catherine Kelley, Ph.D. Senior Faculty Consultant Assistant Professor

2. Meaningful Assessment and the Impact of Technology Assessment of Mobile Computing at Seton Hall Catherine Kelley, Ph.D. Assistant Professor, Psychology Senior Faculty Consultant, TLTC

3. Meaningful Assessment and the Impact of Technology Why assess? Track progress being made on your project Greater understanding of teaching & learning Ensure high-quality experience for your students Communicate successes / challenges to colleagues

4. Meaningful Assessment and the Impact of Technology Traditional (Flawed) theoretical model: Teaching => Learning Teaching + Computers => Improved Learning What is wrong with this model?

5. Meaningful Assessment and the Impact of Technology Traditional (Flawed) Research Design: Control group - no computer Experimental group - computer Measure exam scores What is wrong with this design?

6. Meaningful Assessment and the Impact of Technology Some problems with traditional model Hawthorne effects Multivariate nature of change means that an experiment would be hopelessly confounded Your outcome variables may (will probably!) change with new teaching techniques –You may not know yet what the really important outcome variables are!

7. Meaningful Assessment and the Impact of Technology What about a multivariate model? So Many Variables! Each So Difficult to Measure! Huge n required to do any meaningful multivariate study Not a bad idea, but was outside of the intended scope of our project

8. Meaningful Assessment and the Impact of Technology So what do you do? What about educational quality variables? Theoretical models of educational quality: e.g. Chickering & Gamson, “Powerful Partnerships,” etc. Common themes emerge

9. Meaningful Assessment and the Impact of Technology What is “educational quality”? Student - professor contact Student - student contact Prompt feedback on performance Explore course materials in many ways Take responsibility for learning Etc.

10. Meaningful Assessment and the Impact of Technology Seton Hall Instruments Two Versions: “Standard Form”, “New Form” Standard Form has been used for several years New Form has been piloted with considerable success

11. Meaningful Assessment and the Impact of Technology Standard Form “Compare your mobile courses, or those incorporating information technology, to your non-mobile courses, or those not incorporating information technology. Indicate how the use of information technology impacts each of the following items.”

12. Meaningful Assessment and the Impact of Technology Standard Form (examples) Have contact with my professors on course content Spend time with other students on course related materials Discuss, write and think about course material in depth Get prompt feedback on my performance from my professors …. 29 of these questions

13. Meaningful Assessment and the Impact of Technology New Form Same questions, but students answer only about one course And then indicate how much IT is used in that class But also, how many traditional pedagogical techniques are used in that class

14. Meaningful Assessment and the Impact of Technology Results:Old Form Mobile courses look better on educational quality - Sum across all measures

15. Meaningful Assessment and the Impact of Technology Results: Old Form

16. Meaningful Assessment and the Impact of Technology Results: New Form The more IT, the higher the educational quality score

17. Meaningful Assessment and the Impact of Technology Results: New form

18. Meaningful Assessment and the Impact of Technology Results: New Form The more traditional techniques, the higher the educational quality score – but effect is very weak

19. Meaningful Assessment and the Impact of Technology Results: New Form

20. Meaningful Assessment and the Impact of Technology Summary of findings Students perceive that educational variables are higher in Mobile classes Student satisfaction with Mobile remains high

21. Meaningful Assessment and the Impact of Technology The future Institutionalizing the assessment process –Risks: over-surveying students, poor response rate –Asking some tough questions: Retention, student skill, negative impact of technology Surveys of faculty, administrators

22. Meaningful Assessment and the Impact of Technology The future Assessment reports should also make use of archival data, not just surveys –E.g. how many courses make use of technology? And to what extent? Are we realistic in our expectations of faculty? Administrators?