1. © E. Kowch 2002 1 Telecommunications in Education: Examining Distributed Teaching and Learning Environments (EDER 677 L.91 ) teaching and Learning Environments: Guides for Critical Reviews and Evaluation for the Asynchronous Class and for the Web Site on November 7 from Calgary

2. © E. Kowch 2002 2 Some pointers If you would like to discuss your online project with me, I will be sure to be available at a time when you need… If you would like to discuss your DL portfolio, I would be delighted to discuss that with you too… just let me know… Housekeeping

3. © E. Kowch 2002 3 Presentation Overview In this lesson, we will: Explore the issues around designing, developing and utilizing web based assessment techniques for online learning environments or their parts - from the teacher or designer perspective. Discuss the importance of formative and summative program/course evaluation in the online learning context - from the teacher or designer perspective.

4. © E. Kowch 2002 4 Assessment and Evaluation - Definitions Assessment - determine changes in learner performance Evaluation - make judgments and decisions about learners Program or course evaluation - appraise the success of your program or course

5. © E. Kowch 2002 5 Assessment and Evaluation - Self Activity (something to ponder as you create evaluations for your own online learning project) List examples of web based assessment activities that you prefer (see the resource section of this lesson) What types of student learning do these activities support (e.g. Bloom’s Taxonomy – knowledge, comprehension, application, analysis, synthesis, evaluation)? Identify issues surrounding the use of web based assessment activities that you feel strongly about, and address those issues in your project.

6. © E. Kowch 2002 6 Triangle of Assessment The multidimensional approach to assessment Knowledge Transfer and Performance Knowledge Constructs and Reflection Concrete Knowledge Acquisition Bloom’s Taxonomy http://www.coun.uvic.ca/learn/program/hndouts/bloom.html http://www.coun.uvic.ca/learn/program/hndouts/bloom.html

7. © E. Kowch 2002 7 Concrete Knowledge Acquisition Concept "Factual Foundation Knowledge" - provides the ground floor in a learning hierarchy. You can evaluate the things that must be known - facts. The working knowledge or language of a subject discipline can be considered as a foundational knowledge to succeeding in a learning event, online or anywhere.

8. © E. Kowch 2002 8 Concrete Knowledge Acquisition Working Knowledge Examples Calculus The student can recognize and count numbers The student can add, subtract, multiply and divide Literary criticism and creative writing The student can apply the alphabet and the basic rules of grammar

9. © E. Kowch 2002 9 Concrete Knowledge Acquisition Assessment Techniques Measuring student mastery of the factual “foundation knowledge” Some in instructional methodologies where you use specific learning evaluation techniques (Great source book: Gronlund: The Assessment of Student Learning” 7th Ed.) We take this up in EDER 673 in detail. Drill and practice is one assessment technique:\ with components designed to check the level of mastery of basic knowledge. Drill and Practice multiple choice true and false matching ordering short answer

10. © E. Kowch 2002 10 Concrete Knowledge Acquisition More Assessment Ideas Give a Preview quiz – on preparatory textbook, article or online readings Give a Review quiz – on lecture content Give a Collaborative quiz – online discussion about the questions and answers on a quiz Students can develop their own quiz questions – learning how to design tests Online, other students can respond and evaluate the documents. For stronger students in the class – the writing of mid-term and final exam questions can also result in credit for this online “assignment”. Online quizzes - Integrity test questions – first or last question – Did you do this quiz on your own? (on site supervised testing is a simple and effective option).

11. © E. Kowch 2002 11 Concrete Knowledge Acquisition Web Based Tools for Evaluation: See the Resource Section in the Lesson Plan for November 7th for more of these: Hot Potatoes http://web.uvic.ca/hrd/halfbaked/ Quiz Lab http://www.funbrain.com/quiz/index.html TeAch-nology.com's Crossword Puzzle Maker http://www.teach-nology.com/web_tools/crossword/ Discovery School's PuzzleMaker (Word Searches) http://www.puzzlemaker.com/

12. © E. Kowch 2002 12 Knowledge Constructs and Reflection The Concept of Knowledge Constructs This is the process of students constructing and/or discovering their own knowledge. ( Self evaluation with cognitive (model) direction). Internalization and integration of new information with previously held knowledge (in the good old bad old days, we called this “synthesis” ;-)

13. © E. Kowch 2002 13 Knowledge Constructs and Reflection The Concept of Reflection Students reflecting on the construction of their knowledge as part of the evaluation process: Zone of proximal development can be created online too: the instructor watches and guides students to evaluate each other after they evaluate themselves. Students are comparing thoughts and ideas with others.

14. © E. Kowch 2002 14 Knowledge Constructs and Reflection Assessment Techniques Structured journals (e.g. responding to specific questions) Open-ended reflection journals (e.g. responding to broad or open questions) Collaborative discussions and summaries

15. © E. Kowch 2002 15 Knowledge Constructs and Reflection Assessment Techniques for Online Discussions See our home page links

16. © E. Kowch 2002 16 Knowledge Constructs and Reflection Assessment Examples Example One - Class Discussion Scoring Guide http://www.tageducation.org/webct/webct/level1/6discussrubric.htm a rubric for a 100-point project (60% for responses to weekly readings, 40% for responses to classmates postings) Example Two - Grading Online Discussions http://www.oznet.ksu.edu/dp_iet-edtech/TechforTeaching/grading.htm various models based on support/clarity/helpfulness, an A-F scale, and groups grading each other

17. © E. Kowch 2002 17 Knowledge Constructs and Reflection Assessment Examples Example Three - Evaluation and Grading Criteria for WebBoard Discussion Assignments http://www.ivcc.edu/eng1001/handout_evaluating_discussions.htm an outline of four evaluation criteria for discussion assignments Example Four - Discussion Guidelines and Grading Standards http://www.stedwards.edu/ursery/discuss.htm discussion grade guidelines on an A to F scale, as well as a 10-point guideline for participation

18. © E. Kowch 2002 18 Knowledge Constructs and Reflection Recall the Modes of Web Based Computer Mediated Communication Tools Asynchronous E-mail File transfers Threaded discussion forums Synchronous Virtual chats Shared whiteboards, URLs, and applications MUDs(multi-user dimensions) and MOOs (multi-user dimension - object oriented)

19. © E. Kowch 2002 19 Knowledge Constructs and Reflection Web Based Resources that help you evaluate your online learning environs, considering the mode of delivery / interaction chosen by the designer: Using Asynchronous Conferencing to Promote Critical Thinking http://crlt.indiana.edu/publications/crlt99-8.pdf Instructional Design Principles for Teaching in Computer Conferencing Environments http://www.cvm.tamu.edu/wklemm/instruct.html Shaping Cyberspace Into Human Space http://css.sfu.ca/update/vol6/6.3-harasim.main.html Handbook for Instructors on the use of Electronic Discussions http://www.osu.edu/education/ftad/Publications/elecdisc/pages/index.htm Check for Plagiarism http://www.turnitin.com http://www.turnitin.com

20. © E. Kowch 2002 20 Knowledge Transfer & Performance Concept Performance-based assessment - students demonstrate the (learned) behavior to be measured (by the evaluator). Authentic assessment - students demonstrate the acquisition and mastery of a 'real world' skill or concept. Learners applying knowledge and skills rather than simply remembering facts

21. © E. Kowch 2002 21 Knowledge Transfer & Performance Web Based Resources: Hotlinks to Assessment tools: Performance Assessment Performance Assessment Educational Issues Series Strategic Questions - What to Consider When Planning for Electronic Portfolios Electronic Teaching Portfolios Electronic Portfolio Checklist Portfolio Assessment Student Portfolios: Classroom Uses Student Portfolios: Administrative Uses Alternative Assessment and Technology http://www.ed.gov/databases/ERIC_Digests/ed365312.html http://www.ed.gov/databases/ERIC_Digests/ed365312.html Assessment Terminology: A Glossary of Useful Terms http://www.newhorizons.org/assmtterms.html http://www.newhorizons.org/assmtterms.html Collected Assessment Resources Center for Research on Evaluation, Standards, and Student Testing http://www.cse.ucla.edu/ http://www.cse.ucla.edu/

22. © E. Kowch 2002 22 Online Program/Course Evaluation - Remember this Model from earlier in the course? Now we look at the “Evaluate” loop: (More detail will come in EDER 673)

23. © E. Kowch 2002 23 Online Program/Course Evaluation - Definitions Formative Evaluation The ongoing analysis of your program/course design during development and implementation. Much more on this in EDER 673 and EDER 675 Summative Evaluation Is conducted to determine whether your course or program is worthy of adoption or continuation leads to a “go/no go” decision on the program.

24. © E. Kowch 2002 24 Online Program/Course Evaluation - Formative Some of the on-going process that designers use at each stage of the course development process: They have colleagues and students constantly reviewing the development and implementation of your web course site – internal review They develop an evaluation checklist or form They ensure that each module is piloted They make the necessary revisions based on informal and formal feedback mechanisms

25. © E. Kowch 2002 25 Online Program/Course Evaluation (Summative) Judging the worth or merit of a program/course: This evaluation is conducted after a program/course/design has been in place for a complete cycle – usually an external review to the institutional course distributor Key Questions used in this type of program evaluation: Effectiveness – Do students learn the basic learning objectives or exceed them? Appeal – Does the program/course/design engage learners and students – do they like it? (Affective) Efficiency – Is the program/course cost effective?

26. © E. Kowch 2002 26 Program/Course Evaluation - Resources Article Shrock, S.A. and Geis, G.L. (1999). Evaluation. In H.D. Stolovitch and E.J. Keeps (Eds.) Handbook of human performance technology (2 nd ed.) ( Chapter 10 – Evaluation, pp. 185- 209). San Francisco, Jossey-Bass Pfeiffer. Web sites See the resources section in the lesson plan for hundreds of resources.

27. © E. Kowch 2002 27 For next session Contribute to this week’s discussion forum. Check out the Web Site Home Page for further instructions (core readings). Your online project production should have started by now - your professor will review anything that you like ahead of time… Your Personal DL portfolio web site should be under construction now. Adieu from Calgary..