1. Modeling Education with Work Domain Analysis: A Work in Progress Lee Nickles September 11, 2002

2. Outline Defining Educational Evaluation Obstacles to Performing an Evaluation Work Domain Analysis as Solution Process of Building the Model

3. What is Educational Evaluation? “Evaluation in the context of educational systems is briefly defined as examining the effectiveness of an educational system (or component of that system) in meeting learning and teaching goals.” Based on: (Bloom et al, 1971; Stevens, et al [NSF], 1993; Walker, 1997)

4. Formative vs. Summative Evaluation Formative –During development and implementation –Examines system to reveal problems and improvements Summative –At the end of the system’s life –Examines overall success of the system My focus: Formative

5. Formative Evaluation is a “Good Thing™” Similar to Quality Control, leads to continuous improvement Changes can be made in the current life cycle of the system

6. Evaluation Obstacles (1) Education systems have multiple, interacting components –Work together to produce learning Isolating which component produced what or (how much) learning is difficult –Testing each individually is too great a burden –Components build on each other...

7. Which of these activities produces learning?

8. Evaluation Obstacles (2) There is a lack of validated measurement instruments for evaluation –There may not be a test that analyzes whether or not your specific web site produced learning –You may not have the time or resources to generate a valid measurement

9. Validation Solution Measurement Triangulation –Use multiple measurements of different types to measure the thing you want to evaluate Subjective Opinion/Attitude Performance Process/Behavior –If they give similar/confirming readings, you can have greater confidence in your result

10. Triangulation Problem How do we apply this practically? That is, how do you know your measurements are triangulating properly? –They are measuring slightly different things Need a structured way to apply triangulation

11. Proposed Solution: Work Domain Analysis Interacting components –Specifically examines tightly coupled systems, with many relationships between the components Structure to apply triangulation –Provides the abstraction hierarchy framework to guide measurement triangulation

12. Work Domain Analysis - Abstraction Hierarchy Rasmussen, 1985 Modeling tool Broad and deep view of the system Way to represent the levels of abstraction of a system together and how components, functions, and goals connect to each other

13. Levels of System Abstraction Can describe a system at levels of abstraction from the physical ConcreteAbstract Physical Objects System Functions Goals and Objectives

14. Means-Ends Relationships Items in one level of abstraction are related to others by means-ends relationships 1: Goals 2: Functions 3: Physical Form Means: Level 3 tells HOW Level 2 is accomplished Ends: Level 1 tells WHY Level 2 is performed

15. Example for Community of Scholars

16. Education and the AH Problem: the AH hasn’t really been applied thoroughly to intentional systems –Intentional systems: systems where actions are based on the user’s intentions more than physical constraints Some say it can’t be done (Wong et al, 1998) Same say it can (Hajdukiewicz et al, 1999) No really good examples in the literature

17. Building an Abstraction Hierarchy to Describe Education I’ve gone through this process for some time (too long?) Many iterations, back and forth with Dr. Pritchett NOT Done! Process may be as informative as the product, so here is (approximately) the process...

18. Early Versions Considered various ways an AH has been represented –Separating very different domains that are part of the same system –Think about how cognitive functions, social activities, and information flow should be represented –Systems based on user intention rather than physical constraints

19. Early Framework - Domains Difficult to Separate/Categorize things

20. I noticed something... Process Control –Ongoing process –Goal: keep process stable –Change is bad Education –Defined start/end of educational system –Goal: produce learning in the students –Change is desired AH has typically been applied to process control

21. Education as Change Over Time

22. Education as Pragmatic Change Over Time Is it this simple? Not very general!

23. Education as Pragmatic Domains Interesting, but is the domain only interesting at the top level? Did we lose the element of change?

24. Three Dimensional AH? Maybe this was going to take multiple dimensions of domains to represent About this time, I switched from pen and paper to Excel to create AHs and used the IT Web site for ISyE 4009 as a case

25. Multi-Dimensional, “Unit” Based What is a unit?

26. Multi-Dimensional, Week Based This is getting somewhere! But weeks are not general.

27. Multi-Dimensional, Assessment Based An assessment (homework, exam, project, etc.) as an essential part of course structure

28. Today: Bring Back Domains, Widen Scope

29. Where do I go now? Maybe stop now and model an entire course with this method –Are there things we’re leaving out? –What can we leave out? Consider, how useful is this for evaluation? –Can we easily generate an AH for a course? –Is it general enough to apply broadly? –Is it specific enough to be useful?