Understanding the Effects of Technology Based Enhancement of Professional Development Alan Lesgold February, 2000

Context Sensitive Evaluation

Example Causal Model

Types of Maturity Models Content Maturity Process Maturity Technology Infrastructure Maturity Educational Software Product Maturity People Maturity

Maturity Models Maturity is defined as involving a set of features For each feature, a set of levels or stages is defined, with the assumption that organizations characteristically evolve through these stages in order For each level of each feature, a scoring rubric is established that allows raters to reliably score what level the organization has attained on that feature

Advantages of Maturity Models Can be applied relatively cheaply and can produce scores immediately on site Relevant stakeholder communities can participate directly in discussions of the appropriate features to be included in the model Scoring rubric itself is a guide providing specific goals to which schools can aspire and toward which they can work

Kinds of Studies Needed Studies of Necessary Preconditions Descriptive Studies A description of the intervention over time An assessment of the degree of implementation A tracking of contextual variables that are likely to affect the levels of implementation or outcomes Tracking of important school outcomes through time An assessment of the reasons for high and low levels of implementation

Kinds of Studies Needed Cost-Benefit Analyses Innovations in Data Gathering Task-embedded evaluation Neural networks

The Medical Approach – Following NABBP Study Costs and Benefits of a Longitudinal Context- Sensitive Evaluation of Educational Innovations Set Up a Coordinating Body Develop Plans for Longitudinal Test bed and Select Test bed Sites Develop Maturity Models Specify Global Outcome Variables Gather First Round of Baseline Data and Develop Scoring Rubrics Begin Specific Innovation Evaluation Studies Begin Microstudies of the Test beds Themselves

Issues in the Medical Approach Who goes into the control group? Can you control teacher behavior sufficiently Taking account of training needs and the scaling of training Actually taking account of all relevant populations