Expert Appraisal

Types of Expert Appraisal Some modifications of the instructional material are based on the developer’s hindsight; Most should be based on feedback from others...

This process is called “formative evaluation”... That is, evaluation undertaken for the improvement of the instruction material during its development

Expert appraisal is one stage of formative evaluation. Based on this feedback the material is modified to improve its –appropriateness, –effectiveness, –usability, and –technical quality

Guidelines for conducting expert appraisals Use a varity of specialists Restrict each expert to his/her own field Present only relevant parts of the instructional material for evaluation by each expert Specify the purpose of the review by each expert by defining the context in which the material is to be used and provide a checklist or questionnaire

Technical review Language –To check the appropriateness of the language for the target trainees and the correctness and clarity of it usage –See example

Media –To check the technical quality of the finished product and the compatibility of the materials and equipment –Reviewer should be knowledgeable in the specific medium used--and suggest emphasis upon ways of improving the technical quality.

Format review –Each instruction format has its own standards and criteria, for example: Programmed instruction –Step size, error rate, etc. Simulation games –Fidelity, replicability, and play time/prep time ratio Given definition of the context and the purpose of the review-- appraisal exper should be free to use own procedure for the evaluation

Instructional review Appropriatness –The extent to which the objectives and the content of the instructional material are constant with the goals of the training –See example

Effectiveness –The extent to which instructional objectives are attained by trainees –See example

Feasibility –The extent to which the material is usable for training –See example

From appraisal to revision The developer’s own hindsight on what might have done differently should not be discounted. Not all suggestions from all sources are worthy of being acted upon. Contradictory suggestion from different experts are often encounter. (“more precise terminology” vs “too much jargon”)

Expert appraisal does not reduce the number of decisions necessary; It merely presents different points of view Decisions on which suggestions to act on must be made and then translated into actual modification of the material. When a drastic change is proposed--it is a good idea to check back w/major appraisal experts involved in the original evaluation.