1. Product Development & User Testing
Dr. Minjuan Wang Department of Educational Technology, SDSU

2. Somewhat time and labor intensive
Been accused of being too linear
It has been criticized for being time and labor intensive, and too linear.
A more flexible ADDIE model?

3. The Successive Approximation Model:
An agile design method
Evaluate
Design
Develop
Start
End
We’re going to look more closely at the successive approximation model, which has been getting more popular in recent years.
It is considered a more iterative process that emphasizes prototyping more heavily than the other models.

4. Summative Evaluation Traditional Instructional Design Models Linear
Tasks performed in sequential fashion
Summative evaluation (ADDIE)
Long development phases
Time consuming and expensive to make changes

5. Formative Evaluation Simultaneous design and development
Ongoing feedback from end-users
Iterative design cycle
User community
Design community

6. Iterative Design concept prototype testing product revisions
Formative Evaluation
concept
prototype
testing
product
revisions

7. Designer Blindness

8. Prototype Testing
Main Entry: pro·to·type 1 : an original model on which something is patterned : ARCHETYPE 3 : a standard or typical example 4 : a first full-scale and usually functional form of a new type or
design of a construction (an airplane)

9. Rapid Prototyping Early stages
does the product meet the users' needs and expectations?
Start with a crude prototype and refine
Prototypes may be discarded rather than carried forward as a kernel for further development.
Increased “editing power” makes it easier to alter designs; less need for elaborate specifications.
requires good flow of testing data.

10. Formative Evaluation may…
Be done very early in the design process
when ~10% of project resources have been expended
Provide solid measurements of task performance
Help designers empathize with user’s needs and context
Uncover problems
help developers decide when the project can move on to the next stage
that were not noticed in the design phase

11. Types of Prototypes Low fidelity Mid fidelity High fidelity
paper and pencil sketching; rapid cut-and-paste.
Mid fidelity
screen printout.
High fidelity
working example, similar in appearance and functionality to the proposed final product.

12. When to Test? user testing assess needs define goals generate specs
Create a product
Identify a piece
You may combine Alpha and Beta Test in 572
Fully develop it
alpha test
beta test (field test)

13. TIME Formative Evaluation Summative Evaluation design freeze needs
Design Gel
design freeze
needs
specs
prototype
Formative Evaluation
Summative Evaluation
TIME

14. Instructional Effectiveness
What to test
Instructional Effectiveness
Is this proposed design effective in meeting the instructional goals?
Comprehension
Retention
Transfer

15. Test on Usability
Nielsen (1993) defines usability with the use of measurable components so that usability can be systematically be evaluated and improved.
Learnability: How easy it is to learn to use the system when referring to a novice user’s experience at the initial part of the learning curve.
Efficiency of Use: This attribute refers to expert users and the level of their performance.
Memorability: How easy the system is to remember when casual users return to it in infrequent periods of time.
Errors: How often do users make errors while using the system, how serious are these errors, and how do users recover from these errors
Satisfaction: How pleasant the system is according to the the individual satisfaction of each user.
Learnability
Memorability
Errors
Satisfaction
time on task
Efficiency of Use

16. Formative Evaluation Questions
Are the instructional activities clearly structured?
Is the content perceived as useful and relevant by the learner?
Are the practice exercises easy to understand and complete?
Is your delivery system well organized (web navigation, blackboard setup, moodle layout) easy to use?
Is the instructor guide easy to follow?

17. Abbreviated Formative Evaluation
Focus on potential weaknesses in your design
Formulate formative evaluation questions
probe for and evaluate potential weaknesses
What are you uncertain about? Me, how to teach statistics. Is it better to have them do draw a chart by hand; or it is better to work with software? Which concept to present first?

18. How many test subjects?
A correlational graph. The more you test, the more problems you’ll detect. Since 572 testing is rather informal, you decide how many to test. The more the merrier.

19. How To Test? Here is one approach:
A few at a time
Revise
Test it again
Test more if you wish
Your time is the limit!!!
You can also do one big testing session (with all participants)
I am sure reality will help you decide

20. T3R: Test, Tweak, Test, Report
Here is the rigorous approach. You can only just do one test. As long as your goal is accomplished (you found answers to your questions).
Make recommendations based on your tests

21. Thank You
From crystaljingsr
Flickr-CC 21