Download presentation
Presentation is loading. Please wait.
Published byAmber Fletcher Modified over 9 years ago
1
Kajal Shah, Karina Silva, and Natalya Koehler
2
Rationale for Development It was designed to test a recent prescriptive theory of inductive reasoning by testing a strategy that enables participants to solve any kind of inductive reasoning problem It is accessible on the web site: http://129.186.123.4 http://129.186.123.4 2/18/20162
3
Description of Prototype It consists of Introduction, 6 lessons with quizzes, and 2 extra quizzes Problems and quizzes increase in complexity Quizzes help assess a student’s understanding of the material 2/18/20163
4
Navigation Structure Combination of linear and tree model navigation structure At some point students have no choice but follow the path constructed by the designer At other points it is for students to choose the direction of the instruction 2/18/20164
5
Database for Data Collection Collects data to track student’s behavior and performance A unique user ID is assigned when a student registers with the program The administrator assigns class codes to different grades in different schools in Iowa One can take the training independently under the class code GENERAL 2/18/20165
6
Target Audience Ages: 10 and 11 years old Program designed for 4 th grade students of public schools in Iowa Can also be used as a remedial activity for students up to sixth grade facing difficulty in math. 2/18/20166
7
Data Collection Process: Identifying Evaluation Questions Divergent Phase: Interviewing main stake holders to Gain an idea about expectations and concerns about the product Know the information they seek from the evaluation Know aspects they consider key to meeting the program’s goals. 21 concerns were identified: Concerns about Content Material and Instruction (6) Concerns about Learning (4) Concerns about Design and Navigation (11)
8
Evaluation Questions Convergent Phase: #1 Are participants able to navigate the program? Measures: ease of use by error rate while performing performance-based tasks #2 Are navigation instructions appropriate and sufficient to allow participants ease of navigation? Measures: Clarity of instructions for target audience and navigation errors caused by lack of instructions
9
Evaluation Questions #3 Does the vocabulary and language of lesson instructions address the target audience? Measures: Ability of participants to comprehend lesson instructions #4 How many of the performance tasks are the participants able to complete successfully? Measures: overall effectiveness of the program
10
Evaluation Questions #5 Do participants find the visual design appealing and does it meet their expectations? Measures: likeability and preference #6 How do participants feel about their experience with the program? Measures: satisfaction, likeability, ease of use, and areas of difficulty
11
Evaluative Instruments and Matrix of Objectives Think aloud protocol Performance based tasks Observation grid Debriefing Navigation report from program database
12
Participants Participants drawn from Target Audience : t wo 4 th graders, one 5 th grader, and one 6 th grader Computer experience: use computers both at home and at school for different purposes Language proficiency: near native speakers of English.
13
Process: Challenges and Modifications Think aloud protocol: Had them work in pairs and discuss amongst each other Microphone and Video Recording to gather subtle non-verbal information Observation grid – checklist, actions and behaviors Time to get used to testing area and equipment – snacks Background Questionnaire using interview protocol Debriefing using interview protocol and walk-through on specific problem pages
14
Data Analysis Process Data were typed and compiled to facilitate the analysis. The videos were watched and comments from the think aloud were typed. Data were summarized to address the evaluation questions. A qualitative analysis of the data was conducted. The triangulation of the findings was done. 2/18/201614
15
Findings Program navigation Participants were able to navigate the site with little to no help. ○ They could Register and login Understand error messages Figure out solutions to the problems ○ Intuitive set of buttons However… ○ They had problems with the Summary Page. ○ The 5 th /6 th graders had problems with the feedback of a question in the quiz. ○ They expected to interact more with the program. 2/18/201615
16
Findings (2) Navigation instructions Sometimes they did not know how to go about answering a question. Some of the instructions seemed to be too long or unclear (Participants yawned) Top-down perception error - Some examples given were too close to the answers –confused the 4 th Graders 2/18/201616
17
Findings (3) Vocabulary and language instruction Both pairs claimed to have had no problems reading and understanding the instructions. However… ○ The video recordings showed them having trouble to read some words such as characteristics and appropriate. ○ Reading the instructions again even more carefully, we found an example that does not make much sense to us. 2/18/201617
18
Findings (4) Task completion Both pairs completed all the tasks successfully 2/18/201618
19
Findings (5) Visual design All participants liked the program “very very cool” They also suggested changes: ○ Bigger and clearer pictures ○ Feedback could be easier to see ○ Back to Summary page button (4 th Graders) : Use brighter color Change language to ‘Back to score page’ ○ 5 th Graders – Give specific instructions to press ‘Back to Summary page button 2/18/201619
20
Findings (6) Participant experience The questions made them think. They liked the instructions. 4 th graders thought the quizzes were hard – 5 th /6 th graders liked them. Both pairs said they would like to see some drag and drop. 4 th graders suggested that the questions were categorized according to their level of difficulty. Non-verbal behavior showed participants’ interest. 2/18/201620
21
Recommendations Instructions Reduce the number of instructions and present them in an interactive way. Include a note on the first page telling people not to use the browser/keyboard buttons. Mention that it doesn’t matter if the letters are capitalized or not – quizzes and login pages. Mention that some questions might need the user to work out the fractions before they can see the characteristics or relationships. 2/18/201621
22
Recommendations Navigation Put an error message which gives people a chance to correct their navigation mistakes. Language Use simpler words (e.g., correct) instead of appropriate. Reword some of the sentences in the instruction’s pages. More interactivity Drag and drop Clickable items Space for typing tentative answers 2/18/201622
23
Recommendations Visual design Make pictures bigger and clearer. Make feedback easier to see or use a pop-up window. Make the previous/back buttons invisible in the Review Answers Page. Program Structure Make program less linear and closed – allow users to go back in quizzes and change answers or retake quiz in same user name. This will allow them to demonstrate cognitive change and learning 2/18/201623
24
Limitations of the Usability Testing We told them not to use the back and forward buttons. Observation grid could be improved. We might have lost important data because we had to replace a disk. The kids were highly motivated and computer savvy. The database did not record some of the events. 2/18/201624
25
Methodological Challenges of working with children Children do not remember specific things. Children may not understand some terms such as ‘drag and drop’, ‘feedback’, and ‘interactivity’. They might want to please the adult. Non-verbal gestures might reveal more than specific debriefing questions. Having them work in pairs is a great way to get their initial reactions. 2/18/201625
26
Recommendations for Further Testing Conduct the test separately with each pair of children. Consider having more children of different levels of linguistic proficiency (e.g., ESL learners and remedial children). Conduct a brief survey in schools to get a good user profile and feedback. 2/18/201626
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.