Designing Electronic Performance Support Systems to Facilitate Learning PSYC 512 October 20, 2005 Christina Abbott

Citation Van Schaik, P., Pearson, R & Barker, P., Designing electronic performance support systems to facilitate learning. Innovations in Education & Teaching International, 39(4),

Why This Article Short Short Has some glaring problems Has some glaring problems Relates to terms we know Relates to terms we know Provides a decision-making tree for descriptive and inferential statistics. Provides a decision-making tree for descriptive and inferential statistics.

What an EPSS Should Do? Shiak, Pearson, and Barker state that an EPSS 1 facility designed for educational need should: Shiak, Pearson, and Barker state that an EPSS 1 facility designed for educational need should: Assess skill – fill in the gaps in performance Assess skill – fill in the gaps in performance Act as a transfer agent – by using on- demand mechanisms. Act as a transfer agent – by using on- demand mechanisms. Act as an augmentation aid – improve performance over and above natural ability Act as an augmentation aid – improve performance over and above natural ability 1 EPSS = Electronic Performance Support System

Design of This EPSS Goal: To design a learning tool for psychology students that aids in their understanding of various statistical methods. Goal: To design a learning tool for psychology students that aids in their understanding of various statistical methods. Step 1 – Needs Analysis Step 1 – Needs Analysis Step 2 – Design and Development Step 2 – Design and Development Database Database Help System Help System Advisor System Advisor System Personal Area and Notes Personal Area and Notes

Advisor System The Advisor System is a series of decision- making trees. The one below is for descriptive statistics. The Advisor System is a series of decision- making trees. The one below is for descriptive statistics.

Advisor System The decision-making tree below is for inferential statistics. The decision-making tree below is for inferential statistics.

Purpose There were three main purposes of the study: There were three main purposes of the study: Analyze students’ knowledge of concepts in quantitative research methods. Analyze students’ knowledge of concepts in quantitative research methods. Establish students’ performance with system. Establish students’ performance with system. Establish students’ overall acceptance of the system as a learning aid. Establish students’ overall acceptance of the system as a learning aid.

Hypothesis The researchers could have increased the criticality of the study by including a hypothesis for each of the above purposes. The researchers could have increased the criticality of the study by including a hypothesis for each of the above purposes. For example: For example: Researchers could have developed a hypothesis for the performance of a group who did not use the EPSS to answer knowledge questions (control group) with a group that has full access to the system. Researchers could have developed a hypothesis for the performance of a group who did not use the EPSS to answer knowledge questions (control group) with a group that has full access to the system.

Research Design Participants Participants Sample of 89 sophomore psychology students Sample of 89 sophomore psychology students Students had taken two research classes in their first year of study Students had taken two research classes in their first year of study Three parts to the research Three parts to the research Knowledge of Concepts in Quantitative Research Methods Knowledge of Concepts in Quantitative Research Methods Use of the EPSS Use of the EPSS System Acceptance System Acceptance

Phase 1 - Knowledge of Concepts Independent Variables – the test questions (nominal) Independent Variables – the test questions (nominal) Dependent Variables – the answers (ordinal) Dependent Variables – the answers (ordinal) Confounds Confounds confidence level with computers and prior knowledge of statistical methods confidence level with computers and prior knowledge of statistical methods There was no control group specified, i.e. one that answered the questions without using the EPSS. There was no control group specified, i.e. one that answered the questions without using the EPSS. Could have made the study a within subjects design and measured their knowledge before and after using the EPSS. Could have made the study a within subjects design and measured their knowledge before and after using the EPSS.

Phase 2 – Use of the EPSS Independent Variables – the test questions (nominal) Independent Variables – the test questions (nominal) Dependent Variables – the answers (ordinal) (either correct or found) Dependent Variables – the answers (ordinal) (either correct or found) Confounds Confounds confidence level with computers and prior knowledge of statistical methods were confounds. confidence level with computers and prior knowledge of statistical methods were confounds. there was no control group specified, i.e. one that answered the questions without using the Advisor, Help, and Tools. there was no control group specified, i.e. one that answered the questions without using the Advisor, Help, and Tools. Could have made the study a within subjects design and measured their knowledge before and after using the tool. Could have made the study a within subjects design and measured their knowledge before and after using the tool.

Phase 3 – System Acceptance Survey was a 7 point Lickert Scale with values ranging from strongly agree to strongly disagree. Survey was a 7 point Lickert Scale with values ranging from strongly agree to strongly disagree. Independent Variables – the survey questions (nominal) Independent Variables – the survey questions (nominal) Dependent Variables – the answers (ordinal) Dependent Variables – the answers (ordinal) Confounds Confounds The students success in Phase 1 and 2 may contribute to their overall system acceptance. The students success in Phase 1 and 2 may contribute to their overall system acceptance. In previous studies I have done, when the user is asked to rate the performance of a system, they are really rating his or her success with the system. These users had low expectations of the system to begin with and expected to struggle. Satisfaction level self-reports can be misleading. In previous studies I have done, when the user is asked to rate the performance of a system, they are really rating his or her success with the system. These users had low expectations of the system to begin with and expected to struggle. Satisfaction level self-reports can be misleading. There are more probing questions the researchers could have asked, such as “where did you get stuck” and “when were you lead astray by the system.” There are more probing questions the researchers could have asked, such as “where did you get stuck” and “when were you lead astray by the system.”

Hypothesis Implications The null hypothesis was not stated at the beginning of the study. The null hypothesis was not stated at the beginning of the study. Discussion Section Statements implied that the researchers, though misguided, may have had a hypothesis in mind. Discussion Section Statements implied that the researchers, though misguided, may have had a hypothesis in mind. “The significant correlations between acceptance measures confirmed the association between intention of use and perceived usefulness found in previous research.” “The significant correlations between acceptance measures confirmed the association between intention of use and perceived usefulness found in previous research.” The researchers also cited a correlation, but did not show the data to back up their claim that: The researchers also cited a correlation, but did not show the data to back up their claim that: “Overall student’s performance using the EPSS was higher than unaided performance [control group] on knowledge questions.” “Overall student’s performance using the EPSS was higher than unaided performance [control group] on knowledge questions.” The researchers could have easily correlated the relationship between performance and confidence, as they had already gathered the data. The researchers could have easily correlated the relationship between performance and confidence, as they had already gathered the data.

Results There was scope for improving student’s performance. There was scope for improving student’s performance. Students performance using the EPSS was high. Students performance using the EPSS was high. Students performance using the EPSS was higher than unaided performance on knowledge questions. Students performance using the EPSS was higher than unaided performance on knowledge questions. Non-significant correlation between student’s performance in terms of knowledge of concepts and success in using the EPSS. Non-significant correlation between student’s performance in terms of knowledge of concepts and success in using the EPSS. Students expected that the EPSS would be useful to them for learning, revision, and completing assignments. Students expected that the EPSS would be useful to them for learning, revision, and completing assignments.

Parting Thoughts Ways to improve this study: Ways to improve this study: State the hypothesis clearly. State the hypothesis clearly. Have multiple hypothesis to aid in the criticality of study. Have multiple hypothesis to aid in the criticality of study. Correlate performance with acceptance – this is more powerful than each alone. Correlate performance with acceptance – this is more powerful than each alone. Ways to improve the article: Ways to improve the article: Do not use the term - confirm. Do not use the term - confirm. State all aspects of the research design - include any control group information. State all aspects of the research design - include any control group information. Median measure used for Lickert Scale Data Median measure used for Lickert Scale Data