Presentation is loading. Please wait.

Presentation is loading. Please wait.

LOG O Development of a diagnostic system using a testing-based approach for strengthening student prior knowledge Computers & Education (September 2011)

Published byJerome Bell Modified over 9 years ago

Similar presentations

Presentation on theme: "LOG O Development of a diagnostic system using a testing-based approach for strengthening student prior knowledge Computers & Education (September 2011)"— Presentation transcript:

1 LOG O Development of a diagnostic system using a testing-based approach for strengthening student prior knowledge Computers & Education (September 2011) Yi-Chun Lin, Yen-Ting Lin, Yueh-Min Huang* Department of Engineering Science, National Cheng Kung University

2 Introduction  Assist instructors in diagnosing and strengthening students’ prior knowledge before new instructions and to enable students to attain greater learning motivation and improved learning performance  A testing-based diagnosis system is proposed in this study to cope with these problems

3 Methodology  To measure the strength of understanding of prior knowledge  Prior knowledge diagnosis (PKD) model is proposed  Two data sources:  Testing information assigned by teachers  Testing information derived by students Represents a relationship between each concept and test item in a test, and the relationships among the concepts Represents a relationship between student’s answers and the test items

4 Methodology  A course specifies n concepts  C 1, C 2, C 3,…, C i,… C m,…, C n  Prior knowledge of the subject for r participating students  S 1, S 2, S 3,…, S l,…, S r  Teacher select k test items from the test item bank to form the pre-test  T 1, T 2, T 3,…, T j,…, T k

5 Methodology  X mj indicates the degree of relevance between the m-th concept and the j-th test item  represent the degree of relevance between each concept and test item  Z im indicates the relationship between the ith and the mth concepts (ranged from 0 to 1)  represent the relationship between the concepts

6 Methodology - Strength of concept  The strength of concept C i in the pre-test  Z im represents the relationship between the i-th and the m-th concepts, 0 ≤ Z im ≤ 1  X mj indicates the degree of relevance between the m-th concept and the j-th test item, 0 ≤ X mj ≤ 1 0 ≤ S(C i ) ≤ nk

7 Methodology - Importance ratio of concept  The importance ratio of concept C i in the pre-test  Z im represents the relationship between the i-th and m-th concepts, 0 ≤ Z im ≤ 1  X mj indicates the relevance degree between the m-th concept and the j-th test item, 0 ≤ X mj ≤ 1 0 ≤ IRP(C i ) ≤ 1

8 Methodology - Understanding strength of the lth student  The understanding strength of the lth student on the ith concept  R lj indicates the answer of the l-th student on the j-th test item If the student answers the test item correctly, R lj is 1; otherwise R lj is 0  Z im represents the relationship between the i-th and the m-th concepts  X mj indicates the degree of relevance between the m-th concept and the j-th test item

9 Methodology - Understanding strength of the concept  Translate the importance ratio of the concept into the understanding strength of the concept  To t(C i ) represents the threshold value of the i-th concept, 0 ≤ t(C i ) ≤ 1  m indicates the gradient of the function, m = 1  IRP(C i ) represents the importance ratio of concept C i in the pre-test, 0 ≤ IRP(C i ) ≤ 1  b is the point at which the line crosses the y- axis, b = 0

10 Illustrative example Test itemConcept C1C2C3C4C5 T1 10000 T2 0.200.600.2 T3 01000 T4 00.400 T5 0000.20 Illustrative example of relationship between test items and concepts

11 Illustrative example Concept C1C2C3C4C5 C1100.400 C2010.600.2 C30.40.610.20.6 C4000.210.4 C500.20.60.41 Illustrative example of relationship between concepts

12 Strength of Concept  The strength of the first concept (C 1 ):  S(C 1 ) = Z 11 *X 11 + Z 11 * X 12 + Z 13 * X 32 = 1.0 × 1.0 + 1.0 × 0.2 + 0.4 × 0.6 = 1.44  S(C 2 ) = Z 21 *X 23 + Z 21 * X 24 + Z 23 * X 32 + Z 25 * X 52 + Z 25 * X 54 = 1.0*1.0 + 1.0*0.4 + 0.6*0.6 + 0.2*0.2 + 0.2*0.4 = 1.88 Test item Concept C1C2C3C4C5 T1 10000 T2 0.200.600.2 T3 01000 T4 00.400 T5 0000.20 Concept C1C2C3C4C5 C1100.400 C2010.600.2 C30.40.610.20.6 C4000.210.4 C500.20.60.41

13 Importance Ratio of Concept Importance ratio of concept Concept C1C2C3C4C5 IPR0.190.250.310.070.18  The importance ratio of second concept (C 2 )  IRP(C 2 ) = 1.88 / 7.52 = 0.25

14 Illustrative example Test item Student S1S2S3S4S5 T111011 T210111 T300001 T401011 T500100 Illustrative example of the relationship between students’ answers and test items

15 Relationship between students’ answers and test items (R) Illustrative example Student Concept C1C2C3C4C5 S110.210.520.360.42 S20.690.260.380.290.36 S30.310.210.360.710.48 S410.470.720.640.79 S5110.980.640.94 Relationship between students’ understanding strength and concepts (USS) Test item Student S1S2S3S4S5 I111011 I210111 I300001 I401011 I500100 Test item Concept C1C2C3C4C5 I1 10000 I2 0.200.600.2 I3 01000 I4 00.400 I5 0000.20 Concept C1C2C3C4C5 C1100.400 C2010.600.2 C30.40.610.20.6 C4000.210.4 C500.20.60.41 Relationship between concepts (Z) Relationship between test items and concepts (X) R 41 Z 51 X 11 R 41 Z 52 X 21 R 41 Z 53 X 31 R 41 Z 54 X 41 R 41 Z 55 X 51 R 42 Z 51 X 12 R 42 Z 52 X 22 R 42 Z 53 X 32 R 42 Z 54 X 42 R 42 Z 55 X 52 R 43 Z 51 X 13 R 43 Z 52 X 23 R 43 Z 53 X 33 R 43 Z 54 X 43 R 43 Z 55 X 53 R 44 Z 51 X 14 R 44 Z 52 X 24 R 44 Z 53 X 34 R 44 Z 54 X 44 R 44 Z 55 X 54 R 45 Z 51 X 15 R 45 Z 52 X 25 R 45 Z 53 X 35 R 45 Z 54 X 45 R 45 Z 55 X 55 1.0 x 0.0 x 1.01.0 x 0.2 x 0.01.0 x 0.6 x 0.01.0 x 0.4 x 0.01.0 x 1.0 x 0.0 1.0 x 0.0 x 0.21.0 x 0.2 x 0.01.0 x 0.6 x 0.61.0 x 0.4 x 0.01.0 x 1.0 x 0.2 1.0 x 0.0 x 0.01.0 x 0.2 x 1.01.0 x 0.6 x 0.01.0 x 0.4 x 0.01.0 x 1.0 x 0.0 1.0 x 0.0 x 0.01.0 x 0.2 x 0.41.0 x 0.6 x 0.01.0 x 0.4 x 0.01.0 x 1.0 x 0.4 1.0 x 0.0 x 0.01.0 x 0.2 x 0.01.0 x 0.6 x 0.01.0 x 0.4 x 0.21.0 x 1.0 x 0.0 1.0 x 0.0 x 1.01.0 x 0.2 x 0.01.0 x 0.6 x 0.01.0 x 0.4 x 0.01.0 x 1.0 x 0.0 1.0 x 0.0 x 0.21.0 x 0.2 x 0.01.0 x 0.6 x 0.61.0 x 0.4 x 0.01.0 x 1.0 x 0.2 0.0 x 0.0 x 0.00.0 x 0.2 x 1.00.0 x 0.6 x 0.00.0 x 0.4 x 0.00.0 x 1.0 x 0.0 1.0 x 0.0 x 0.01.0 x 0.2 x 0.41.0 x 0.6 x 0.01.0 x 0.4 x 0.01.0 x 1.0 x 0.4 0.0 x 0.0 x 0.00.0 x 0.2 x 0.00.0 x 0.6 x 0.00.0 x 0.4 x 0.20.0 x 1.0 x 0.0

16 Illustrative example Threshold value of concept Concept C1C2C3C4C5 IPR0.190.250.310.070.18 *m = 1, b = 0 for threshold function in this case

17 PKT&D System Architecture

18 Experiment  Participants :  A course instructor  80 university students  Course: bioinformatics  Group:  Control group: 40 students (used the PKT&D system)  Experiment group: 40 students (did not use the PKT&D system)

19 Experiment Subject: sequence analysis approaches and tools Concepts in prior knowledge: sequence characteristics and structures, statistical hypothesis testing, and formula expression format UnitInstruction activitiesTime (min) Understanding the importance of similarity 1.A series of guided questions (5) 2.Slide presentation (15) 3.Discussions (10) 30 Introduction to the most popular data-mining tool: BLAST 1.A series of guided questions (5) 2.Slide presentation (15) 3.Practice (10) 30 BLASTing protein sequences 1.A series of guided questions (5) 2.Slide presentation (10) 3.Practice (15) 30 Understanding BLAST output 1.Slide presentation (15) 2.Discussions (15) 30 BLASTing DNA sequences 1.A series of guided questions (5) 2.Slide presentation (10) 3.Practice(15) 30 The BLAST way of doing things 1.Slide presentation (15) 2.Practice (15) 30

20 Experiment Process

21 The learning motivation post-test score between the two groups Group Number of students MeanS.D. Adjusted mean F(1, 77)P-value Experimental group 4046.453.4246.41116.340.00* Control group 4041.057.5941.086 Total number of students 8043.756.38  To measure the students’ learning motivation, Motivated Strategies for Learning Questionnaire was adopted in this study.  using nine questionnaire items and a seven-point Likert scale

22 The paired t-test results of learning motivation for the two groups of students GroupTestsNMeanS.D.t(39) Experiment al group Pre-test4034.854.451-9.688* Post-test4046.453.425 Control group Pre-test4033.19.072-3.074* Post-test4041.057.591

23 Students’ attitude towards bioinformatics learning #Item Experiment Group (Mean, S.D.) Control Group (Mean, S.D.) t-value 1I like learning bioinformatics5.45/0.875.05/1.081.82 2 The bioinformatics learning activities are helpful 5.68/0.765.35/0.831.82 3 I like to practice using the software in the bioinformatics learning 5.75/0.875.53/0.881.15 4 I had enough ability to learn the bioinformatics material 5.28/0.994.38/1.443.25* 5 I can meet the instructor’s requirements during the bioinformatics learning process 5.38/0.904.40/1.523.50* 6 I can understand the bioinformatics material taught by the instructor 5.45/0.854.55/1.553.22*

24 Experiment group students’ perceptions of using the PKT&D system #QuestionEU(%)QU(%)SU(%) Neither(%)SL(%)QL(%)EL(%)Mean 1 Using the PKT&D system in learning bioinformatics would enable me to diagnose and strengthen prior knowledge more effectively 057.5527.532.522.55.43 2 Using the PKT&D system would improve my bioinformatics learning performance 0052030 155.30 3 Using the PKT&D system in learning bioinformatics would increase my learning comprehension productivity 002.512.542.53012.55.38 4 Using the PKT&D system would make it easier to learn bioinformatics 02.57.51037.5 55.15 5 I would find the PKT&D system useful in the bioinformatics class 0001042.53512.55.50 Note: EU: Extremely Unlikely; QU: Quite Unlikely; SU: Slightly Unlikely; SL: Slightly Likely; QL: Quite Likely; EL: Extremely Likely.

25 Independent pre-test on knowledge of bioinformatics of the two groups Variable Pre-test t-value NMeanS.D. Experiment group4050.2514.32-0.385 Control group4051.5014.77

26 The paired t-test results of the learning improvement for the two groups GroupTestsNMeanS.D.t-value Experimental group Pre-test4050.2514.23-8.460* Post-test4069.5011.54 Control group Pre-test4051.5014.77-9.595* Post-test4063.5013.12 * p<0.05.

27 Independent Post-test on knowledge of bioinformatics of the two groups Variable Post-test t-value Number of students MeanS.D. Experimental group 4069.5011.542.172* Control group4063.5013.12

28 Example interview comments about the three topics Inductive topics PerspectivesIntervieweesTranscript sample of interview comments Instruction Realization of students IN I felt that the students in the experiment group better met my requirements during the course. Achievement of students IN I felt that the students in the experiment group demonstrated high performance in each learning activity. Progress of instruction IN I had much more time to teach the concepts in more detail and interact with the students of experiment group. Learning situationIN Only one third of the students from the control group could fully follow the activities and understand my instructions. SC & SE Learning about bioinformatics software through practicing using it was interesting. I would have preferred more direct instruction from the course instructor. SC I felt some of the concepts were difficult to grasp, which led to obstacles when I used the software during the course. Interaction Discussion willingness IN The students in the experiment group had better discussions than the control group. Responsiveness of students IN The students in the experiment group often gave feedback and asked questions. Technology System usefulness SE I felt the user interface of the PKT&D system was clear, straightforward, and convenient to use..I clearly saw the diagnostic results and learning suggestions. Auxiliary components IN & SE I felt that the PKT&D system served as a guide that helps the students to diagnose the weakness of their concepts. I can learn more prior knowledge using the PKT&D system.

29 Evaluation of correctness rate results for the three concepts diagnosis

30 Student ID12345678 Correctness rate of diagnoses Expert 1100% 66.60%100% Expert 2100%66.60%100%66.60%100% Student ID910111213141516 Correctness rate of diagnoses Expert 166.60%100%66.60%100% Expert 2100% 33.30%100% Student ID1718192021222324 Correctness rate of diagnoses Expert 1100% Expert 2100% 66.60%100% Student ID2526272829303132 Correctness rate of diagnoses Expert 1100%66.60%33.30%100% 66.60% Expert 2100% 66.60%100% Student ID3334353637383940 Correctness rate of diagnoses Expert 1100% 33.30%100%33.30%100% Expert 2100% 33.30%100%

31 Evaluation of correctness rate results for the five concepts diagnosis Student ID12345678 Correctness rate of diagnoses Expert 180%100% 80%100% 80%100% Expert 2100% 80%100% Student ID910111213141516 Correctness rate of diagnoses Expert 1100% 80%100%80%100%60% Expert 2100% 80%100%40% Student ID1718192021222324 Correctness rate of diagnoses Expert 1100%60%100%60%100% Expert 260%40%80%60%100%80%100% Student ID2526272829303132 Correctness rate of diagnoses Expert 140%80%100% Expert 2100%80%100% 80%100% Note: Correctness rate obtained by comparing the diagnoses of the experts with those obtained using the proposed approach to artificial intelligence course. The average correctness rates are 90.625% and 90% for the students.

32 Conclusion & Discussion  Propose a testing-based approach to diagnose the strength of individual students’ prior knowledge of concepts, and then provide them with appropriate materials to strengthen this  Provide instructors can undertake their teaching more smoothly  Educators can use the proposed system in different educational contexts

33 Limitation  Two variables of the linear function have to be adjusted based on instructors’ expertise in different educational contexts

34 Future Work  The number of test items in the item bank should be continually increased to address various subject objectives and the instructors’ needs  Students’ learning portfolio can be integrated into the proposed system to develop more appropriate diagnosis mechanisms

Download ppt "LOG O Development of a diagnostic system using a testing-based approach for strengthening student prior knowledge Computers & Education (September 2011)"

Similar presentations

Assessing Student Performance

Assessing Student Performance
Importance of Questioning and Feedback Technique in developing 3 Cs www.schoolofeducators.com.

Importance of Questioning and Feedback Technique in developing 3 Cs
Using the IDEA Student Ratings System: An Introduction University of Saint Thomas Fall 2012 www.theideacenter.org.

Using the IDEA Student Ratings System: An Introduction University of Saint Thomas Fall
Presenter: Han, Yi-Ti Adviser: Chen, Ming-Puu Date: Jan 19, 2009 Sitthiworachart, J. & Joy, M.(2008). Computer support of effective peer assessment in.

Presenter: Han, Yi-Ti Adviser: Chen, Ming-Puu Date: Jan 19, 2009 Sitthiworachart, J. & Joy, M.(2008). Computer support of effective peer assessment in.
The world is becoming more and more technologically oriented. Hence, every educator should be modern with their outlook in teaching. The use of multimedia.

The world is becoming more and more technologically oriented. Hence, every educator should be modern with their outlook in teaching. The use of multimedia.
Effect Size and Meta-Analysis

Effect Size and Meta-Analysis
Dr. Hari Singh School of Business Humboldt State University.

Dr. Hari Singh School of Business Humboldt State University.
Specialized Understanding of Mathematics: A Study of Prospective Elementary Teachers Meg Moss.

Specialized Understanding of Mathematics: A Study of Prospective Elementary Teachers Meg Moss.
Critiquing Research Articles For important and highly relevant articles: 1. Introduce the study, say how it exemplifies the point you are discussing 2.

Critiquing Research Articles For important and highly relevant articles: 1. Introduce the study, say how it exemplifies the point you are discussing 2.
Characteristics of on-line formation courses. Criteria for their pedagogical evaluation Catalina Martínez Mediano, Department of Research Methods and Diagnosis.

Characteristics of on-line formation courses. Criteria for their pedagogical evaluation Catalina Martínez Mediano, Department of Research Methods and Diagnosis.
Student Assessment 2014-2015 CERRA National Board Candidate Support Workshop Toolkit WS5 2014.

Student Assessment CERRA National Board Candidate Support Workshop Toolkit WS
Comparison of an Integrated (HBSE/Practice) Blended Learning Course with Non-integrated Face-to-Face Courses Rose McCleary Leigh Collins Sam Jenkins California.

Comparison of an Integrated (HBSE/Practice) Blended Learning Course with Non-integrated Face-to-Face Courses Rose McCleary Leigh Collins Sam Jenkins California.
ISEM 3120 Seminar in ISEM Semester

ISEM 3120 Seminar in ISEM Semester
Assessment Assessment involves the sampling of some aspect of a person's learning/knowledge at a particular moment. Depending upon the kind of sample.

Assessment Assessment involves the sampling of some aspect of a person's learning/knowledge at a particular moment. Depending upon the kind of sample.
The Effectiveness of Supplemental Online vs. Traditional Tutorials on Students’ English Proficiency and Learning Achievement Ponlak Pantahachart Faculty.

The Effectiveness of Supplemental Online vs. Traditional Tutorials on Students’ English Proficiency and Learning Achievement Ponlak Pantahachart Faculty.
Quantitative Research

Quantitative Research
Robert delMas (Univ. of Minnesota, USA) Ann Ooms (Kingston College, UK) Joan Garfield (Univ. of Minnesota, USA) Beth Chance (Cal Poly State Univ., USA)

Robert delMas (Univ. of Minnesota, USA) Ann Ooms (Kingston College, UK) Joan Garfield (Univ. of Minnesota, USA) Beth Chance (Cal Poly State Univ., USA)
The Impact of On-line Teaching Practices On Young EFL Learners' Instruction Dr. Trisevgeni Liontou RHODES MAY 28-31 2015.

The Impact of On-line Teaching Practices On Young EFL Learners' Instruction Dr. Trisevgeni Liontou RHODES MAY
Factors affecting contractors’ risk attitudes in construction projects: Case study from China 2011. 12. 13 박병권.

Factors affecting contractors’ risk attitudes in construction projects: Case study from China 박병권.
Animation Based Learning of Electronic Devices: Practical Engineering Student Case A. Gero, W. Zoabi & N. Sabag Technion – Israel Institute of Technology.

Animation Based Learning of Electronic Devices: Practical Engineering Student Case A. Gero, W. Zoabi & N. Sabag Technion – Israel Institute of Technology.

Similar presentations

Ads by Google