1. By: ODUNTAN, ODUNAYO ESTHER AAA110007 Ph.D Qualifying Examination
DEVELOPMENT OF AN AUTOMATED ESSAY-TYPE GRADING SYSTEM USING MODIFIED PRINCIPAL COMPONENT ANALYSIS
By:
ODUNTAN, ODUNAYO ESTHER
AAA110007
Ph.D Qualifying Examination
Supervised by:
Prof. S.O. Olabiyisi Prof. E.O. Omidiora Dr. I.A. Adeyanju
Supervisor Co-supervisor Co-supervisor

2. Introduction What is Assessment ? Assessment Processes Testing Grading
Objective Mode
Multiple Choice Question (MCQ)
Multiple Choice Multiple Answer(MCMA)
Theory Mode
Essay
Essay-Type

3. Problem Statement Methods of Grading Existing Grading systems
Manual
Time Consuming & Human factor such as biasness
Automated
Eliminates physical stress & Ensures fairness to all student.
Existing Grading systems
Objectives and Essays assessment
Challenges
polysemy and word order
Integrating n-gram into Existing Principal Component Analysis for Automated Essay-type Grading.

4. Aim and Objectives of Study
To develop an automated essay-type grading system using generalized principal component analysis.
Objectives
Comparatively analyze keywords and n-gram document representation in automated essay-type grading.
Modify existing principal component analysis algorithm by integrating n-grams.
Design an automated essay-type grading system using modified principal component analysis
Develop (iii) using Matrix Laboratory (MATLAB)
Evaluate the performance of the developed system using Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), Pearson Correlation Coefficients (R), and Coefficient of determination (R2).
Scope of Study
Essay-type answers to exam questions
Textual in nature

5. Related Work S/N System Technique Dataset Extracted Features
Document Representation 1
PEG
(Page,1996)
Pattern Matching
Essay
Liquistic features such as proxes and trins
Standard regression equation 2
E-Rater
(Burstein et al., 2001)
Information Retrieval and Natural Language Processing
Discourse structure, syntactic structure and vocabulary usage
Regression Analysis 3
BETSY
(Rudner and Liang,2002)
Text classification and Multivariant Bernoulli Model
Content and styles
Probabilistic 4
Automark System
(Mitchell, 2002)
Information Extraction with Natural Language Processing
Errors in spellings, syntax and semantics of essays
Vector Space Model 5
IntelliMetric
(Shermis et al. 2003)
Artificial Intelligence using neuro-synthetic approach
Semantic, syntactic and discourse level features of an essay 6
Automated Essay Assessor
(Kakkonen, 2004).
Natural Language Processing
Syntactic and constraint grammar parser 7
Intelligent Essay Grader
(Kakkonen et al., 2005)
Latent Semantic Analysis
Content, style, mechanics, coherency 8
Expert System for AES
Ade-Ibijola, A.O., (Wakama, I, Amadi, J.C.2012)
Shallow Natural language Techniques
Free -Text Answers
Content
Fuzzy Scoring Model 9
Automated Essay Grading using GLSA
(Islam and Hoque 2012)
Generalized Latent Semantic Analysis

6. Methodology
Figure 3.2: FRAMEWORK OF AUTOMATED ESSAY-TYPE GRADING SYSTEM USING MPCA

7. N-gram Document Representation
Algebraic model for representing textual documents
dj denotes the jth document
wij denotes the weighting of the ith term in the jth document
Document term matrix for Essay-type grading
A document is an essay-type answer to an exam question
A term is typically a keyword in the document
But we generalize a term as a n-gram
subsequence of n words from a document
e.g. n = 2 is called a bigram (2-gram)

8. N-gram Document Representation
Figure 3.2: Text Processing with Vector Space Model

9. Modified PCA for Feature Extraction
Standard PCA for text processing
Uses keyword document representation
“Colorless green ideas sleep furiously”
contains five keywords:
“colorless”, “green”, “ideas”, “sleep”, and “furiously”,
Keywords are sometimes not very informative
Polysemy and word order not captured
Modified PCA uses n-gram(n>= 2)
contains four equivalent bigrams:
“Colorless green”, “green ideas”, “ideas sleep”, “sleep furiously”
Bigrams are more informative
Polysemy and word order addressed
Each n-gram represents a term in the Document Term Matrix

10. Modified Principal Component Analysis Algorithm(MPCA)
Step1:
Input n-gram from document vector(n=2,3…)
Step 2:
Subtract the mean from the data dimension
Step 3:
Calculate the covariance matrix
Step 4:
Calculate the eigenvalue and eigenvector of the covariance
Step 5:
Choose Highest components to form a normalized document vector.

11. Modified Principal Component Analysis Algorithm(MPCA)
In Text Processing;
For MPCA, Terms are n-grams (n>= 2)
Form a Spanned Matrix
Subtracting off mean of each dimension
Compute Covariance of the spanned matrix
b refers to number of n-gram terms, AT is the transpose of spanned matrix
Derive the Eigen Space
Calculate eigenvalue and arrange in descending order
Choose the highest eigenvalue
Derive the eigenvector
Derive the Normalised Document Vector
Project the Eigen space into the spanned matrix
G = A * ET * E
G is Normalised Document Vector, A is the spanned document term matrix,
ET is the transpose of the Eigen space known as the resulting vector.

12. Documents Similarities
Documents similarities compares two separate documents.:
Normalised document vector
pre-processed marking scheme
pre-processed students’ answers.
Cosine Similarity Measure
dij denotes the weight of the ith term in the essay-type marking scheme document term matrix
(Dj) and qi denotes the weight of the ith term in the essay-type student answer document term matrix(Q).

13. Performance Measure Compares human score with machine score using:
Mean Absolute Error(MAE).
Root Mean Squared Error(RMSE)
Pearson coefficient correlation(R).
Coefficient of Determination (R2 )
Mean Absolute Error formular:
Root Mean Squared Error (RMSE)
Xi is the human score
Yi is the machine score
n is the number of data analysed
i is the mean of value actually observed
yi is the value actually observed
n is the number of data

14. Performance Measure Contd.,
Pearson Correlation Coefficient (r)
X is the human score
Y is the machine score
N is the number of data analysed
Coefficient of determination (R2)
is the actual value of the original assessor
is the predicted value by the developed system
is the mean of the actual value
is the mean of the predicted value

15. Experimental Design Data acquisition: Experimental methodology
Department of computer science, Federal Polytechnic Ilaro, Ogun State
COM 317 and COM 325 Data sets
One Marking Scheme each and 35 Students’ Answers.
Experimental methodology
Text pre-processing
Document Representation: keyword and bigram
Document Similarities: cosine similarity measure.
Experimental Analysis
Keyword System
Ngram (n=2) System
Principal Component Analysis(PCA)
Modified Principal Component Analysis(MPCA)
Development Tool used :
MATLAB (Matrix Laboratory)

16. Implementation
Figure 4.1:Screenshot of a GUI of the Automated Essay Type Grading System

17. Figure 4.2:Screenshot of a Student’s response in .txt file format
Data Acquisition
Figure 4.2:Screenshot of a Student’s response in .txt file format

18. Figure 4.3:Screenshot of Pre-processed Text
Text Pre-processing
Figure 4.3:Screenshot of Pre-processed Text

19. N-Gram Representation of Text
'research'
'process'
'arriving'
'dependable'
'methodical'
Figure 4.4a:Screenshot of Keyword Document Representation

20. N-Gram Representation of Text
'research process '
' process arriving'
' arriving dependable'
' dependable solutions'
' solutions problems'
' problems planned'
' planned systematic' …
Figure 4.4b:Screenshot of N-gram Document Representation

21. Mean Absolute Error Results
Bigram has a lower estimate
MPCA System has the least MAE value.
 Dataset
Keyword System
Bigram System
PCA System
GPCA System
COM 317
18.59
12.16
5.67
4.4
COM 325
16.5
11.5
10.22
5.04

22. Mean Absolute Error Chart
MPCA System has the least MAE value.

23. Root Mean Squared Error Results
KEYWORD
BIGRAM
PCA
MPCA
COM 317
30.50
27.50
22.80
21.80
COM 325
28.50
22.50
25.20
24.20
MPCA System has the least RMSE value.

24. Root Mean Squared Error Chart
MPCA System has the least RMSE value.

25. Pearson Correlation Results
 Dataset
Keyword System
Bigram System
PCA System
GPCA System
COM 317
0.25
0.42
0.45
0.70
COM 325
0.3
0.50
0.75
MPCA has the highest correlation coefficient

26. Pearson Correlation Results Chart
Bigrams and MPCA System shows better correlation when evaluated with the Assessor.

27. Coefficient of determination Results
KEYWORD
BIGRAM
PCA
MPCA
COM 317
0.15
0.45
0.20
0.50
COM 325
0.10
0.40
0.25
0.60
Bigram and MPCA System shows better correlation when evaluated with the Assessor.

28. Coefficient of determination Chart
Bigram and MPCA System shows better correlation when evaluated with the Assessor.

29. Time and Memory Utilization Results
Metric
Keyword
Bigram
PCA
MPCA
Time (secs)
35.251
25.456
28.991
10.582
Memory Usage(kb)
15.8
7.58
13.9
3.9
GPCA is the fastest and occupies the least memory space

30. Time and Memory Utilization Chart
MPCA is the fastest and occupies the least memory space

31. Performance measure of MPCA with Existing State–of-Art (GLSA)
Generalized Latent Semantic Analysis(GLSA)
Developed by Islam and Hogue(2012)
Latent Semantic Analyser is for Indexing Terms
Focused on Essay Dataset
Used Term Frequency Scheme
Modified Principal Component Analysis(MPCA)
Uses N-gram in Document Representation
Principal Component Analysis is dimensionality reduction Technique (Omidiora, 2006)
Focuses on Essay-Type Dataset
Binary Weighting Scheme

32. Mean Absolute Error Results for GLSA and GPCA
MPCA System
GLSA
COM 317
4.4
5.24
COM 325
5.84
7.55
MPCA has lower estimate

33. Mean Absolute Error Chart for GLSA and GPCA
MPCA has lower estimate

34. Pearson Correlation Result for GLSA and MPCA
MPCA System
GLSA
COM 317
0.7
0.4
COM 325
0.75
0.45
MPCA has higher correlation

35. Pearson Correlation Chart for GLSA and MPCA
MPCA has higher correlation

36. Coefficient of Determination Chart for GLSA and MPCA
MPCA has better accuracy

37. Contributions to Knowledge
Comparatively analyse Keyword and N-gram Document Representation
Apply n-gram document representation into PCA to derive MPCA for automated essay-type graders.
This have addressed the issue of polysemy.
Perform MPCA Feature Extraction
flexibility in word sequencing.
Develop and evaluate an Automated Essay-Type Grading System
high positive correlation
low mean absolute error
with the original assessor and the existing state-of-art (GLSA).

38. Conclusion A high positive correlation with the original Assessor
The developed Automated Essay-Type Grading System using Modified Principal Component system(MPCA) had:
A high positive correlation with the original Assessor
Lower Mean Absolute Error