1. USING SOFTWARE METRICS IN EDUCATIONAL ENVIRONMENT Ivan Pribela, Zoran Budimac, Gordana Rakić

2. Content Motivation Assessment process A case study Conclusion

3. Motivation Automated assessment systems are very helpful – Reduce load for instructors – Quicker feedback for students But have limited scope – Black-box (input-output) tests, unit tests – Lately code style checks Software metrics are not used – Utilized only by plagiarism detection – Not to help students improve their skills

4. Motivation (cont.) Software metrics can tell a lot about a program – Have highly diverse methodologies and objectives – Difficult to define quality for various programs in uniform way – What which metric means for particular assignment? – What metrics matter the most? We explore the usage of software metrics in automated assessment – To cover testing of aspects that demand instructor attention – Algorithm complexity – Number and size of programming units and functions

5. Assessment process Student solution Test report

6. Student solution MODULE Triplets1;... VAR x, y, z: INTEGER; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END Triplets1. Student solution

7. Parsing the solution eCST generator (the SMIILE tool) – Recognizes input language – Calls appropriate scanner & parser – Generates an eCST representation Enriched Concrete Syntax Tree – Modified Concrete Syntax Tree – Enriched with marker nodes Unit: class, module… Loop Statement: for, while, repeat… Branch Statement: if, case, switch… – Independent of programming language – Basis for calculation of software metrics Student solution eCST Parse

8. Measuring eCST Metrics calculator (the SMIILE tool) – Software metrics tool in the development – Analyzes the eCST representation – Calculates software metrics – Exports results in XML file Final result is one XML file with values of all calculated metrics Current prototype of SMIILE tool supports several software metrics – Lines of Code – Cyclomatic Complexity – … Student solution Metric values in xml eCST Parse Measure

9. Metric values in XML <sourceElement endLine="24" firstLine="1" name="" type=""> <sourceElement endLine="24" firstLine="1" name="Triplets1" type="CONCRETE_UNIT_DECL"> <sourceElement endLine="23" firstLine="9" name="FOR" type="LOOP_STATEMENT"> <sourceElement endLine="22" firstLine="10" name="FOR" type="LOOP_STATEMENT"> <sourceElement endLine="21" firstLine="11" name="FOR" type="LOOP_STATEMENT">... Student solution Metric values in xml eCST Parse Measure

10. Transform XML to properties XSL Transformator – Uses XSLT stylesheet Input XML file – Values of calculated software metrics Output XML file – Can be manipulated easily inside the testing system Student solution Metric values in xml eCST Matric values as properties Parse Measure Transform

11. Metric values as properties 24 4 Student solution Metric values in xml eCST Matric values as properties Parse Measure Transform

12. Testing the values Testing system (Testovid) – Implemented as a framework for running domain specific testers Domain specific testers – Written as Apache Ant scripts Using software metrics – Script runs SMIILE tool – Transforms XML file with metrics values – Loads calculated values – Freely uses them for grading, intelligent advice generation… Final report – Contains advices – Success/failure information – Presented to the student Student solution Metric values in xml Script with metric control values eCST Matric values as properties Test report Parse Measure Transform Test

13. Test report Course: Data structures and algorithms Assignment: Assignment 1 - Pythagorean triplets Student: John Doe Time: 05.09.2012. 11:15:00 Compilation 100% (from 2 points) All is well, no errors. Correctness 33% (from 6 points) Not all triplets were found, check loop boundaries. Optimality 50% (from 2 points) Try using Euclid's formula. -+-+-+-+-+-+-+-+- Total: 5 out of 10 points. Student solution Metric values in xml Script with metric control values eCST Matric values as properties Test report Parse Measure Transform Test

14. A case study Data structures and algorithms course Testing efficiency of a Modula 2 program Using cyclomatic complexity metric – detect loop and branch statements Created a domain specific tester for Testovid – differentiate between typical student solutions – award points accordingly The assignment – Write a program which prints Pythagorean triplets, positive integer numbers x, y and z for which x 2 +y 2 =z 2

15. Solution 1: Naive solution MODULE Triplets1;... VAR x, y, z: INTEGER; zreal: REAL; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO zreal := REAL(Sqrt(LONGREAL(x*x + y*y))); z := TRUNC(zreal); IF zreal = FLOAT(z) THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END Triplets1. Cyclomatic complexity 3 Efficiency Average Points 50%

16. Solution 2: Brute force solution MODULE Triplets2;... VAR x, y, z: INTEGER; BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END Triplets2. Cyclomatic complexity 4 Efficiency Bad Points 0%

17. Solution 3: Using Euclid’s formula MODULE Triplets;... VAR x, y, z, m, n: CARDINAL; BEGIN FOR m := 1 TO Gr DO FOR n := 1 TO m-1 DO x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END Triplets. Cyclomatic complexity 2 Efficiency Good Points 100%

18. Non solution MODULE Triplets4;... VAR x, y, z, m, n, w, i, temp : CARDINAL; BEGIN w := 1; n := 0; FOR i := 1 TO Gr DO m := n + w; x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2); temp := w; w := 3*w + 4*n; n := 2*temp + 3*n END END Triplets4. Cyclomatic complexity 1 Efficiency Excellent Points 100% Lose points for correctness

19. Summary The greater the cyclomatic complexity the worst the solution efficiency Awarded points should be reverse proportional to the cyclomatic complexity – Inefficient solutions – no points – Average solutions – half the maximum points – Efficient solutions – all the points Instructor should use knowledge and experience to – Choose metrics – Define minimum and maximum metric values – Define awarded points for those cases Testing system can automatically – Classify student solutions – Grade them accordingly SolutionEfficiencyMetricPoints Triplets1Average350% Triplets2Bad40% Triplets3Good2100% Triplets4Excellent1!!! 100%

20. Conclusion Utilized software metrics in the assessment process Increased the scope of aspects that can be covered by automatic tests Platform and programming language independent Support a wide range of metrics Left great flexibility in selecting interesting metrics Can provide hints and advices to students Added intelligent assistance Improved student learning experience

21. Thank you for your attention Questions?