+ Auto-Testing Code for Teachers & Beginning Programmers Dr. Ronald K. Smith Graceland University

+ Introduction My story

+ Beginning Programmers 1....seldom understand the importance of writing robust code generally believe the programming task ends with the first syntax–error free run 3....have no idea how to design good test data sets.

+ Beginning programmer teachers 1....spend inordinate amounts of time grading code that either doesn't work or works only on one specific example don't have time to run student programs on multiple sets of data often end up testing student programs on non-uniform data sets.

+ Wouldn't it be nice if......teachers had an easy way to design and save data sets for testing programs?...graders could test programs on multiple sets of data simultaneously?...students could "blind test" programs on good data sets, getting helpful feedback without seeing the actual data sets?... programs worked before being submitted for grading?

+ AutoTest System overview

+ AutoTest MasterGraderStudent Designs Problems & Data Sets Grades Student Programs Writes Programs "Blind tests" Data

+ Master View The master Writes problem specs, including input and output. Writes a master program. Designs multiple test data sets. AutoTest Generates a master file containing each data set and its corresponding output.

+ Student view A Student Writes a program to meet I/O specs. Tests the program with the sample data set and corresponding output provided with the problem. AutoTest Matches a master file to the student program and compares output from each master data set to the output in the master file in a summary table.

+ Grader View Grader Chooses a master file Sets the comparison level (strict, ignore white space, etc.) Selects the folder containing student program folders. Autotest Displays the student program, student output on each data set, and a summary table.

+ Example

+ Problem Specs

+ Master Program

+ Data Sets Sample Data Zero Lines Blank Lines Single line Names hint at aspect tested.

+ Master View "File Input" opens existing master file. "File Output" generates output. "Save Master" encodes all the info into a single text file.

+ (Coded) Master File This file is distributed to student machines along with a compiled student version of AutoTest. I2VjaG8gNQojU2FtcGxlIERhdGEKSGVsbG8gV29ybGQhCiNOdW1lc mljYWwKNDIKI0VtcHR5IExpbmUKCiNFeHRyYSBJbnB1dApMaW5lIDE KTGluZSAyCiNlbmQgaW5wdXQKI1NhbXBsZSBEYXRhCkhlbGxvIFdvc mxkIQoKI051bWVyaWNhbAo0MgoKI0VtcHR5IExpbmUKCgojRXh0c mEgSW5wdXQKTGluZSAxCgojZW5kIG91dHB1dA==

+ Grader View Current Program Current Output Comparison Level Summary

+ Some Programs have small errors. Student left out a required space after line numbers. Rerunning at the "letters" level shows formatting is the only error. At the "Exact" level, only one set is error free.

+ Some bugs cannot be ignored. This program had an undefined variable name in the print statement.

+ Student View Comparison Level Summary Student View = Grader Vew –( Program & Output)

+ Implementation

+ Some Nuts and Bolts A compiled version (.pyc) of AutoTest along with a script (.bat) file with which to execute it is distributed to each computer in the CSIT lab. Students make a shortcut to the AutoTest program and store it in a folder in which they will be testing programs. AutoTest uses digits in the program name to match appropriate master files. Finished programs are uploaded using my.graceland.edu. When downloaded, each program is in a folder with the student name, ready for batch processing with AutoTest. New Master files are distributed each time an assignment is made.

+ Security AutoTest is written in Python, so it is virtually platform independent. I used to distribute copies to the lab machines, and did not tell the students about the master files (which were base 64 coded). Last year, one of my students asked me about base 64 coding, having gone through the code to hack the system! We considered a web-based solution, but allowing any Python program to run from the web is very insecure.

+ Summary

+ AutoTest Is a GUI interface to a system with 3 views: Master Grader Student Facilitates testing/grading of programs with multiple data sets simultaneously.

+ Advantages Beginning students are encouraged to write more robust code. Students can verify that I/O specs are met before turning in their programs. Program are tested on multiple data sets. Program grades are based on uniform tests. Files containing multiple data sets with input and output are easily made and edited. Program output can be checked with various matching criteria. Most programs turned in work! Grades can be based on other criteria. Everyone wins

+