Testing versus instance generation Roger L. Costello April 22, 2018

Traditional software development Create software. Create tests. Execute software on tests. Evaluate the test results. If any tests failed, then modify the software and goto step 1.

Alloy development Create model. Have the Alloy Analyzer generate instances. Examine the instances. Are the instances the kind of instances that I expect? If no, modify the model and goto step 2.

Testing vs instance generation You (a human) create the instance (test) and then see if it satisfies the constraints described by the software (i.e., you run the test on the software). With Alloy, the instances are automatically generated by the Alloy Analyzer (a machine) from the model. You examine the instances to see if it is the kind of instance you expect.

Testing vs instance generation (cont.) With traditional software development, tests represent what you meant for the software to say. By executing the tests on the software you see if the software means what you think it means. “This is what I meant to say, did I actually say that?” Alloy instances show you what you said in the model. You examine the instances and decide if that’s what you meant for the model to say. “This is what you said, is that what you meant to say?”

Peter Kriens says … The key reason to create instances of models is to show where you still misunderstand the problem. I’ve come to the conclusion that Alloy is more a specification tool than anything else. I heavily use it to understand a problem domain. The task of modeling something you don’t understand is the most valuable part of Alloy. I’ve probably got hundreds of unfinished models on my computer because once I understood the problem the need disappeared to model it …

Yes, you described in your own words the difference between validation (did i build the right thing) and verification (did i build it right). While "testing" as you named it, is used to verify that the software has been built correctly, Alloy can IMHO be used both for validation and verification, as explained in the following: Validation: Instance generation from an empty predicate gives you instances conforming to your specification allowing you to have a grasp on what you have specified. You can use the knowledge gained from reviewing those instances to validate your specification. Indeed, those instances will help you answer the question: Are the generated instances mirroring what I meant to specify? (did I build the right thing). Verification In Alloy, you can also verify that certain property are respected in your specification through the use of assertions. Checking assertions yields existing counter examples that, when existing, disprove the compliance of your specification to the violated assertion. If any, the counter-examples generated from checking assertion help you answer the question: Did I specify my model correctly w.r.t. my assertions? (did I build it right?). Loïc Gammaitoni