1. Unbounded Data Model Verification Using SMT Solvers Jaideep NijjarTevfik Bultan University of California, Santa Barbara ASE 2012

2. Web Software Everywhere Commerce, entertainment, social interaction We will rely on web apps more in the future Web apps + cloud will make desktop apps obsolete

3. Acknowledgement: NSF Support

4. It is not just NSF

5. Web Application Dependability

7. Web Application Dependability is a Problem Web applications are hard to program Distributed behavior, interaction among many components and many languages Web applications are hard to test Highly dynamic behavior and concurrency Web applications are easy targets for hackers They are notorious for security vulnerabilities and unreliable behavior My research group’s goal: Improving dependability of web applications using automated verification techniques!

8. We have a hammer Automated Verification

9. Unfortunately, life is complicated Automated verification techniques Web application dependability problems

10. Making it work Separation of concerns + modularity + abstraction/extraction SMT-based verification Data model problems Formal data model + Properties

11. Three-Tier Architecture Backend Database Browser Web Server

12. Three-Tier Arch. + MVC Pattern Backend Database Browser Model ViewsController Web Server MVC pattern has become the standard way to structure web applications: Ruby on Rails Zend for PHP CakePHP Struts for Java Django for Python …

13. Web Application Ruby on Rails Data Model ActiveRecords Formal Model Sets and Relations Unbounded Verification SMT Solver Our Approach MVC Design Pattern Automatic Extraction Automatic Translation + Automatic Projection + Properties

14. Outline Motivation Overview of Our Approach Rails Semantics Translation to SMT-LIB Experiments Related Work Conclusions

15. A Rails Data Model Example class User < ActiveRecord::Base has_and_belongs_to_many :roles has_one :profile, :dependent => :destroy has_many :photos, :through => :profile end class Role < ActiveRecord::Base has_and_belongs_to_many :users end class Profile < ActiveRecord::Base belongs_to :user has_many :photos, :dependent => :destroy has_many :videos, :dependent => :destroy, :conditions => "format='mp4'" end class Tag < ActiveRecord::Base belongs_to :taggable, :polymorphic => true end class Video < ActiveRecord::Base belongs_to :profile has_many :tags, :as => :taggable end class Photo < ActiveRecord::Base... Role * 0..1 1 User Profile * 1 Video * 1 Taggable * Tag 1 * 1 Photo * 1 format=.‘mp4’

16. Rails Data Models Data model verification: Analyzing the relationships between data objects Specified in Rails using association declarations inside the ActiveRecord files The basic relationships One-to-one One-to-many Many-to-many Extensions to the basic relationships using Options :through, :conditions, :polymorphic, :dependent

17. The Three Basic Relationships in Rails One-to-One (One-to-ZeroOrOne). One-to-Many class User < ActiveRecord::Base has_one :profile end. class Profile < ActiveRecord::Base belongs_to :user end class Profile < ActiveRecord::Base has_many :videos end. class Video < ActiveRecord::Base belongs_to :profile end User Profile 0..1 Profile Video * 1 1

18. The Three Basic Relationships in Rails Many-to-Many class User < ActiveRecord::Base has_and_belongs_to_many :users end class Role < ActiveRecord::Base has_and_belongs_to_many :roles end User Role * *

19. Options to Extend the Basic Relationships :through Option To express transitive relations :conditions Option To relate a subset of objects to another class :polymorphic Option To express polymorphic relationships :dependent Option On delete, this option expresses whether to delete the associated objects or not

20. The :through Option class User < ActiveRecord::Base has_one :profile has_many :photos, :through => :profile end class Profile < ActiveRecord::Base belongs_to :user has_many :photos end class Photo < ActiveRecord::Base belongs_to :profile end Profile User Photo * * 0..11 1 1

21. The :dependent Option :delete directly delete the associated objects without looking at its dependencies :destroy first checks whether the associated objects themselves have associations with the :dependent option set class User < ActiveRecord::Base has_one :profile, :dependent => :destroy end class Profile < ActiveRecord::Base belongs_to :user has_many :photos, :dependent => :destroy end PhotoProfileUser * 11 0..1

22. Formalizing Rails Semantics S: The sets and relations of the data model (data model schema) e.g. { Photo, Profile, Role, Tag, Video, User} and the relations between them C: Constraints on the relations Cardinality constraints, transitive relations, conditional relations, polymorphic relations D: Dependency constraints Express conditions on two consecutive instances of a relation such that deletion of an object from the first instance leads to the other instance Formal data model: M =

23. Formalizing Rails Semantics I = is an instance of the data model M =, denoted by I |= M, iff 1.the sets in O and the relations in R follow the schema S, and 2.R |= C Given a pair of data model instances I = and I’ = (I, I’) is a behavior of the data model M =, denoted by (I, I’) |= M, iff 1.O and R and O’ and R’ follow the schema S 2.R |= C and R’ |= C, and 3.(R,R’) |= D

24. Data Model Properties Given a data model M =, we define four types of properties: 1.state assertions (A S ): properties that we expect to hold for each instance of the data model 2.behavior assertions (A B ): properties that we expect to hold for each pair of instances that form a behavior of the data model 3.state predicates (P S ): properties we expect to hold in some instance of the data model 4.behavior predicates (P B ): properties we expect to hold in some pair of instances that form a behavior of the data model

25. Outline Motivation Overview of Our Approach Rails Semantics Translation to SMT-LIB Experiments Related Work Conclusions

26. Translation to SMT-LIB Given a data model M = we translate the constraints C and D to formulas in the theory of uninterpreted functions We use the SMT-LIB format We need quantification for some constraints

27. Translation to SMT-LIB class Profile has_many :videos end class Video belongs_to :profile end (declare-sort Profile 0) (declare-sort Video 0) (declare-fun my_relation (Video) Profile). SMT-LIB: RAILS: One-to-Many Relation

28. Translation to SMT-LIB class User has_one :profile end class Profile belongs_to :user end (declare-sort User 0) (declare-sort Profile 0) (declare-fun my_relation (Profile) User). (assert (forall ((x1 Profile)(x2 Profile)) (=> (not (= x1 x2)) (not (= (my_relation x1) (my_relation x2) )) ) )) SMT-LIB: RAILS: One-to-One Relation

29. Translation to SMT-LIB class User has_and_belongs_to_many :roles end class Role has_and_belongs_to_many :users end (declare-sort Role 0) (declare-sort User 0) (declare-fun my_relation (Role User) Bool) SMT-LIB: RAILS: Many-to-Many Relation

30. Translating the :through Option class Profile < ActiveRecord::Base belongs_to :user has_many :photos end class Photo < ActiveRecord::Base belongs_to :profile End class User < ActiveRecord::Base has_one :profile has_many :photos, :through => :profile end (declare-sort Profile 0) (declare-sort Photo 0) (declare-sort User 0) (declare-fun profile_photo (Photo) Profile) (declare-fun user_profile (Profile) User) (declare-fun user_photo (Photo) User) (assert (forall ((u User)(ph Photo)) (iff (= u (user_photo ph)) (exists ((p Profile)) (and (= u (user_profile p)) (= p (profile_photo ph)) )) )) ) Profile User Photo 0..1 * * 1 1 1

31. Translating the :dependent Option The :dependent option specifies what behavior to take on deletion of an object with regards to its associated objects To incorporate this dynamism, the model must allow analysis of how sets of objects and their relations change from one state to the next class User < ActiveRecord::Base has_one :account, :dependent => :destroy end. class Profile < ActiveRecord::Base belongs_to :user end (declare-sort Profile 0) (declare-sort User 0) (declare-fun Post_User (User) Bool) (declare-fun Post_Profile (Profile) Bool) (declare-fun user_profile (Profile) User) (declare-fun Post_user_profile (Profile User) Bool)

32. Translating the :dependent Option Update sets relations of its associated object(s) based on the use of the :dependent option A relation is only updated if it is a :belongs_to or : has_and_belongs_to_many relationship In the database, the foreign key is stored with the object that has the :belongs_to relationship (assert (not (forall ((x User)) (=> (and (forall ((a User)) (ite (= a x) (not (Post_User a)) (Post_User a))) (forall ((b Profile)) (ite (= x (user_profile b)) (not (Post_Profile b)) (Post_Profile b) )) (forall ((a Profile) (b User)) (ite (and (= b (user_profile a)) (Post_Profile a)) (Post_user_profile a b) (not (Post_user_profile a b)) )) ) ;Remaining property-specific constraints go here )))

33. Verification Once the data model is translated to SMT-LIB format we can state properties about the data model again in SMT-LIB and then use an SMT-Solver to check if the property holds in the data model However, when we do that, for some large models, SMT- Solver times out! Can we improve the efficiency of the verification process?

34. Property-Based Data Model Projection Basic idea: Given a property to verify, reduce the size of the generated SMT-LIB specification by removing declarations and constraints that do not depend on the property Formally, given a data model M = and a property p,  (M, p) = M P where M P = S, C P, D P is the projected data model such that C P ⊆ C and D P ⊆ D Key Property: For any property p, M |= p ⇔  (M, p) |= p Implemented as part of our tool Algorithm Input: Active Record files, property p Output: The projected SMT-LIB specification Removes constraints on those classes and relations that are not explicitly mentioned in the property nor related to them based on transitive relations, dependency constraints or polymorphic relations

35. Data Model Projection: Example Property, p: A User’s Photos are the same as the User’s Profile’s Photos. Role * 0..1 1 User Profile * 1 Video * 1 Taggable * Tag 1 * 1 Photo * 1 Data Model, M: 0..1 1 User Profile * 1 Photo * 1 (M, p) =

36. Verification Overview Translator SMT Solver (Z3) Verified Counter-example Data Model Instance Unknown Active Records Formal Data Model Data Model Properties Projection SMT-LIB Specification

37. Outline Motivation Overview of Our Approach Rails Semantics Translation to SMT-LIB Experiments Related Work Conclusion

38. Experiments We used five open-source Rails apps in our experiments: LovdByLess: Social networking site Tracks: An application to manage things-to-do lists OpenSourceRails(OSR): Social project gallery application Fat FreeCRM: Customer relations management software Substruct: An e-commerce application We wrote 10 properties for each application LovdBy Less TracksOSR Fat Free CRM Substruct LOC3787606242951206915639 Data Model Classes13 152017

39. Types of Properties Checked Relationship Cardinality Is an Opportunity always assigned to some Campaign? Transitive Relations Is a Note’s User the same as the Note’s Project’s User? Deletion Does Not Cause Dangling References Are there any dangling Todos after a User is deleted? Deletion Propagates to Associated Objects Does the User related to a Lead still exist after the Lead has been deleted? Note UserProject

40. Experimental Results 50 properties checked, 16 failed, 11 were data model errors For example in Tracks, a Note’s User can be different than Note’s Project’s User Currently being enforced by the controller Since this could have been enforced using the :through option, we consider this a data-modeling error From OpenSourceRails: User deletion fails to propagate to associated Bookmarks Leaves orphaned bookmarks in database Could have been enforced in the data model by setting the :dependent option on the relation between User and Bookmark BookmarkUser * 1

41. Performance To measure performance, we recorded The amount of time it took for Z3 to run and check the properties The number of variables produced in the SMT specfication The time and number of variables are averaged over the properties for each application To compare with bounded verification, we repeated these experiments using the tool from our previous work and Alloy Analyzer The amount of time it took for Alloy to run The number of variables generated in the boolean formula generated for the SAT solver Taken over an increasing bound, from at most 10 objects for each class to at most 35 objects for each class

42. Performance: Verification Time

43. Performance: Formula Size (Variables) Z3 Alloy

44. Unbounded vs Bounded Performance Why does unbounded verification out-perform bounded so drastically? Possible reasons: SMT solvers operate at a higher level of abstraction than SAT solvers Z3 uses many heuristics to eliminate quantifiers in formulas Implementation languages are different Z3 implemented in C++ Alloy (as well as the SAT Solver it uses) is implemented in Java

45. Outline Motivation Overview of Our Approach Rails Semantics Translation to SMT-LIB Experiments Related Work Conclusions

46. Related Work Previous work on Data Model Verification using Alloy [Cunha and Pacheco, SEFM 2009] maps relational database schemas to Alloy; not automated [Wang et al, ASWEC 2006] translates ORA-SS specifications to Alloy, and uses the Analyzer to produces instances of the data model to show consistency These approaches bounded, not unbounded technique like ours [Borbar et al, Trends 2005] uses Alloy to discover bugs in browser and business logic interactions A different class of bugs than the data model related bugs we focus on Unbounded verification of Alloy specifications using SMT Solvers [Ghazi et al, FM 2011], approach not implemented More challenging domain since Alloy language contains constructs such as transitive closures which do not appoar in the data models we extract

47. Related Work Specification and Analysis of Conceptual Data Models [McGill et al ISSTA 2011, Smaragdakis et al ASE 2009, Halpin et al IMRD 2009] Model-driven (forward engineering) approaches, whereas we performed model extraction (reverse engineering) Formal modeling of Web Applications [Book et al ASE 2004, Hallé et al ASE 2010, Han et al MoDELS 2007] Focus on navigation aspects as opposed to data model SMT Solvers More Efficient than SAT Solvers Observed in other verification domains [Cordeiro et al, ASE 2009] The data model verification problem we investigate is different from the problems studied in earlier works

48. Conclusions Goal: To automatically discover data model errors in Ruby on Rails web applications Approach: Automatically extract a formal data model, translate it to the theory of uninterpreted functions, and verify using an SMT-solver Use property-based data model projection for efficiency Implementation: An automatic translator from Rails ActiveRecords to SMT-LIB Handles three basic relationships and several options (:through, :conditions, :polymorphic, :dependent) Experiments: Found multiple data model errors on five open source applications Unbounded verification of data models is feasible and more efficient than bounded verification!

49. Future Work Analyzing dynamic behavior Model object creation in addition object deletion Fuse the data model with the navigation model in order to analyze dynamic data model behavior Check temporal properties Automatic Property Inference Current work requires manual property writing Use the inherent graph structure in the the data model to automatically infer properties about the data model Automatic Repair When verifier concludes that a data model is violated, automatically generate a repair that establishes the violated property

50. Questions?