Northeastern University, CCIS/PRL More Flexible Software By Favoring Implicit Calls and Implicit Communication by Karl Lieberherr Northeastern University, CCIS/PRL Joint work with Bryan Chadwick, Ahmed Abdelmeged and Therapon Skotiniotis 11/28/2018 Bern CHOOSE 2007

Overview Why is traversal abstraction important? generalized data abstraction: What is AP? expression problem AP-F and AP-P by example Container capacity checking Translation tasks Implementation: Demeter-F for AP-F and Demeter-P for AP-P Conclusions 11/28/2018 Bern CHOOSE 2007

Important Flexible Software Topics To Which We Contribute with Demeter Generalizing the Data Abstraction Problem Contributing to Solving the Expression Problem Better Abstractions for Traversals 11/28/2018 Bern CHOOSE 2007

Generalizing Data Abstraction well-accepted Data Abstraction Principle: the implementation of objects can be changed without affecting clients provided the interface holds. Adaptive Programming (AP) Principle: the interface of objects can be changed without affecting clients provided the Demeter interface holds. 11/28/2018 Bern CHOOSE 2007

Expression Problem Expression Problem (named by Phil Wadler), which separates structure from computations performed on that structure, while still enabling modular extensions to both the structure and the computations. We address the Expression Problem by practicing structure-shy programming: loosely couple structure and computations. Supports modular extensions to both structure and computations. 11/28/2018 Bern CHOOSE 2007

Traversal Abstraction Traversal of an object is fundamental to any computation on the object. Express WhereToGo using a domain-specific language for navigation control (a-la XPath). But traversals perform diverse computations: Interpretation and Translation. 11/28/2018 Bern CHOOSE 2007

Lack of Traversal Abstraction leads to tight coupling between structure and computations. hinders using the adaptive programming principle hinders solving the expression problem 11/28/2018 Bern CHOOSE 2007

Traversal Abstraction A new way of thinking about traversal abstraction. Explain with Law of Demeter: Talk only to your friends. New: Listen only to your friends. Your friends satisfy all your needs! Two ways: AP-F: F for Functional: communicate through arguments. AP-P: P for interPosition: communicate through variables. 11/28/2018 Bern CHOOSE 2007

What happens if you don’t use traversal abstraction? You write a lot of boiler plate code! Related work: Scrap Your Boilerplate (SYB) started by Laemmel and Peyton-Jones. You tightly couple structure (changes frequently) and computation (more stable). That is against common sense. You always deal with the worst-case scenario of AP-F or AP-P. 11/28/2018 Bern CHOOSE 2007

Related work in Bern Magritte Project (Lukas Renggli (Diplomarbeit), Stephane Ducasse, Adrian Kuhn) Magritte supports end-user customization of meta models. Facilitates creation of infrastructure for changed meta models (editors, views, persistency tools). AP supports customization of meta models including the application-specific code. 11/28/2018 Bern CHOOSE 2007

A Quick Intro to AP-F An AP-F program is defined by three sets of functions which adapt behavior of a predefined traversal with a multiple dispatch function. The three sets of functions: transformers builders augmentors 11/28/2018 Bern CHOOSE 2007

Intro to AP-F Think of object computation as moving information down (augmentors) and pushing up (builders and transformers). Allow transformation at each node (transformers). Send information down as needed (augmentors). Default behavior: copy object. 11/28/2018 Bern CHOOSE 2007

Motto of AP-F Law of Demeter: Talk only to your friends. Law of AP-F: Listen only to your friends and selectively receive information from your superiors. You might not need all the information you get from your friends. You only take what you need and what the communication protocol requires. 11/28/2018 Bern CHOOSE 2007

Motto of AP-F augmentors A object graph Listen to C and D Receive from A B builders/ transformers Does NOT Receive from A C Receive from A D E Receive from A 11/28/2018 Bern CHOOSE 2007

DemeterF implements AP-F Terminology map: AP-F to Java transformers -- apply methods take one argument default: identity builders – combine methods several arguments (from your friends) default: copy augmentors – update methods two arguments (where, what to pass down) default: identity (return second argument) 11/28/2018 Bern CHOOSE 2007

Class Diagram Nested Container Capacity Checking (CCC) Item Element weight: int Container cap: int items * 11/28/2018 Bern CHOOSE 2007

Illustration of combine for capacity constraint violation (w1+w2+w3+w4,2) :Cons (w1+w2+w3+w4,1) :E (w4,0) :Cons (w1+w2+w3,1) :C (w2+w3,1) :Cons (w1,0) :Cons (w2+w3,0) :E (w1,0) :Empty (0,0) :E (w3,0) :Cons (w2,0) both containers (C) violate capacity constraints :E (w2,0) :Empty (0,0) 11/28/2018 Bern CHOOSE 2007

Illustration of combine for capacity constraint violation 19 :C (w1+w2+w3+w4,2) :Cons (w1+w2+w3+w4,1) :E (w4,0) 1 20 2 18a 3 18 3a 13a :Cons (w1+w2+w3,1) after blue arrow combine is active (like after) 17a :C (w2+w3,1) 4 12a 17 :Cons (w1,0) 5 13 14 12 16a :Cons (w2+w3,0) 15a 6 :E (w1,0) :Empty (0,0) 11a 7a 15 16 11 :E (w3,0) :Cons (w2,0) 8 both containers (C) violate capacity constraints 7 10a 9a :E (w2,0) :Empty (0,0) 9 11/28/2018 Bern CHOOSE 2007 10

Container-specific code for CCC AP-F: Listen through arguments | AP-P: Listen through variables Container-specific code for CCC Manual Traversal Pair check(){ Pair p = items.check(); return p.add(0,(p.w > cap)?1:0); } Pair combine(Container c, Integer i, Pair wv) { return wv.add(0,(wv.w > c.cap)?1:0); } void after(Container c) { myWeight += c.myWeight; totalViolations+= ((c.cap<c.myWeight)?1:0); } TALK TO YOUR FRIENDS AP-F AP-P LISTEN TO YOUR FRIENDS 11/28/2018 Bern CHOOSE 2007

AP-F for CCC in DemeterF class Check extends IDb{ // generic combines for passing up pairs Pair combine(Object o, Pair f, Pair r) // pair addition {return f.add(r.w, r.v);} Pair combine(Object o, Pair f) {return f;} // pass it up Pair combine(Object o) { return Pair.make(0,0); } // init // container capacity checking specific combines Pair combine(Element e) {return Pair.make(e.weight,0);} Pair combine(Container c, Integer i, Pair wv) { return wv.add(0, (wv.w>c.cap)?1:0); } } 11/28/2018 Bern CHOOSE 2007

Calling the Function Object Pair check(Item i) { return new Traversal(new Check()).traverse(i); } 11/28/2018 Bern CHOOSE 2007

Class Diagram with Noise for CCC Noise is very common because the objects are used for several purposes. 0 outgoing has-a edges 1 outgoing has-a edge 2 outgoing has-a edges Item Element weight: int Container cap: int items * Noise1 Noise2 Noise3 noise3 noise2 noise1 noise5 noise0 OptItem Noise4 EmptyItem noise4 11/28/2018 Bern CHOOSE 2007

Any program change? Item Element weight: int Container cap: int items * class Check extends IDb{ // generic combines for passing up pairs Pair combine(Object o, Pair f, Pair r) // pair addition {return f.add(r.w, r.v);} Pair combine(Object o, Pair f) {return f;} // pass it up Pair combine(Object o) { return Pair.make(0,0); } // container capacity checking specific combines Pair combine(Element e) {return Pair.make(e.weight,0);} Pair combine(Container c, Integer i, Pair wv) { return wv.add(0, (wv.w>c.cap)?1:0); } } Item Element weight: int Container cap: int items * Noise1 Noise2 Noise3 noise3 noise2 noise1 noise5 noise0 OptItem Noise4 EmptyItem noise4 11/28/2018 Bern CHOOSE 2007

Any Change needed to CCC program? default operation: 1: send up: Pair combine(Object o, Pair f) { return f; } 2: add: Pair combine(Object o, Pair f, Pair r) { return f.add(r.w, r.v); } 3: init: Pair combine(Object o) { return Pair.make(0,0); } 4: problem specific Item Element weight: int Container cap: int items * Noise1 Noise2 Noise3 noise3 noise2 noise1 noise5 noise0 OptItem Noise4 EmptyItem noise4 11/28/2018 Bern CHOOSE 2007

Benefit of Traversal Abstraction NO CHANGE! although meta model changed significantly. 11/28/2018 Bern CHOOSE 2007

Quick Intro to AP-P AP-P is concerned about achieving the most structure-shy yet reusable form of communication among different pieces of traversal advice. The approach proposed by AP-P for structuring communication is to assign a set of interposition variables in the visitor class to store information related to a specific type of objects. 11/28/2018 Bern CHOOSE 2007

Quick Intro to AP-P Pieces of traversal advice communicate through interposition variables which give access to the innermost enclosing object of a given class. An important benefit: communication can take place even across different visitors. An interposition variable is a context sensitive variable. Its context is defined by the traversal. 11/28/2018 Bern CHOOSE 2007

Intro to AP-P Think of computation as traversing the object and collecting information in a visitor object. Use both regular visitor variables as well as interposition variables. Interposition variables facilitate implicit communication. Default behavior: traverse object. 11/28/2018 Bern CHOOSE 2007

CCC for AP-P in DemeterP class Checker extends Visitor{ public int totalViolations = 0; @Interposition (className = "Container") public int myWeight = 0; public void after(Element e) { myWeight += e.weight; } public void after(Container c) { myWeight += c.myWeight; totalViolations+=((c.cap<c.myWeight)?1:0); } public Pair getReturnValue() {return new Pair(myWeight, totalViolations);} Pair check(Item i) { return cg.traverse(i, "from Container to Element",new Checker());} 11/28/2018 Bern CHOOSE 2007

Current Traversal Abstraction Restriction Abstract traversal code non-cyclic objects explicit and implicit cyclic objects cause problems 11/28/2018 Bern CHOOSE 2007

Module = Interface + Implementation + DemeterInterface uses (imports) module M2 module M1 faceM2 mentM2 AdapConM1 faceM1 mentM1 uses (imports) faceM1 mentM1’ module M1 faceM1’ mentM12 faceM1 mentM1’’ faceM1’’ mentM13 change implementation change interface DemeterInterface(M1) 11/28/2018 Bern CHOOSE 2007

What Hinders Creation of Flexible Software Manually following rules like: Follow the structure, follow the grammar. Actively call traversal methods (explicit traversal problem). Also leads to manual passing of arguments (explicit argument problem). 11/28/2018 Bern CHOOSE 2007

Type Unifying: information flow T combine(A a, T t) A object graph T apply(T t) T combine(B b, T t1, T t2) B T apply(T t) T combine(C c, T t1) C D* T apply(D d) T apply(E e) E* 11/28/2018 Bern CHOOSE 2007

Type Unifying: information flow T combine(A a, T t) A object graph T apply(T t) T combine(B b, T t1, T t2) B T apply(T t) T combine(C c, T t1) C D T apply(D d) T apply(E e) E 11/28/2018 Bern CHOOSE 2007

Type Unifying: information flow T combine(A a, T t) A object graph T apply(T t) T combine(B b, T t1, T t2) B T apply(T t) T combine(C c, T t1) C D T apply(D d) T apply(E e) E 11/28/2018 Bern CHOOSE 2007

Type Unifying: information flow T combine(A a, T t) A object graph T apply(T t) T combine(B b, T t1, T t2) B T apply(T t) T combine(C c, T t1) C D T apply(D d) T apply(E e) E 11/28/2018 Bern CHOOSE 2007

Type Unifying: information flow T combine(A a, T t) A object graph T apply(T t) T combine(B b, T t1, T t2) B T apply(T t) T combine(C c, T t1) C D T apply(D d) T apply(E e) E 11/28/2018 Bern CHOOSE 2007

semantics: apply-combine expression Tree = <n> Node [<left> Tree] [<right> Tree]. Node = [IdentityApply] <o> Object. apply(combine(this, apply(combine(left,…)), apply(combine(right,…)) )) Translate tree object into apply-combine expression. 11/28/2018 Bern CHOOSE 2007

Still follow the grammar? Might have to write customized methods for every node. Extreme case. Encode local information about structure in personalized methods. 11/28/2018 Bern CHOOSE 2007

Simple application Program transformation Old E : Num | Var | Op | Call … Op : Plus | Equals. Equals = “=“. New E : … | Bool. Bool : True | False. class BoolTrans extends IDf { static E newtrue = Call.parse(“(= 1 1)”), static E newfalse= Call.parse(“(= 1 0) “); E apply(True t) {return newtrue; } E apply(False t) {return newfalse; } } apply for transformation of result returned by builder 11/28/2018 Bern CHOOSE 2007

de Bruijn indices Old New Var : Sym. Sym = Ident. Var : Sym | Addr. for later de Bruijn indices Old Var : Sym. Sym = Ident. New Var : Sym | Addr. Addr = Integer. class AddrTrans extends IDf { Var apply(Var var, SymList senv) { return new Addr(senv.lookup(var));} } class SymExtender extends IDa { SymList update(Lambda l, SymList senv) { return senv.push(l.formals); } 11/28/2018 Bern CHOOSE 2007

Related Work Modularity First: A Case for Mixing AOP and Attribute Grammars, Pavel Avgustinov, Torbjorn Ekman, Julian Tibble, Programming Tools Group, University of Oxford, AOSD 2008 SYB home page AP home page 11/28/2018 Bern CHOOSE 2007

Conclusions Rely more on your friends. Listen to them on the agreed upon communication channels like your Facebook wall. Don’t tightly couple structure (volatile) and computation (more stable). Use AP-F and AP-P ideas as your design notations. To execute your designs: DemeterF home page. 11/28/2018 Bern CHOOSE 2007

Operation of AP-F 11/28/2018 Bern CHOOSE 2007

AP-F for SDG AP-F: think of generic propagation first add specific processing 11/28/2018 Bern CHOOSE 2007

generic propagation in SDG push down predicate, retrieve an object satisfying predicate. push down assignment, combine integer or formula. push down predicate. 11/28/2018 Bern CHOOSE 2007

Design space for Demeter-F programs ID use builders and transformers 11/28/2018 Bern CHOOSE 2007

managing software development Learn by evolving a software product: an algorithmic financial derivative trading game Learn by working in a small team using pair programming Learn by integrating software from the teams 11/28/2018 Bern CHOOSE 2007

SDG Features outcome determined by owner/buyer type outcome raw material (owner) finished product (buyer) pay (quality) outcome determined by environment pay 11/28/2018 Bern CHOOSE 2007

SDG Features outcome determined by owner/buyer type T set of relations of rank k exactly rank k, at most rank k. size of set, e.g., 2. use a predicate to constrain permitted relations, e.g. only OR relations -> CNF outcome raw material (owner) CSP(T) formula finished product (buyer) assignment to variables of CSP(T) formula. pay (quality) satisfaction_ratio all-or-nothing: satisfaction_ratio > price:1:0 outcome determined by environment type pay 11/28/2018 Bern CHOOSE 2007

SDG Features synchronous: rounds in fixed order: p1 p2 p3 p1 p2 p3 …Can buy and offer when it is your turn. On each turn, must buy derivative or re-offer all existing derivatives at lower price. asynchronous: Can buy and offer at any time. Time interval RV. In every RV, must buy derivative or re-offer all existing derivatives at lower price. 11/28/2018 Bern CHOOSE 2007

SDG Features Security 11/28/2018 Bern CHOOSE 2007

Domain Design for SDG Domain Specific Languages Need languages for types raw material finished product configuring the game game moves game rules? 11/28/2018 Bern CHOOSE 2007

Five Deep Questions in Computing Jeannette Wing, CACM Jan. 2008 P=NP What is computable? What is intelligence? What is information? How can we build complex systems simply? 11/28/2018 Bern CHOOSE 2007

Deep? Speak to the foundations of the field. Understanding and expanding the frontiers of computing. P=NP: would have profound practical consequences, shaking the foundations of cryptography. P!=NP: profound theoretical consequences. 11/28/2018 Bern CHOOSE 2007

SDG and P=NP “Is there an assignment that satisfies at least k clauses”: is in NP and P=NP if this problem has a polynomial time algorithm. 11/28/2018 Bern CHOOSE 2007

What is intelligence? Can a computer learn to play SDG well without being programmed explicitly? What are the algorithms that the computer would invent? Does it require intelligence to play the game well? 11/28/2018 Bern CHOOSE 2007

How can we build complex systems simply? Can we build systems with simple and elegant designs that are easy to understand, modify and evolve yet still provide the functionality we want today and dream of for tomorrow? 11/28/2018 Bern CHOOSE 2007

Programming in DemeterF Every programming technology has difficulties associated with it. Programmers then use style rules to avoid those difficulties. Examples: null pointer exceptions in Java. Rule: A = [X]. -> A = XOpt. XOpt : X | EmptyX. excessive structural dependencies in OO. Rule: Law of Demeter. 11/28/2018 Bern CHOOSE 2007

DemeterF Example when I write: Integer combine(Object o, Integer n1, Integer n2) {return n1+n2;} I intend that n1 and n2 are returned by an apply or combine method. They should not be accidental integers that happen to be parts of object o. 11/28/2018 Bern CHOOSE 2007

Possible style rule: Capture Avoidance Rule The return type of a combine method should not appear in the structure being traversed (Do not return a traversed type). Example: If the structure contains Integer parts and the purpose is to add, we could use Float to count (see project 3). 11/28/2018 Bern CHOOSE 2007

Implementation of Capture Avoid.: Terminal Buffer Rule with Built-ins The terminal buffer rule (TBR) says that all terminal classes should be buffered by a class that has the terminal class as a part class. TBR improves readability of programs. violation: City = <name> String <zip> Integer. with TBR: City = CityName Zip. CityName = String. Zip = Integer. 11/28/2018 Bern CHOOSE 2007

Terminal Buffer Rule with Built-ins CityName and Zip are made built-in classes so that the traversal does not go further. Then we are free to use Integer as return type of combine. 11/28/2018 Bern CHOOSE 2007

Dispatch function of DemeterF chooses most specific method more information means more specific only list arguments up to last one that is needed 11/28/2018 Bern CHOOSE 2007