1. Northeastern University, CCIS/PRL
More Flexible Software By Favoring Implicit Calls and Implicit Communication
by Karl Lieberherr
Northeastern University, CCIS/PRL
Joint work with Bryan Chadwick, Ahmed Abdelmeged and Therapon Skotiniotis
11/28/2018
Bern CHOOSE 2007

2. Overview Why is traversal abstraction important?
generalized data abstraction: What is AP?
expression problem
AP-F and AP-P by example
Container capacity checking
Translation tasks
Implementation: Demeter-F for AP-F and Demeter-P for AP-P
Conclusions
11/28/2018
Bern CHOOSE 2007

3. Important Flexible Software Topics To Which We Contribute with Demeter
Generalizing the Data Abstraction Problem
Contributing to Solving the Expression Problem
Better Abstractions for Traversals
11/28/2018
Bern CHOOSE 2007

4. Generalizing Data Abstraction
well-accepted
Data Abstraction Principle: the implementation of objects can be changed without affecting clients provided the interface holds.
Adaptive Programming (AP) Principle: the interface of objects can be changed without affecting clients provided the Demeter interface holds.
11/28/2018
Bern CHOOSE 2007

5. Expression Problem
Expression Problem (named by Phil Wadler), which separates structure from computations performed on that structure, while still enabling modular extensions to both the structure and the computations.
We address the Expression Problem by practicing structure-shy programming: loosely couple structure and computations.
Supports modular extensions to both structure and computations.
11/28/2018
Bern CHOOSE 2007

6. Traversal Abstraction
Traversal of an object is fundamental to any computation on the object.
Express WhereToGo using a domain-specific language for navigation control (a-la XPath).
But traversals perform diverse computations: Interpretation and Translation.
11/28/2018
Bern CHOOSE 2007

7. Lack of Traversal Abstraction
leads to tight coupling between structure and computations.
hinders using the adaptive programming principle
hinders solving the expression problem
11/28/2018
Bern CHOOSE 2007

8. Traversal Abstraction
A new way of thinking about traversal abstraction.
Explain with Law of Demeter: Talk only to your friends.
New: Listen only to your friends. Your friends satisfy all your needs! Two ways:
AP-F: F for Functional: communicate through arguments.
AP-P: P for interPosition: communicate through variables.
11/28/2018
Bern CHOOSE 2007

9. What happens if you don’t use traversal abstraction?
You write a lot of boiler plate code! Related work: Scrap Your Boilerplate (SYB) started by Laemmel and Peyton-Jones.
You tightly couple structure (changes frequently) and computation (more stable). That is against common sense.
You always deal with the worst-case scenario of AP-F or AP-P.
11/28/2018
Bern CHOOSE 2007

10. Related work in Bern
Magritte Project (Lukas Renggli (Diplomarbeit), Stephane Ducasse, Adrian Kuhn)
Magritte supports end-user customization of meta models. Facilitates creation of infrastructure for changed meta models (editors, views, persistency tools).
AP supports customization of meta models including the application-specific code.
11/28/2018
Bern CHOOSE 2007

11. A Quick Intro to AP-F
An AP-F program is defined by three sets of functions which adapt behavior of a predefined traversal with a multiple dispatch function. The three sets of functions:
transformers
builders
augmentors
11/28/2018
Bern CHOOSE 2007

12. Intro to AP-F
Think of object computation as moving information down (augmentors) and pushing up (builders and transformers).
Allow transformation at each node (transformers).
Send information down as needed (augmentors).
Default behavior: copy object.
11/28/2018
Bern CHOOSE 2007

13. Motto of AP-F Law of Demeter: Talk only to your friends.
Law of AP-F: Listen only to your friends and selectively receive information from your superiors.
You might not need all the information you get from your friends. You only take what you need and what the communication protocol requires.
11/28/2018
Bern CHOOSE 2007

14. Motto of AP-F augmentors A object graph Listen to C and D
Receive from A B
builders/
transformers
Does NOT Receive from A C
Receive from A D E
Receive from A
11/28/2018
Bern CHOOSE 2007

15. DemeterF implements AP-F
Terminology map: AP-F to Java
transformers -- apply methods
take one argument
default: identity
builders – combine methods
several arguments (from your friends)
default: copy
augmentors – update methods
two arguments (where, what to pass down)
default: identity (return second argument)
11/28/2018
Bern CHOOSE 2007

16. Class Diagram Nested Container Capacity Checking (CCC)
Item
Element
weight: int
Container
cap: int
items *
11/28/2018
Bern CHOOSE 2007

17. Illustration of combine for capacity constraint violation
(w1+w2+w3+w4,2)
:Cons
(w1+w2+w3+w4,1) :E
(w4,0)
:Cons
(w1+w2+w3,1) :C
(w2+w3,1)
:Cons
(w1,0)
:Cons
(w2+w3,0) :E
(w1,0)
:Empty
(0,0) :E
(w3,0)
:Cons
(w2,0)
both containers (C) violate
capacity constraints :E
(w2,0)
:Empty
(0,0)
11/28/2018
Bern CHOOSE 2007

18. Illustration of combine for capacity constraint violation19 :C
(w1+w2+w3+w4,2)
:Cons
(w1+w2+w3+w4,1) :E
(w4,0) 1 20 2
18a 3 18 3a
13a
:Cons
(w1+w2+w3,1)
after blue arrow
combine is
active (like after)
17a :C
(w2+w3,1) 4
12a 17
:Cons
(w1,0) 5 13 14 12
16a
:Cons
(w2+w3,0)
15a 6 :E
(w1,0)
:Empty
(0,0)
11a 7a 15 16 11 :E
(w3,0)
:Cons
(w2,0) 8
both containers (C) violate
capacity constraints 7
10a 9a :E
(w2,0)
:Empty
(0,0) 9
11/28/2018
Bern CHOOSE 2007 10

19. Container-specific code for CCC
AP-F: Listen through arguments | AP-P: Listen through variables
Container-specific code for CCC
Manual Traversal
Pair check(){
Pair p = items.check();
return p.add(0,(p.w > cap)?1:0); }
Pair combine(Container c, Integer i, Pair wv)
{ return wv.add(0,(wv.w > c.cap)?1:0); }
void after(Container c)
{ myWeight += c.myWeight;
totalViolations+= ((c.cap<c.myWeight)?1:0); }
TALK TO YOUR FRIENDS
AP-F
AP-P
LISTEN TO YOUR FRIENDS
11/28/2018
Bern CHOOSE 2007

20. AP-F for CCC in DemeterF
class Check extends IDb{
// generic combines for passing up pairs
Pair combine(Object o, Pair f, Pair r) // pair addition
{return f.add(r.w, r.v);}
Pair combine(Object o, Pair f) {return f;} // pass it up
Pair combine(Object o) { return Pair.make(0,0); } // init
// container capacity checking specific combines
Pair combine(Element e) {return Pair.make(e.weight,0);}
Pair combine(Container c, Integer i, Pair wv)
{ return wv.add(0, (wv.w>c.cap)?1:0); } }
11/28/2018
Bern CHOOSE 2007

21. Calling the Function Object
Pair check(Item i)
{ return new
Traversal(new Check()).traverse(i); }
11/28/2018
Bern CHOOSE 2007

22. Class Diagram with Noise for CCC
Noise is very common because the objects are used for several purposes.
0 outgoing has-a edges
1 outgoing has-a edge
2 outgoing has-a edges
Item
Element
weight: int
Container
cap: int
items *
Noise1
Noise2
Noise3
noise3
noise2
noise1
noise5
noise0
OptItem
Noise4
EmptyItem
noise4
11/28/2018
Bern CHOOSE 2007

23. Any program change? Item Element weight: int Container cap: int items*
class Check extends IDb{
// generic combines for passing up pairs
Pair combine(Object o, Pair f, Pair r) // pair addition
{return f.add(r.w, r.v);}
Pair combine(Object o, Pair f) {return f;} // pass it up
Pair combine(Object o) { return Pair.make(0,0); }
// container capacity checking specific combines
Pair combine(Element e) {return Pair.make(e.weight,0);}
Pair combine(Container c, Integer i, Pair wv)
{ return wv.add(0, (wv.w>c.cap)?1:0); } }
Item
Element
weight: int
Container
cap: int
items *
Noise1
Noise2
Noise3
noise3
noise2
noise1
noise5
noise0
OptItem
Noise4
EmptyItem
noise4
11/28/2018
Bern CHOOSE 2007

24. Any Change needed to CCC program?
default operation:
1: send up: Pair combine(Object o, Pair f) { return f; }
2: add: Pair combine(Object o, Pair f, Pair r) { return f.add(r.w, r.v); }
3: init: Pair combine(Object o) { return Pair.make(0,0); }
4: problem specific
Item
Element
weight: int
Container
cap: int
items *
Noise1
Noise2
Noise3
noise3
noise2
noise1
noise5
noise0
OptItem
Noise4
EmptyItem
noise4
11/28/2018
Bern CHOOSE 2007

25. Benefit of Traversal Abstraction
NO CHANGE! although meta model changed significantly.
11/28/2018
Bern CHOOSE 2007

26. Quick Intro to AP-P
AP-P is concerned about achieving the most structure-shy yet reusable form of communication among different pieces of traversal advice.
The approach proposed by AP-P for structuring communication is to assign a set of interposition variables in the visitor class to store information related to a specific type of objects.
11/28/2018
Bern CHOOSE 2007

27. Quick Intro to AP-P
Pieces of traversal advice communicate through interposition variables which give access to the innermost enclosing object of a given class.
An important benefit: communication can take place even across different visitors.
An interposition variable is a context sensitive variable. Its context is defined by the traversal.
11/28/2018
Bern CHOOSE 2007

28. Intro to AP-P
Think of computation as traversing the object and collecting information in a visitor object.
Use both regular visitor variables as well as interposition variables.
Interposition variables facilitate implicit communication.
Default behavior: traverse object.
11/28/2018
Bern CHOOSE 2007

29. CCC for AP-P in DemeterP
class Checker extends Visitor{
public int totalViolations = 0;
@Interposition (className = "Container")
public int myWeight = 0;
public void after(Element e) { myWeight += e.weight; }
public void after(Container c)
{ myWeight += c.myWeight;
totalViolations+=((c.cap<c.myWeight)?1:0); }
public Pair getReturnValue() {return new Pair(myWeight, totalViolations);}
Pair check(Item i)
{ return cg.traverse(i, "from Container to Element",new Checker());}
11/28/2018
Bern CHOOSE 2007

30. Current Traversal Abstraction Restriction
Abstract traversal code
non-cyclic objects
explicit and implicit cyclic objects cause problems
11/28/2018
Bern CHOOSE 2007

31. Module = Interface + Implementation + DemeterInterface
uses (imports)
module M2
module M1
faceM2 mentM2 AdapConM1
faceM1 mentM1
uses (imports)
faceM1 mentM1’
module M1
faceM1’ mentM12
faceM1 mentM1’’
faceM1’’ mentM13
change implementation
change interface
DemeterInterface(M1)
11/28/2018
Bern CHOOSE 2007

32. What Hinders Creation of Flexible Software
Manually following rules like: Follow the structure, follow the grammar.
Actively call traversal methods (explicit traversal problem).
Also leads to manual passing of arguments (explicit argument problem).
11/28/2018
Bern CHOOSE 2007

33. Type Unifying: information flow
T combine(A a, T t) A
object graph
T apply(T t)
T combine(B b, T t1, T t2) B
T apply(T t)
T combine(C c, T t1) C D*
T apply(D d)
T apply(E e) E*
11/28/2018
Bern CHOOSE 2007

34. Type Unifying: information flow
T combine(A a, T t) A
object graph
T apply(T t)
T combine(B b, T t1, T t2) B
T apply(T t)
T combine(C c, T t1) C D
T apply(D d)
T apply(E e) E
11/28/2018
Bern CHOOSE 2007

35. Type Unifying: information flow
T combine(A a, T t) A
object graph
T apply(T t)
T combine(B b, T t1, T t2) B
T apply(T t)
T combine(C c, T t1) C D
T apply(D d)
T apply(E e) E
11/28/2018
Bern CHOOSE 2007

36. Type Unifying: information flow
T combine(A a, T t) A
object graph
T apply(T t)
T combine(B b, T t1, T t2) B
T apply(T t)
T combine(C c, T t1) C D
T apply(D d)
T apply(E e) E
11/28/2018
Bern CHOOSE 2007

37. Type Unifying: information flow
T combine(A a, T t) A
object graph
T apply(T t)
T combine(B b, T t1, T t2) B
T apply(T t)
T combine(C c, T t1) C D
T apply(D d)
T apply(E e) E
11/28/2018
Bern CHOOSE 2007

38. semantics: apply-combine expression
Tree = <n> Node
[<left> Tree]
[<right> Tree].
Node = [IdentityApply]
<o> Object.
apply(combine(this,
apply(combine(left,…)),
apply(combine(right,…)) ))
Translate tree object into apply-combine expression.
11/28/2018
Bern CHOOSE 2007

39. Still follow the grammar?
Might have to write customized methods for every node. Extreme case.
Encode local information about structure in personalized methods.
11/28/2018
Bern CHOOSE 2007

40. Simple application Program transformation
Old
E : Num | Var | Op | Call …
Op : Plus | Equals.
Equals = “=“.
New
E : … | Bool.
Bool : True | False.
class BoolTrans extends IDf {
static E newtrue = Call.parse(“(= 1 1)”),
static E newfalse= Call.parse(“(= 1 0) “);
E apply(True t) {return newtrue; }
E apply(False t) {return newfalse; } }
apply for transformation of result returned by builder
11/28/2018
Bern CHOOSE 2007

41. de Bruijn indices Old New Var : Sym. Sym = Ident. Var : Sym | Addr.
for later
de Bruijn indices
Old
Var : Sym.
Sym = Ident.
New
Var : Sym | Addr.
Addr = Integer.
class AddrTrans extends IDf {
Var apply(Var var, SymList senv)
{ return new Addr(senv.lookup(var));} }
class SymExtender extends IDa {
SymList update(Lambda l, SymList senv)
{ return senv.push(l.formals); }
11/28/2018
Bern CHOOSE 2007

42. Related Work
Modularity First: A Case for Mixing AOP and Attribute Grammars, Pavel Avgustinov, Torbjorn Ekman, Julian Tibble, Programming Tools Group, University of Oxford, AOSD 2008
SYB home page
AP home page
11/28/2018
Bern CHOOSE 2007

43. Conclusions
Rely more on your friends. Listen to them on the agreed upon communication channels like your Facebook wall.
Don’t tightly couple structure (volatile) and computation (more stable).
Use AP-F and AP-P ideas as your design notations. To execute your designs: DemeterF home page.
11/28/2018
Bern CHOOSE 2007

44. Operation of AP-F
11/28/2018
Bern CHOOSE 2007

45. AP-F for SDG AP-F: think of generic propagation first
add specific processing
11/28/2018
Bern CHOOSE 2007

46. generic propagation in SDG
push down predicate, retrieve an object satisfying predicate.
push down assignment, combine integer or formula.
push down predicate.
11/28/2018
Bern CHOOSE 2007

47. Design space for Demeter-F programsID
use builders and transformers
11/28/2018
Bern CHOOSE 2007

48. managing software development
Learn by evolving a software product: an algorithmic financial derivative trading game
Learn by working in a small team using pair programming
Learn by integrating software from the teams
11/28/2018
Bern CHOOSE 2007

49. SDG Features outcome determined by owner/buyer
type
outcome
raw material (owner)
finished product (buyer)
pay (quality)
outcome determined by environment
pay
11/28/2018
Bern CHOOSE 2007

50. SDG Features outcome determined by owner/buyer
type T
set of relations of rank k
exactly rank k, at most rank k.
size of set, e.g., 2.
use a predicate to constrain permitted relations, e.g. only OR relations -> CNF
outcome
raw material (owner)
CSP(T) formula
finished product (buyer)
assignment to variables of CSP(T) formula.
pay (quality)
satisfaction_ratio
all-or-nothing: satisfaction_ratio > price:1:0
outcome determined by environment
type
pay
11/28/2018
Bern CHOOSE 2007

51. SDG Features
synchronous: rounds in fixed order: p1 p2 p3 p1 p2 p3 …Can buy and offer when it is your turn. On each turn, must buy derivative or re-offer all existing derivatives at lower price.
asynchronous: Can buy and offer at any time. Time interval RV. In every RV, must buy derivative or re-offer all existing derivatives at lower price.
11/28/2018
Bern CHOOSE 2007

52. SDG Features
Security
11/28/2018
Bern CHOOSE 2007

53. Domain Design for SDG Domain Specific Languages Need languages for
types
raw material
finished product
configuring the game
game moves
game rules?
11/28/2018
Bern CHOOSE 2007

54. Five Deep Questions in Computing Jeannette Wing, CACM Jan. 2008
P=NP
What is computable?
What is intelligence?
What is information?
How can we build complex systems simply?
11/28/2018
Bern CHOOSE 2007

55. Deep? Speak to the foundations of the field.
Understanding and expanding the frontiers of computing.
P=NP: would have profound practical consequences, shaking the foundations of cryptography.
P!=NP: profound theoretical consequences.
11/28/2018
Bern CHOOSE 2007

56. SDG and P=NP
“Is there an assignment that satisfies at least k clauses”: is in NP and P=NP if this problem has a polynomial time algorithm.
11/28/2018
Bern CHOOSE 2007

57. What is intelligence?
Can a computer learn to play SDG well without being programmed explicitly?
What are the algorithms that the computer would invent?
Does it require intelligence to play the game well?
11/28/2018
Bern CHOOSE 2007

58. How can we build complex systems simply?
Can we build systems with simple and elegant designs that
are easy to understand, modify and evolve
yet still provide the functionality we want today and dream of for tomorrow?
11/28/2018
Bern CHOOSE 2007

59. Programming in DemeterF
Every programming technology has difficulties associated with it.
Programmers then use style rules to avoid those difficulties.
Examples:
null pointer exceptions in Java. Rule: A = [X]. -> A = XOpt. XOpt : X | EmptyX.
excessive structural dependencies in OO. Rule: Law of Demeter.
11/28/2018
Bern CHOOSE 2007

60. DemeterF Example when I write:
Integer combine(Object o, Integer n1, Integer n2) {return n1+n2;}
I intend that n1 and n2 are returned by an apply or combine method. They should not be accidental integers that happen to be parts of object o.
11/28/2018
Bern CHOOSE 2007

61. Possible style rule: Capture Avoidance Rule
The return type of a combine method should not appear in the structure being traversed (Do not return a traversed type).
Example: If the structure contains Integer parts and the purpose is to add, we could use Float to count (see project 3).
11/28/2018
Bern CHOOSE 2007

62. Implementation of Capture Avoid.: Terminal Buffer Rule with Built-ins
The terminal buffer rule (TBR) says that all terminal classes should be buffered by a class that has the terminal class as a part class. TBR improves readability of programs.
violation: City = <name> String <zip> Integer.
with TBR: City = CityName Zip. CityName = String. Zip = Integer.
11/28/2018
Bern CHOOSE 2007

63. Terminal Buffer Rule with Built-ins
CityName and Zip are made built-in classes so that the traversal does not go further.
Then we are free to use Integer as return type of combine.
11/28/2018
Bern CHOOSE 2007

64. Dispatch function of DemeterF
chooses most specific method
more information means more specific
only list arguments up to last one that is needed
11/28/2018
Bern CHOOSE 2007