1. Virtual Scientific-Community-Based Foundations for Popperian e-Science Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston 9/17/20151

2. 2 Ontology Computer Science Programming Socio-Technical System The Global Brain Mathematics Game Theory ExtensiveFormMetaGaming Mathematical Logic Dialog GamesIF Logic inspired by ScienceWISE Organization Mechanism Design

3. A claim is … information about one’s performance when interacting with another clever being in a specific domain. information about the performance of one’s program. 4/24/20113Crowdsourcing

4. Outline Theory Methods Results Conclusion Introduction Theoretical Background Methods of Exploration Results Conclusions and Future Work 9/17/20154

5. Introduction Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, tunedit.org, Renaissance, … Make SCG a part of cyber-infrastructure (e-science) to support teaching and innovation in constructive domains. SCG usage for teaching – Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis – Avatar competitions are not for teaching (but for competitive innovation) Theoretical Properties of SCG Take Home: EASY USE WITH STANDARD TOOLS group research, teaching, (intelligent) crowd sourcing 9/17/20155 Introduction Theory Methods Results Conclusion SCG = Scientific Community Game = Specker Challenge Game

6. Popper One of the philosophers of science who has had a big impact. Popper’s solution: Falsification: A claim is falsifiable if you can imagine an observation that would cause you to reject the claim. That a claim is "falsifiable" does not mean it is false; rather, that if it is false, then some observation or experiment will produce a reproducible result that is in conflict with it. 9/17/20156 Introduction Theory Methods Results Conclusion

7. What SCG helps with Build and maintain knowledge bases (sets of claims believed to be true). How to identify experts? How to decide if an answer is worthwhile? –Use scholars to choose the winners How to organize egoistic scholars to produce social welfare: knowledge base and know- how how to defend it. –The scholars try to reverse engineer the solutions of winning scholars. 9/17/20157 Introduction Theory Methods Results Conclusion

8. Abstraction from 4 Examples From a CS journal paper Insilico experiment From kaggle.com: Facebook competition From a calculus problem 9/17/20158

9. Example 1: From an Abstract of a 2005 Journal Paper An instance of a constraint satisfaction problem (CSP) is variable k-consistent if any subinstance with at most k variables has a solution. For a fixed constraint language L, r(k,L) is the largest ratio such that any variable k- consistent instance has a solution that satisfies at least a fraction of r(k,L) of the constraints. 9/17/20159

10. Example 1 From a 2005 TCS paper: Locally Consistent Constraint Satisfaction Problems by Manuel Bodirsky and Daniel Kral. Example – L = CNF – k = 1 – What is r(1,CNF)? – Claims: r(1,CNF) = 0.6, r(1,CNF) = 0.7 9/17/201510

11. Example 1: Making a game to determine r(1,CNF) Observation: claims are falsifiable playing a two person game. 9/17/201511

12. Example 2: Claim involving Insilico Experiment Claim InsilicoExperimental(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. 12Crowdsourcing4/24/2011

13. Example 2: Making a game to determine InsilicoExperimental(X,Y,q,r) Observation: claims are falsifiable playing a two person game. 9/17/201513

14. Example 3: Data mining Facebook competition from Kaggle.com: – Given a social network graph x with deleted edges and the original social network graph gs (secret, from a family X of social networks) – guess the complete social network graph y – quality(x, gs, y) = mean average precision (adapted from IR) – I claim I can achieve a mean average precision of q for social graphs in family X: DM1(X,q) for a specific reduced social graph: DM2(x,q) 9/17/201514 Introduction Theory Methods Results Conclusion

15. Example 3: Making a game to determine the optimal claims Observation: claims DM1(X,q) and DM2(x,q) are falsifiable playing a two person game. 9/17/201515

16. Example 4: Specker Claims: – Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c Example 1 – X = Conjunctive Normal Forms with various restrictions – Y(X) = Assignments to CNFs – f(x,y) = fraction of satisfied clauses in x under y – c in [0,1], e.g., c= 0.61 Example 2 (a reduction of example 1) – X = [0,1] – Y(X) = [0,1] – f(x,y)=x*y+(1-x)(1-y^2)) – c in [0,1], e.g., c=0.61 9/17/201516 Introduction Theory Methods Results Conclusion

17. Example 4: Specker Observation: claims Specker(X,Y,f,c) are falsifiable playing a two person game. 9/17/201517

18. What is the abstraction? Sets of claims Claims are falsifiable … 9/17/201518

19. 9/17/201519 RP1 PG1 claims C11 C12 C13 … SC1 SC2 SC1 RP2 PG2 claims C21 C22 C23 … SC3 SC4 SC5 SC1 SCG defines: refutation protocol interface generic rules for all playgrounds Each playground defines: domain claims language specific protocol data exchanged configuration data Playgrounds D1 D2

20. Example 1: Making a game to determine r(1,CNF) Observation: claims are falsifiable playing a two person game. defendable = !refutable – propose r(1,CNF) = 0.7 refutable – propose r(1,CNF) = 0.6 can be strengthened to r(1,CNF) = 0.61 which is defendable (refutation attempts will be unsuccessful) – propose r(1,CNF) = (sqrt(5)-1)/2 ~ 0.618 … optimum: defendable and cannot be strengthened 9/17/201520

21. Who are the scholars? Scientists Students in a class room – High school – University Members of the Gig Economy – Between 1995 and 2005, the number of self- employed independent workers grew by 27 percent. Potential employees (Facebook on kaggle.com) Anyone with web access; Intelligent crowd. 9/17/201521

22. Kaggle.com Competitions 2012 Facebook recruiting competitions – Task: Data scientist – Reward: Job – Teams: 197 Heritage Health Prize – Task: Hospital admissions – Reward: $ 3 million – Teams: 1118 Chess ratings – Elo versus the Rest of the World – Task: Predict outcome of chess games – Reward: $ 617 – Teams: 257 9/17/201522

23. Kaggle.com Competitions 2012 Eye Movements Verification and Identification – Task: Identify people – Reward: Kudos – Teams: 51 EMC Data Science Global Hackathon – Task: Air Quality Prediction – Reward $ 7030 – Teams: 114 9/17/201523

24. What Scholars think about! If I propose claim C, what is the probability that – C is successfully refuted – C is successfully strengthened If I try to refute claim C, what is the probability that I will fail. If I try to strengthen claim C, what is the probability that I will fail? Scholars are free to invent; game rules don’t limit creativity! 249/17/2015 Introduction Theory Methods Results Conclusion

25. Degree of automation with SCG(X) 25 no automation human plays full automation avatar plays degree of automation used by scholar some automation human plays 0 1 more applications: test constructive knowledge transfer to reliable, efficient software avatar Bob scholar Alice 9/17/2015 Introduction Theory Methods Results Conclusion

26. Organizational Problem Solved How to design a happy scientific community that encourages its members to really contribute. Control of scientific community – tunable SCG rules – Specific domain, claim definition to narrow scope. 9/17/201526 happy = can be creative, can thrive, have opportunity to learn, not ignored

27. Playground defines – what is wanted, e.g., an algorithm S in a particular domain (inputs/outputs) – evaluation, e.g., how S is evaluated (quality) – claims, e.g., what kind of claims can be made about S (expression with quantifiers) A playground defines WHAT is desired and the scholars/avatars define the HOW.

28. Theory Extensive Form Representation of Game Properties – Community Property: All faulty actions can be exposed. – SCG Equilibrium – Convergence to optimum claim 9/17/201528 Introduction Theory Methods Results Conclusion

29. Extensive-form representation 1.the players of a game: 1 and 2 2.for every player every opportunity they have to move 3.what each player can do at each of their moves 4.what each player knows for every move 5.the payoffs received by every player for every possible combination of moves 9/17/201529 Introduction Theory Methods Results Conclusion

30. Large Action Spaces Thick arrows mean: select from a usually large number of choices 1 2 9/17/201530

31. Refutation Protocol Collects data given to predicate p. Alternates. refute(C,proposer,other) p(C, …)?(1,-1):(-1,1) claimpayoff for proposer if p true (defense) payoff for other if p true (defense) payoff for other if p false (refutation) payoff for proposer if p false (refutation) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense. 9/17/201531 Introduction Theory Methods Results Conclusion

32. 1 propose claim C from Claims 2 refute(C,1,2) p(C, …)?(1,-1):(-1,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(1,-1):(-1,1) 9/17/2015 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) Introduction Theory Methods Results Conclusion p(C’, …)?(-1,1):(1,-1) p(C, …)?(0,0):(1,-1) 32 SCG Core

33. Game Rules for Playground All objects exchanged during protocol must be legal and valid. Each move must be within time-limit. Scholar who first violates a playground rule, loses. 9/17/201533 Introduction Theory Methods Results Conclusion

34. 4/24/2011Crowdsourcing34 goodbad Logic with Soundness claims sentences not just true/false claims, but optimum/non-optimum claims: good: true/optimum bad: false/non-optimum Introduction Theory Methods Results Conclusion

35. bad 4/24/2011Crowdsourcing35 good Scientific Community Game Logic with Community Principle agreed by two scholars disagreed by two scholars there exists a two-party certificate to expose misclassification claims sentences Introduction Theory Methods Results Conclusion

36. Comparison Logic and SCG Logic sentences – true – false proof for being true – proof system, checkable – guaranteed defense proof for being false – proof system, checkable – guaranteed refutation Universal sentences Scientific Community Game sentences = claims – good – bad evidence for goodness – defense, checkable – uncertainty of defense evidence for badness – refutation, checkable – uncertainty of refutation Personified sentences 4/24/2011Crowdsourcing36

37. Community Property For every faulty decision action there exists an exposing reaction that blames the bad decision. – Reasons: We want the system to be egalitarian. – It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. Faulty decisions must be exposable. It may take effort. 9/17/201537 Introduction Theory Methods Results Conclusion

38. Methods of Exploration Developed Platform SCG Court = Generator of teaching/innovation playgrounds – http://sourceforge.net/p/generic-scg/code-0/11 http://sourceforge.net/p/generic-scg/code-0/11 0/tree/GenericSCG/ – Developed numerous playgrounds for avatars. Developed Algorithms Course using Piazza based on SCG Court experience – role of scholar played by humans – piazza.com: encourages students to answer each other’s questions. 9/17/201538 Introduction Theory Methods Results Conclusion

39. Avatar Interface AvatarI – public List propose(List forbiddenClaims); – public List oppose(List claimsToBeOpposed); – public InstanceI provide(Claim claimToBeProvided); – public SolutionI solve(SolveRequest solveRequest); from http://sourceforge.net/p/generic-scg/code- 0/110/tree/GenericSCG/src/scg/scg.beh Introduction Theory Methods Results Conclusion 9/17/201539

40. Instance Interface (Domain) InstanceI – boolean valid(SolutionI solution, Config config); – double quality(SolutionI solution); Introduction Theory Methods Results Conclusion 9/17/201540

41. InstanceSet Interface (Domain) InstanceSetI – Option belongsTo(InstanceI instance); – Option valid(Config config); }} 9/17/201541

42. Protocol Interface ProtocolI – double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); – ProtocolSpec getProtocolSpec(); – boolean strengthenP(Claim oldClaim, Claim strengthenedClaim); 9/17/201542

43. Claim Class, for all playgrounds Claim – public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence) 9/17/201543

44. Protocol Library ExistsForAll.java ForAllExists.java Renaissance.java AsGoodAsYou.java Survivor.java 9/17/201544 Introduction Theory Methods Results Conclusion

45. Second Method: Piazza Experience Gale-Shapley We propose that, for all integers n > 0, the maximum iterations the Gale-Shapely algorithm with n men and n women can produce is n(n-1)+1. Note: Thus far, the inputs used for all other claims arrives at only (n(n+1))/2. 9/17/201545

46. Piazza Experience Leaf Covering: Improved running time from quadratic to constant time. 9/17/201546

47. Results Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, Operations Research Competitions, tunedit.org, http://eterna.cmu.edu/ … SCG usage for teaching using forum – Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis Avatar competitions are not for teaching (but good for competitive innovation) Theoretical Properties of SCG 9/17/201547 SCG = Scientific Community Game = Specker Challenge Game Introduction Theory Methods Results Conclusion

48. Competition tuning: minimum For each scholar – count claims that were successfully opposed (refuted or strengthened) encourages strong claims gather information from competitors for free – count claims that were not successfully agreed Good for teaching – students want minimum competition – good students want to build social capital and help weaker students 9/17/201548 Introduction Theory Methods Results Conclusion

49. Piazza Results Do not give hints at solutions. This significantly decreased the amount of discourse taking place. 9/17/201549 Introduction Theory Methods Results Conclusion

50. Conclusions and Future Work We propose a systematic gamification of teaching STEM domains: – Design an SCG playground where the winning students demonstrate superior domain knowledge. 9/17/201550 STEM = Science, Technology, Engineering, and Mathematics Introduction Theory Methods Results Conclusion

51. Gamification of Software Development for Computational Problems Want reliable software to solve a computational problem? Design an SCG playground where the winning team will create the software you want. playground design = requirements – Programming the Global Brain socio-technical system (playground) will produce solution to requirements. Crowdsourcing514/24/2011 Introduction Theory Methods Results Conclusion

52. Conclusions Flexible use of SCG using a forum environment with threads and replies using optimization playgrounds is productive: – teams took turns leapfrogging each other – reached state-of-the-art and even improved it SCG has desirable theoretical properties. – faulty decision –> exposing reaction – equilibria – convergence to optimum claim 9/17/201552 Introduction Theory Methods Results Conclusion

53. Future Work Make SCG part of cyber-infrastructure (e-science) both for avatars and human scholars. Polish SCG Court – The administrator software needs to be very reliable (to avoid cheating by avatars). – Playground development and testing needs tool support. Further develop SCG with forum software – Playground design defines requirements for know-how. – Hierarchical playgrounds: partitioning into balanced groups. – Restart playground after publishing all current ideas in playground (if optimum is not yet reached). 9/17/201553 Introduction Theory Methods Results Conclusion

54. Links / Questions SCG Home – http://www.ccs.neu.edu/home/lieber/evergreen/specker/scg- home.html http://www.ccs.neu.edu/home/lieber/evergreen/specker/scg- home.html Piazza page for Algorithms – http://piazza.com/class#winter2012/cs4800/0 http://piazza.com/class#winter2012/cs4800/0 Algorithms Home – http://www.ccs.neu.edu/home/lieber/courses/algorithms/cs4800/sp1 2/course-description.html http://www.ccs.neu.edu/home/lieber/courses/algorithms/cs4800/sp1 2/course-description.html Algorithms Feedback – http://www.ccs.neu.edu/home/lieber/courses/algorithms/cs4800/sp1 2/feedback/ http://www.ccs.neu.edu/home/lieber/courses/algorithms/cs4800/sp1 2/feedback/ SCG Court Source – http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/ http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/ 9/17/201554

55. The End More Questions? 9/17/201555

56. Extra slides 9/17/201556

57. Essence of Game Rules without Payoff scholars: 1, 2 LifeOfClaim(C) = propose(1,C) followed by (oppose(1,2,C)|agree(1,2,C)). oppose(1,2,C) = (refute(1,2,C)|strengthen(1,2,C,C’)), where stronger(C,C’). strengthen(1,2,C,C’) = !refute(2,1,C’). agree(1,2,C) = !refute(2,1,C) 9/17/201557 blamed decisions: propose(1,C) refute(1,2,C) strengthen(1,2,C,C’) agree(1,2,c)

58. 1 propose claim C from Claims 2 refute(C,1,2) p(C, …)?(1,-1):(-1,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(1,-1):(-1,1) 9/17/2015 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful Introduction Theory Methods Results Conclusion p(C’, …)?(-1,1):(1,-1) u:1 2s:1 2 u:1 2 p(C, …)?(0,0):(1,-1) s:1 2u:1 2 58

59. 1 propose claim C from Claims 2 refute(C,1,2) p(C, …)? (0,0) :(0,1) 1 scholar 2 scholar strengthen attempt C’ => C refute(C’,2,1) agree attempt C refute(C,2,1) 9/17/2015 refute attempt C refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful p(C’, …)?(0,1): (0,0) u:1 2s:1 2 u:1 2 p(C, …)?(0,0): (1,0) s:1 2u:1 2 Low competition Introduction Theory Methods Results Conclusion 59

60. Conclusions for Teaching Transition – refute: (1,-1):(-1,1) -> (0,0) :(0,1) – strengthen: (-1,1):(1,-1) -> (0,1): (0,0) – agree: (0,0):(1,-1) -> (0,0): (1,0) creates better playgrounds for learning by lowering competition and increasing teaching between scholars. 9/17/201560 Introduction Theory Methods Results Conclusion

61. Claims Protocol. Defines scientific discourse. Scholars make a prediction about their performance in protocol. Predicate that decides whether refutation is successful. Refutation protocol collects data for predicate. As a starter: Think of a claim as a mathematical statement: EA or AE. – all planar graphs have a 4 coloring. 9/17/201561

62. More examples of Protocols Let f(x,y)=x*y+(1-x)(1-y^2)). Alice claims Math(0.61): Bob constructs an x in [0,1] and Alice constructs a y in [0,1], and Alice guarantees that f(x,y)> 0.61. True claim but can be strengthened to 0.618. Alice claims Solar(RawMaterials,m,0.61). Bob constructs raw materials r in RawMaterials and Alice constructs a solar cell s in Solution from r using money m and so that efficiency(s)> 0.61. 9/17/201562 Introduction Theory Methods Results Conclusion

63. Questions received In learning game, give credit to all contributors, not just final one (DARPA 10 ball challenge) Predicate logic -> SCG: make explicit Playground design: involve competitors 9/17/201563

64. Questions Credit – first time the best claim is made – linear order by time – linear order by strength (quality) 9/17/201564

65. New insight Need to know very little about refutation protocol. – collect data, what is available when is not important – evaluate predicate with collected data 9/17/201565

66. What is a loose collaboration? Scholars can work independently on an aspect of the same problem. Problem = decide which claims in playground to oppose or agree with. How is know-how combined? Using a protocol. – Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. – Bug reports that need to be addressed and corrections. 9/17/201566 Playground = Instantiation of Platform Introduction Theory Methods Results Conclusion

67. Example: Independent Set Alice = proposer, Bob = other. Protocol / claim: AtLeastAsGood. Alice claims to be at least as good as Bob at IS. – Bob provides undirected graph G. – Bob computes independent set sB for G (secret). – Alice computes independent set sA for G. – Alice wins, if size(sA) >= size(sB) (= p(sA,sB)). 9/17/201567

68. Specker Claims: – Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c Example 1 – X = Conjunctive Normal Forms with various restrictions – Y(X) = Assignments to CNFs – f(x,y) = fraction of satisfied clauses in x under y – c in [0,1], e.g., c= 0.61 Example 2 (a reduction of example 1) – X = [0,1] – Y(X) = [0,1] – f(x,y)=x*y+(1-x)(1-y^2)) – c in [0,1], e.g., c=0.61 9/17/201568 Introduction Theory Methods Results Conclusion

69. Kaggle.com Facebook competition: – X = Social Network Graph with deleted edges, Original Social Network Graph (secret) – Y(X) = estimated complete Social Network Graph – quality(x,y) = mean average precision adapted from IR 9/17/201569 Introduction Theory Methods Results Conclusion

70. Simpler talk Introduction: parameterized models of scientific communities Theory 9/17/201570

71. Abstraction from 4 Examples From a CS journal paper Insilico experiment From kaggle.com: Facebook competition From a calculus problem 9/17/201571

72. Example 1: From an Abstract of a 2005 Journal Paper An instance of a constraint satisfaction problem (CSP) is variable k-consistent if any subinstance with at most k variables has a solution. For a fixed constraint language L, r(k,L) is the largest ratio such that any variable k- consistent instance has a solution that satisfies at least a fraction of r(k,L) of the constraints. 9/17/201572

73. Example 1 From a 2005 TCS paper: Locally Consistent Constraint Satisfaction Problems by Manuel Bodirsky and Daniel Kral. Example – L = CNF – k = 1 – What is r(1,CNF)? – Claims: r(1,CNF) = 0.6, r(1,CNF) = 0.7 9/17/201573

74. Example 1: Making a game to determine r(1,CNF) Observation: claims are falsifiable playing a two person game. 9/17/201574

75. Example 2: Claim involving Insilico Experiment Claim InsilicoExperimental(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. 75Crowdsourcing4/24/2011

76. Example 2: Making a game to determine InsilicoExperimental(X,Y,q,r) Observation: claims are falsifiable playing a two person game. 9/17/201576

77. Example 3: Data mining Facebook competition from Kaggle.com: – Given a social network graph x with deleted edges and the original social network graph gs (secret, from a family X of social networks) – guess the complete social network graph y – quality(x, gs, y) = mean average precision (adapted from IR) – I claim I can achieve a mean average precision of q for social graphs in family X: DM1(X,q) for a specific reduced social graph: DM2(x,q) 9/17/201577 Introduction Theory Methods Results Conclusion

78. Example 3: Making a game to determine the optimal claims Observation: claims DM1(X,q) are falsifiable playing a two person game. Claim DM2(x,q) is falsifiable when the secret is revealed. 9/17/201578

79. Example 4: Specker Claims: – Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c Example 1 – X = Conjunctive Normal Forms with various restrictions – Y(X) = Assignments to CNFs – f(x,y) = fraction of satisfied clauses in x under y – c in [0,1], e.g., c= 0.61 Example 2 (a reduction of example 1) – X = [0,1] – Y(X) = [0,1] – f(x,y)=x*y+(1-x)(1-y^2)) – c in [0,1], e.g., c=0.61 9/17/201579 Introduction Theory Methods Results Conclusion

80. Example 4: Specker Observation: claims Specker(X,Y,f,c) are falsifiable playing a two person game. 9/17/201580

81. What is the abstraction? Sets of claims Claims are falsifiable … 9/17/201581

82. Example 1: Making a game to determine r(1,CNF) Observation: claims are falsifiable playing a two person game. defendable = !refutable – propose r(1,CNF) = 0.7 refutable – propose r(1,CNF) = 0.6 can be strengthened to r(1,CNF) = 0.61 which is defendable (refutation attempts will be unsuccessful) – propose r(1,CNF) = (sqrt(5)-1)/2 ~ 0.618 … optimum: defendable and cannot be strengthened 9/17/201582

83. What we get Engaged software developers – let them produce software that models an organism that fends for itself in a real virtual world while producing the software we want. Have fun. Focus them. – let them propose claims about the software they produce. Reward them when they defend their claims successfully or oppose the claims of others successfully. Crowdsourcing83 Clear FeedbackSense of Progress Possibility of Success Authenticity 4/24/2011

84. Reinterpret Refutation Refutation leads to successful strengthening or successful agreement. 9/17/201584