1. Evaluating Legal Argument Instruction with Graphical Representations using LARGO Niels Pinkwart, Vincent Aleven, Kevin Ashley, and Collin Lynch Clausthal University of Technology Carnegie Mellon University University of Pittsburgh

2. 2 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Outline LARGO (Legal ARgument Graph Observer): ITS to engage students in analyzing & reflecting about examples of expert Socratic reasoning  US Supreme Court Oral Arguments as learning resources  Diagrams to visualize argument as hypothesis testing: analysis and feedback  Evaluation of LARGO

3. 3 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO US Supreme Court Oral Arguments  Important part of decision process  Attorneys propose a decision rule (“test”) to determine how to decide a case  Justices challenge these tests by posing hypothetical scenarios (meaning, consistency with past cases, legal and policy implications)

4. 4 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO An Example Case  Example: Kathy Keeton v. Hustler Magazine, 465 U.S. 770 (1984)  Facts: Kathy Keeton sued Hustler Magazine for libel in US District Court in New Hampshire. Hustler is an Ohio corporation with its principal place of business in California. Ms. Keeton was not a New Hampshire resident and had almost no ties there. Each month, Hustler sold about 15,000 issues in New Hampshire, which was only state where Keeton was not time-barred under a statute of limitations.  Question: Did the courts in New Hampshire have personal jurisdiction over Hustler Magazine?  Conflicting principles: Fairness to defendant vs. right of state to redress in-state injury.

5. MR. GRUTMAN: The synthesis of those cases holds that where you have purposeful conduct by a defendant directed at the forum in question and out of which conduct the cause of action arises or is generated that satisfies the formula of those minimum contacts which substantial justice and reasonable fair play make it suitable that a defendant should be hailed into that court and be amenable to suit in that jurisdiction. JUSTICE: Would it apply in Alaska? MR. GRUTMAN: It would apply, Mr. Justice Marshall, wherever the magazine was circulated. It would apply in Honolulu if the publication were circulated there. It would apply theoretically and, I think, correctly wherever the magazine was circulated, however many copies were circulated. JUSTICE: What if the publisher had no intention of ever selling any magazines in New Hampshire? MR. GRUTMAN: A very different case, Mr. Justice White. … because in that case you could not say, as you do here, that you have purposeful conduct. There you have to look for other -- I think your phrase is affiliating circumstances, other connections, judicially cognizable ties -- Example: Tests and Hypotheticals Test Hypo Response: Distinguish, Modify Hypo Response: Analogize

6. 6 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Educational Value  Training in skills of argument important part of legal argumentation  Transcripts realistic and valuable examples of expert Socratic reasoning  Analysis of transcripts: can help learning these skills provides opportunities for reflection  But: complex material, hard to understand!

7. 7 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO A LARGO Argument Diagram LARGO Argument Diagram Palette

8. 8 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Intelligent Support: How to interpret diagrams?  Problem 1: legal argumentation is an ill- defined domain Different views possible Difficult to define “correctness” in this interpretive field  Problem 2: arguments consist of natural language texts – NLP based approaches seem error-prone here  LARGO approach: Attempt to find characteristics in argument diagram Weaknesses (areas of potential problems) or Opportunities for reflection

9. 9 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Diagram characteristics  Context: links between graph elements and important passages in the transcript.  Structure: graph portions that correspond to pre-defined “argument patterns”.  Content: adequateness of students` free-text descriptions in diagram boxes. Example: No “test” elements linked to transcript passage which contains important test Detected by graph grammar. Example: Diagram contains hypothetical not related to any fact or test element. Detected by graph grammar. Example: A description of a proposed test in a “test” diagram element is of poor quality. Detected by collaborative filtering.

10. 10 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Detection of Content Weaknesses through Collaborative Filtering Student A Student B Student C A rates the test descriptions provided by B and C after having entered his own.

11. 11 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Feedback: Self Explanation Prompts  Principle: Use detected characteristics for self explanation prompts  Research results: SE beneficial for learning, effective metacognitive strategy  Offer opportunities for reflection about specific parts of Socratic reasoning examples: the presumably weak parts of student’s argument analysis, and the good parts still worth reflection

12. 12 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Feedback Example

13. 13 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO LARGO v. Cognitive Tutors  Study of expert argument examples  No cognitive model of diagram construction  LARGO does not insist that students create a “correct graph” Because of ill-structure, it may not be feasible to identify a comprehensive set of correct graphs Getting the diagram right is not the main thing: we want students to reflect on the quality of the arguments Student may be more deliberate and thoughtful without corrective feedback

14. 14 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Experiment (Fall 2006) Goal: Evaluate LARGO, compared to standard note taking Hypothesis: Using LARGO improves understanding hypothetical reasoning method, ability to recognize/reason about examples in near and far transfer legal domains. Participants: 28 volunteer law students in 1st-semester Legal Process course. Task: Read SCOTUS oral arguments; represent hypothetical reasoning. Experimental condition: Use LARGO graphical argument representation and feedback to identify/relate elements of hypothetical reasoning. Control condition: Same oral arguments and focus on hypothetical reasoning, but use tool with text-based word-processing and highlighting (no feedback). Procedure: Over a four-week period: 1. (2h): Pre-test & tool introduction 2. & 3. (2x2h): Apply tools to Burnham & Burger King (personal jurisdiction) arguments 4. (3h): Post-test

15. 15 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Control Condition Tool

16. 16 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Pre/post test measures Hypothetical reasoning √√General questions re hypotheticals Test general understanding of role of hypotheticals √√Hypothetical reasoning about tennis club policy Test hypothetical reasoning skills in everyday context √Near transfer of hypothetical reasoning (Keeton, personal jurisdiction) Study new SCOTUS oral argument transcript without tool, answer questions about the relation between tests and hypotheticals √Far transfer of hypothetical reasoning (Sony, copyright) PrePostQuestion typeKnowledge/skills tested Legal argumentation √√LSAT-type legal argument questions Test legal argumentation skills in a context that does not pre-suppose any specific legal knowledge Substantive legal knowledge √√Personal jurisdiction questionsTest students ’ knowledge of personal jurisdiction √Copyright questionTest students ’ knowledge of copyright law

17. 17 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Overall test scores

18. 18 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Test scores by item

19. 19 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Divide sample by LSAT score GroupLSAT scoreNAvg Scores DiagramTextPre-Test*Post-Test* HIGH162-16526.660.622 MED16181.615.549 LOW158-16055.599.544 *p<.05

20. 20 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Post-test scores in Low LSAT Group *p<.05 *

21. 21 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Post-Hoc analysis: Classify questions in terms of argument model. One category: evaluating hypotheticals with respect to a proposed test Assume that Mr. Grutman’s proposed test is as follows: …. The following hypotheticals either were or could have been posed in the oral argument. Each of them is followed by four explanations why the hypothetical is or is not problematic for Mr. Grutman’s proposed test. For each hypothetical, please check ALL of the explanations that are plausible. [First hypothetical:] “ what if the plaintiff was totally unknown in the jurisdiction before the magazine was circulated?” oThe hypothetical is problematic for Mr. Grutman’s proposed test. The decision rule applies by its terms, but arguably the publisher should not be subject to personal jurisdiction in the state under those circumstances. oThe hypothetical is not problematic for Mr. Grutman’s proposed test. … o… [Second hypothetical:] … Result: In near- and far-transfer problem questions of this type, LOW and MED students in experimental group scored higher than those in the control group, and the difference was significant (t(1,17)=2.73, d=1.00, p<.05, 1-sided).

22. 22 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO LARGO advice usage  LARGO feedback is on-demand  Student requested help 10.1 times per transcript (1 hr). LOW 12.3 MED 6.2 HIGH 17.9  Advice usage did not decrease over time: 7.3 and 9.8 in the first 2 transcripts 12.2 and 8.6 in the last 2 transcripts  In 75% of requests, students selected one of the three short hints and read detailed feedback  Conclusion: Feedback was appreciated

23. 23 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO Conclusion & Outlook  LARGO: an ITS for the ill-defined domain of legal argumentation. Uses examples of expert Socratic reasoning to “ask good questions” about student- created argument diagrams.  Experiment showed LARGO did not lead to better learning across the whole sample. But LARGO helps lower ability students learn argumentation skills and understanding the role of hypotheticals.  Ongoing and future work: Predictive value of argument graphs Extend LARGO so students make arguments Collaborative approach to argument construction

24. 24 Niels Pinkwart AIED 2007 Evaluating Legal Argument Instruction with Graphical Representations using LARGO  Please visit our project website: http://www.cs.cmu.edu/~hypoform http://www.cs.cmu.edu/~hypoform  Email: niels.pinkwart@tu-clausthal.de niels.pinkwart@tu-clausthal.de