Privileged & Confidential Ready or Not, Here it Comes…. Why It’s Likely that You’ll be Employing Predictive Coding Technology In 2013 Texas Lawyer In-House Counsel Summit April 4, 2013 © DiscoverReady 2013

Privileged & Confidential Introductions Moderator: Amy Hinzmann, SVP, DiscoverReady LLC Panel: Thomas Gottsegen, Lead Legal Counsel, Statoil Jim Wagner, CEO, DiscoverReady LLC © DiscoverReady 2013

Privileged & Confidential Agenda and Goals What is predictive coding? Key terms and concepts you need to know Case law overview Potential predictive coding use cases Issues most likely to be negotiated with opposing counsel Steps you should take to be prepared for your first predictive coding matter © DiscoverReady 2013

Privileged & Confidential Warning This Next Slide is Going to Freak You Out….. © DiscoverReady 2013

Privileged & Confidential Welcome to the World of Predictive Coding…. F-score (F β ) is a “simple” calculation of how well your predictive coding is performing calculated as (1 + β 2 ) x Precision x Recall ÷ (β 2 x Precision + Recall) where the Beta refers to the relative weighting of precision and recall, and a Beta of one weights the value of precision and recall equally. © DiscoverReady 2013

Privileged & Confidential What is Predictive Coding? Many variations and processes, but the general principle in layman’s terms is … “A combination of human and automated processes in which humans review a subset of documents, the computer analyzes those documents and decisions, and then the computer analyzes the remaining set of documents and recommends decisions for them.” © DiscoverReady 2013

Privileged & Confidential Where We Currently Stand Single most discussed topic in the industry At least 6 cases authorizing or discussing use of predictive coding Regulatory agencies actively pursuing predictive coding solution Abundance of scholarly commentary: –Grossman/Cormack Study –Significant discussion by Software Providers (Equivio / Relativity / Recommind) –15 Sessions at LegalTech 2013 primarily focused on predictive coding Most formal commentary and “buzz” focuses on technology as substitute for human review Use cases suggest implementations focus on “Predictive Culling”™ in lieu of search terms, with manual review prior to production No standards or safe harbors Not necessarily a cost saver © DiscoverReady 2013

Privileged & Confidential Key Terms and Concepts You Need to Know Richness (prevalence of relevant data) Precision Recall Confidence level F-score Sample size Margin of error Control set Training set © DiscoverReady 2013

Privileged & Confidential Understanding the Importance of Richness The prevalence of likely relevant documents (or richness) of a data set impacts every aspect of a predictive coding process –Efficacy of overall process –Training –Statistical Validation –Cost Low Richness = More Review = Higher Costs Strategies for identifying and overcoming low richness –Statistical sampling to determine richness as part of “evaluation” process –“Seeding” to increase richness –Custodian selection –Identification and use of exemplar documents What should you disclose regarding your seeding efforts? © DiscoverReady 2013

Privileged & Confidential Sample Size Not Impacted by Total Population

Privileged & Confidential Practical Examples of the Importance of Richness Sampling a randomly selected group of documents at the outset of a matter can provide valuable information regarding the potential efficacy of a party’s predictive coding efforts. However, the volume of data that must be sampled to gain an adequate level of confidence in the richness of the data set depends on both the volume of likely relevant documents in the data set and the desired margin of error associated with the results 95% Confidence Level © DiscoverReady 2013

Privileged & Confidential Practical Examples of the Importance of Richness Similarly, a party will statistically sample the “results” of their predictive coding efforts to determine the “precision” and “recall” of the predictive coding process Again, the volume of data that must be sampled to validate the precision and recall of the predictive coding efforts turn on the richness of the data set and the desired margin of error. Estimated Recall of 80% Estimated Precision of 65% © DiscoverReady 2013

Privileged & Confidential Distinguishing Relevance vs. Responsiveness In the context of Predictive Culling™ (using predictive coding to cull data in lieu of search terms), it’s critical to distinguish between “likely relevance” for culling and “responsiveness” to discovery requests –Parties frequently take a broad view of relevance for purposes of “training the system” and identifying the data set that will be subject to manual review by producing party –The producing party often takes a much narrower view of responsiveness for purposes of review and production, focusing on responsiveness to specific discovery requests rather than general concept of relevance –It is critical that both parties understand that only a subset of the documents identified as likely relevant during culling actually will be produced as responsive documents © DiscoverReady 2013

Privileged & Confidential What Have the Courts Said? Da Silva Moore v. Publicis Groupe (11 Civ S.D.N.Y) –Parties agreed to use of predictive coding technology to cull data in lieu of search terms –Dispute over implementation EORHB, Inc. v. HOA Holdings, LLC (No 7409 Del. Chancery Court) –Court issued sua sponte order dictating that parties would use predictive coding even though neither party requested it –Also ordered parties to agree on single vendor to house documents Global AeroSpace, Inc. et al. v Landow Aviation, LP (CL Loudoun County VA Circuit Court) –Virginia State Court Judge ordered parties to employ predictive coding over objections of Plaintiff –According to court and counsel, technology successfully deployed resulting in cost savings and increased efficiency © DiscoverReady 2013

Privileged & Confidential What Have the Courts Said? In Re Actos (Pioglitazone) Products Liability Litigation (MDL No. 6:11-md-2299, W.D. La.) –Parties reached agreement on joint e-discovery protocol utilizing predictive coding to cull data Kleen Products, LLC et al. v. Packaging Corporation of America, et al. (No N.D. Ill) –Plaintiffs requested that Court order Defendants to use predictive coding technology to cull data following substantial productions based on search term based culling –Ultimately ceded to Judge Nolan’s “suggestion” that parties develop acceptable keyword search strategy Gabriel Technologies Corp et al. v, Qualcomm Inc., et al. (Case No 08CV1992 S.D. Cal) –In landmark cost shifting case, court ordered Plaintiffs to reimburse defendants— among total fees of $12 million—$2.82 million for “computer assisted algorithm driven document review” of 12 million records –Also awarded $400,000 for contract attorney costs for reviewing resultant data –While award was based on bad faith of plaintiff, demonstrates potential costs of predictive coding solutions © DiscoverReady 2013

Privileged & Confidential Potential Use Cases…. ApproachProsCons In lieu of keyword searching  Keyword searching frequently grossly inexact  Generates higher precision and higher recall rates  Results can be used for review prioritization reduces review of irrelevant data  Technology likely costs substantially more than keyword searching  Training can take more time than creation of keyword searches  Can generate more documents for review or production than keyword searching (not necessarily a negative)  Complexity of negotiating cut-off point with opposing counsel  Must be prepared to argue proportionality in terms of tradeoff of precision and recall  All documents above cutoff point still have to be reviewed (identical to keyword searching) Review Prioritization  Accelerated access to most relevant data  Ability to direct most relevant data to specific reviewers  Defer review of irrelevant data  Potential use for early case assessment  Significant technology costs  Typically defers review, but does not eliminate review of less relevant data  Higher “relevance” score does not necessarily translate to “hot doc” In lieu of human review (post culling)  More consistent decisions  Potentially eliminate review of irrelevant files (low scores)  Significantly reduce costs  Risk of production of data that has not been reviewed (privilege, privacy and confidentiality)  Opposing counsel may not approve use of predictive coding decision of “irrelevant” if the document contains a keyword that has been stipulated Inbound productions  Rapid access to most relevant data  Potential to use outbound data set and decisions for training purposes  Defer or eliminate review of low relevance files  Technology expense  May not find “hot documents” Produce all to regulatory agency  Allows regulators complete access to data set  Particularly beneficial when there is low confidence in efficacy of keyword searching  Requires privilege, privacy and confidentiality screening of entire data set  Many clients require review of any document produced prior to production  Harder to find same documents that regulators may identify as important © DiscoverReady 2013

Privileged & Confidential Comparing Your Options ApproachRiskPotential for Cost SavingsPrevalence In lieu of keyword searching Low  Should yield better results than many keyword searches  Can measure efficacy (precision / recall) Mixed  Technology costs could be higher  May lower cost by eliminating review of irrelevant data  May increase cost by generating more files in total requiring review High  Most common public commentary is inadequacy of poorly constructed search terms  Predictive coding will likely beat keyword searches “head to head” and could become a default standard for culling data Review Prioritization Lowest  Absent combination with another approach, producing party will still review entire data set for relevance, privilege, and confidentiality prior to production Mixed  Technology costs could be higher  May lead to hot docs quickly  Typically only defers review, does not eliminate the review of irrelevant files Highest  Very similar to traditional methods of “clustering” review set based on likely similarity between the documents  Accepted process that provides increased accuracy and efficiency In lieu of human review (post culling) High  Risk of production of data that has not been reviewed (privilege, privacy and confidentiality) High  Cost savings of 50%-75% (or more) for every computer decision vs. human decision Very low  Clients uncomfortable producing data that has not been reviewed Inbound productions Low  No worries of inadvertent productions or material omissions  Possible to miss hot docs High  Can significantly reduce volume of data reviewed  Client can make risk/reward decision of not reviewing low ranking documents Medium  Represents a “baby step” into predictive coding Produce all to regulatory agency High  Could be difficult to identify same documents as regulators identify  Risk of production of privileged, private or confidential info None  Creates burden to analyze and review entire data set Low (but increasing)  Traditionally regulatory agencies have avoided predictive coding  Likely increasing as evidenced by SEC’s recent agreement to bring Recommind in house © DiscoverReady 2013

Privileged & Confidential Issues Most Likely to be Negotiated Degree of collaboration/transparency Software Pre-training data culling (de-dupe/de-Nist/etc.) Training methodology (richness of data directly impacts training time and cost) Addressing document families (analysis is usually on individual documents) Who reviews the training set Privilege, privacy and confidentially screening Negotiating a cut-off point for irrelevant data Negotiating acceptable margin of error and confidence level Production format © DiscoverReady 2013

Privileged & Confidential 3 Places Where Actos Went Wrong 1.Use of 4 custodians for training set –Documents possessed by the selected custodians may not be representative of the documents possessed by the larger custodian group, making training at best incomplete and at worst inappropriate for the larger data set 2.Establishing target confidence levels and margin of error without any knowledge regarding potential richness of data set –The volume of data required to be reviewed to establish specific confidence levels and margins of error will depend in large part on the richness of the data set. 3.Collaborative joint training –This is certainly the largest issue with the Actos protocol, as it would require 3 subject matter experts from each party to review a control set, training set, and samples of the resulting work product. –In addition to providing unprecedented levels of participation in the discovery process, likely will drive large inefficiencies and drive significant costs associated with training process both because of the extensive level of collaboration required and the lack of specific protocols to resolve disputes among the subject matter experts © DiscoverReady 2013

Privileged & Confidential Options For Your Negotiations Actos Other Approaches DiscoverReady Best Practice Software Equivio RelevanceGlobal Aerospace—OrcaTech Da Silva Moore—Recommind Predictive coding should be more about process than software. As such, the selection of software will depend on your goals and objectives in a particular matter Data Culling Data population de-duplicated and de- Nisted with spam, commercial (blasts) and no-text files removed Global Aerospace—de-duplication and de- Nisting only De-duplication and de-Nisting, with evaluation of removing commercial blasts, highly unlikely to be responsive based on sender domain, and no-text files Training methodology Training to occur on major issues only (Responsive/Non-Responsive/Skip) Equivio—in addition to major categories, contemplates training on individual issues to organize review Grossman—contemplates training on both major issues and four individual issues Training on major issues with review prioritization based on likely relevance. Must distinguish between relevance for purposes of culling and relevance for purposes of production Depending on the richness of the data set and the scope of the responsiveness determinations, consideration should be given to potentially “seeding” the data set to include a higher volume of responsive documents than might otherwise appear across the data set (hopefully diminishing the training required due to low richness an very targeted responsiveness criteria) Who reviews the training set Collaborative, which Actos defines as having 3 representatives from each party review all training documents and achieve consensus across each document in training set Global Aerospace—producing party to train individually and then provide entire training set (absent privileged and sensitive NR documents) to plaintiff Da Silva—producing party to train using keyword hits (based on PC tool) and provide hit counts and documents containing hits to plaintiff Equivio—collaborative, which is defined as having 3 reviewers from producing party review and reach consensus on each document Grossman—Individual decisions with single “gold standard” assessor for QC Producing party trains using individual subject matter experts with QC by “gold standard” assessor. Plaintiff’s validation of training to be accomplished by production of precision and recall scores, and samples of documents defined to be non-responsive beneath proposed data cutoff. A producing party should resist requests from the opposing party to participate in training or otherwise be allowed to review the entirety of the training sets to avoid broad interpretation of relevance beyond that established by the producing party unless having similar access to the opposing party’s data likely would prove beneficial. © DiscoverReady 2013

Privileged & Confidential Options For Your Negotiations Actos Other Approaches DiscoverReady Best Practice Addressing Document Families Not addressed Training is performed on individual documents without taking family members into account for purposes of determining relevance. For any document with a sufficiently high relevance score, all members of the document’s family will be included in the attorney review prior to production. Documents will be produced, and withheld (as appropriate) as families. Negotiating Cutoff Point for Irrelevant Data Collaborative approach, with plaintiffs allowed to statistically sufficient sample set of documents determined to fall below proposed cutoff Global Aerospace and Da Silva similarly advocate collaborative approach built around representative sampling of data below proposed cut-off Follow Actos and agree to collaborative approach based on representative sampling of data below proposed cut- off Consider attempting to reach agreement on floor or standard at which cutoff will occur (i.e. not agreeing to a specific “score” but instead agreeing that cutoff will occur no lower than at a point at which the percentage of documents likely to be responsive falls below a certain threshold regardless of the score at which that occurs) Negotiating Acceptable Margin of Error / Confidence Level Collaborative approach based on identification of control sets prior to review. Parties agreed to 95% confidence level and +/- 5% margin of error Global Aerospace-train to 75% recall level Da Silva—Iterative training until change in total number or relevant documents predicted by the system is less than 5% as compared to the prior iteration and no new documents are found that are predicted to be hot Follow Actos in spirit, but make no commitment to specific margin of error and confidence until more is known about the data set for purposes of identifying level of review required to achieve desired levels in light of richness of data population. Objective is to avoid necessity of sampling inordinate volume of data. © DiscoverReady 2013

Privileged & Confidential Other Considerations Selecting your predictive coding software Know that opposing counsel is going to actively participate in your discovery process to a level uncommon for manual review Use Actos protocol as a stylistic guide to set out SI protocol to define and set out: –Data sources and custodians subject to predictive coding process –Process to be followed (and tight dispute resolution provisions) –Effective claw back agreement Utilize automated privilege screening process or other well defined quality privilege screening process for training sets and sample sets Document your process and have an audit trail for all “predicted” decisions Mocking your 30(b)(6) and your pre-trial conferences Understand whether you are achieving your goals taking into account technology, consulting, project management and all forms of review © DiscoverReady 2013

Privileged & Confidential Discover Better. DiscoverReady. Thank you.

Privileged & Confidential Data Set Training Set Likely Not Relevant Indeterminate Likely Relevant How does it work? Basic predictive coding Number of documents required for training depends on a) richness of the data and b) desired confidence level Example Predictive Coding Score Experts review a sample “training set” and the predictive coding tool “scores” the remaining documents in the data set based upon likely relevance © DiscoverReady 2013