© 2011 Winston & Strawn LLP
Today’ss eLunch Presenters Today eLunch Presenters
© 2011 Winston & Strawn LLP
John Rosenthal
Scott Cohen
Litigation Washington, D.C.
Director of E‐Discovery Support Services New York
JRosenthal@winston com
[email protected]
SCohen@winston com
[email protected]
2
What Was Advertised
Effective strategies for reducing the cost of an electronic ec e s a eg es o educ g e cos o a e ec o c document review Use of second generation search technologies to both filter document sets and more efficiently organize the materials for review, including concepting, clustering, threading and near duping technologies threading and near‐duping technologies Implementing best practices in the review process to ensure a high degree of precision and recall g g p Effective strategies for handling privilege documents Use of lower‐cost reviewers
© 2011 Winston & Strawn LLP
3
Why Should You Listen to Us? Why Should You Listen to Us?
Spent the last two years studying the e‐discovery p y y g y marketplace and designing a new platform Benchmarked the entire industry, including other l f law firms, vendors and software providers d d f d Built an e‐discovery consulting shop behind Winston & Strawn’ss firewall to include: & Strawn firewall to include:
Collection Processing Analytics Review Platform Review Center Review Center
© 2011 Winston & Strawn LLP
4
Electronic Document Review
Excessive and unpredictable costs: p
Traditional document review is not accurate:
58 % to 70 % of total litigation costs Document review costs are rising due to the increasing amount of electronic information Evidence suggest there are high error rates in linear manual review review Error rates lead to likelihood of inadvertent production of privileged or sensitive information
Inability to defend the review process: Inability to defend the review process:
Judges are increasingly focusing on the need for validation of review processes
© 2011 Winston & Strawn LLP
5
5
Goals of ESI Review
Recall ‐ Identification and prioritization of relevant material Recall Identification and prioritization of relevant material Precision ‐ Elimination of irrelevant/non‐ responsive material Identification of privileged material Relevant Data N Non‐relevant l and retrieved d i d
Relevant and not retrieved
Retrieved D Data © 2011 Winston & Strawn LLP
Relevant and retrieved
6
Accuracy of Human Review y
Recall Number of responsive documents retrieved Total number of responsive documents in the collection
Precision Number of responsive documents identified Total number of documents retrieved
© 2011 Winston & Strawn LLP
7
Accuracy of Human Review y Perfection
100% 90%
Blair & Maron (1985) 80%
Typical result in a Typical result in a manual responsiveness review
Preccision
70% 60% 50% 40% 30% 20% TREC Best Benchmark (Best performance on Precision at a given Recall)
10% 0% 0%
10%
20%
30%
40%
50%
60%
Recall M. Grossman Presentation - Technology-Assisted Review Can be More Effective andWinston More Efficient © 2011 & StrawnThan LLP Exhaustive Manual Review
8
70%
80%
90%
100%
Traditional Electronic Document Review = Linear Review
Over collection Over collection Little or no culling Ad hoc use of Boolean searches Linear review of the data set Use of traditional associate work f force to perform review t f i
Traditional Approach
Manually Acquire Broad Amounts of Data
Process Data
First Level Review Second Level Review
Produced Documents
© 2011 Winston & Strawn LLP
9
Linear Review
© 2011 Winston & Strawn LLP
10
Traditional Approach = $$$ pp $$$
Each custodian = 4 and 8 GB 1 GB of data = 60,000 pages Average review rate = 350 to 500 pages per hour 1 GB = 120 hours of review time time Law firm dynamic – 1 GB = $36,000 or $1.60 per page Outsourced review 1 GB = $6,500 a GB to $11,000 (60 to 95 cents per page) to 95 cents per page)
© 2011 Winston & Strawn LLP
11
Why Has Legal Market Been So Slow to Adopt Changes?
FFailure to understand the traditional model is broken il t d t d th t diti l d li b k Does not understand how to use the new technologies Mistakenly interprets move toward outsourced review as a result of the economic downturn review as a result of the economic downturn Continues to permit individual litigators to select vendors review platforms and implement their own vendors, review platforms and implement their own processes
© 2011 Winston & Strawn LLP
12
The Good News!
Cost associated with reviewing large volumes of ESI C t i t d ith i i l l f ESI is forcing a change Vendor entry into review space has accelerated Vendor entry into review space has accelerated change Clients are demanding greater use of: Clients are demanding greater use of:
Low‐cost domestic review attorneys Off‐shore review facilities Adoption of new technologies
© 2011 Winston & Strawn LLP
13
Document Review Models Outsourced Manual Review • Most prominent model used today • Limited culling and Li i d lli d analysis • Heavy reliance on attorney review • Use of sampling to f ensure quality control
© 2011 Winston & Strawn LLP
Predictive Coding Predictive Coding
Process Oriented Review Process Oriented Review
• Great deal of confusion regarding what it means • Uses attorneys to Uses attorneys to develop a seed set of data that can be fed into a black box to find similar documents similar documents • Emphasizes sampling of inclusion set and exclusion set • Never tested or accepted in any court
• Development of broad and defensible relevance criteria relevance criteria through integrative ECA process • Process approach to review to increase review to increase efficiency, recall and precision , using legally accepted tool sets: • Threading • Near‐Duping • Advance search • Clustering
14
A Word On “Predictive Coding” g
© 2011 Winston & Strawn LLP
15
Technology‐Assisted Reviews gy
= TREC 2009 Manual Review = TREC 2009 TechnologyAssisted Review
M. Grossman Presentation - Technology-Assisted Review Can be More Effective and Than Exhaustive Manual Review © 2011 Winston & More StrawnEfficient LLP
16
Developing a Seed Set ‐ Precision
Document Set for Review
Source: Servient http://www.servient.com/ © 2011 Winston Inc. & Strawn LLP
17
Developing a Seed Set: Recall
Documents Set Excluded From Review
Source: Servient http://www.servient.com/ © 2011 Winston Inc. & Strawn LLP
18
Our Take on Predictive Codingg
Not ready for prime time as a final determinant of o eady o p e e as a a de e a o relevance and privilege:
Never been accepted by a court – do you want to be the first? “Predictive coding” technologies and processes vary wildly As with any statistical model, caution should be exercised (“Torture numbers, and they'll confess to anything”) Seed set methodology:
Garbage in = garbage out Who is picking your seed set Who is picking your seed set Cases change rapidly within the first few months Is this scalable to all types of cases Not able to address (images graphics excel files video voice etc ) Not able to address (images, graphics, excel files, video, voice, etc.)
© 2011 Winston & Strawn LLP
19
Our Take on Predictive Coding (cont’d) g( )
Over time, predictive coding and other technology Over time, predictive coding and other technology assisted methodologies will become both prevalent and accepted Under the appropriate circumstances (e.g., agreement by the parties), it can be a valuable and cost effective tool Until then ‐‐ predictive coding can be used to help organize and make a review more efficient d k ff
© 2011 Winston & Strawn LLP
20
© 2011 Winston & Strawn LLP
21
A Process Oriented Electronic Document Review
© 2011 Winston & Strawn LLP
22
Old vs. New Models Traditional Approach
Process Oriented Approach
Manually Acquire Broad Amounts of Data
Analytics/
Process Data
Search, Acquire, and Process Narrower Amounts of Data Filter and Cull
First Level Review Second Level Review
Review
Produced Documents
P d Produced dD Documents t
© 2011 Winston & Strawn LLP
23
Phases to A Process Oriented Document Review Analytical
• Working with client and data to develop a set of Working with client and data to develop a set of defensible “relevance criteria” to select data subject to review
Collection
• Use of search and retrieval at the front end can dramatically reduce the volume and cost • Risk consideration
• Employ more sophisticated processing tools to Processing– filtering and culling further reduce the volume set • Unilaterally vs. negotiate
Non‐Linear Review
© 2011 Winston & Strawn LLP
• Employ lower cost reviewers • Use technology and process to increase precision and recall across the data set d ll th d t t
24
Selecting the Review Set g
Recent focus by courts y Recognition that parties have failed to adequate construct and test search terms to withstand judicial scrutiny (Victor Stanley) scrutiny (Victor Stanley) Absent agreement, you will have to defend your relevance criteria in terms of reasonableness and responsiveness:
Selecting test data Quality controlling the results Quality controlling the results Revising queries Transparent records from which you can defend your decision
© 2011 Winston & Strawn LLP
25
Selecting the Review Set (cont’d.) g ( ) ESI Sources
Select Test Data Set
Archiver
Archiver
Network Stores
Filter Test Data Set Based Upon Queries Date Range Process
Network Shares
PCs/Laptops
PCs/Laptops
SharePoint
SharePoint
Loose Media
Loose Media
QC via Sampling
Dedupe, De-NIST, Extract
Custodians File Types
Responsive Test Data Set
Location Key Words
Filter Entire Data Set Based Upon Relevance Queries Process Revise Queries as Appropriate
Dedupe, De-NIST, Extract
Date Range
Exceptions
C t di Custodians
Responsive Y/N?
File Types
Responsive Data Set Location Key Words
© 2011 Winston & Strawn LLP
26
Rapid Filtering/Culling Rapid Filtering/Culling
© 2011 Winston & Strawn LLP
27
Multi‐Faceted Display of Results p y
© 2011 Winston & Strawn LLP
28
Instant File Type Analysis yp y
© 2011 Winston & Strawn LLP
29
Visualization of Email Traffic
© 2011 Winston & Strawn LLP
30
Advanced Search
© 2011 Winston & Strawn LLP
31
Review Phase
70% of your production is e‐ 70% of your production is e mail 15% to 40% are duplicates or near duplicates near duplicates Reviewers can review data at a higher rate with a greater d degree of precision and recall f ii d ll when looking at like information
© 2011 Winston & Strawn LLP
32
Review Phase Store NR d t in data i
Remove from DB?
QC via Sampling p g
No Non-Responsive Cull via Metadata and Smart Filters Date Range Initial Data Set
Custodians File Types
QC via Sampling
Batch Documents Privilege Filter
Custodian
To/From
Key Word
Location
Metadata
Key Words
E-mail Thread
Review
Location
Near-Dupes
Key Words
Separate Attorneys Att
© 2011 Winston & Strawn LLP
Non-Responsive
Cluster
Send S d to t Privilege P i il Review
33
Responsive
Privileged?
Organizing the Data For Review g g
Batching will be the most significant decision in Batching will be the most significant decision in terms of expediting the review Our approaches: pp
E‐mail threading Near Duplicates Review in the remainder
Clustering algorithm Boolean queries Boolean queries Key custodian review is appropriate Types of Documents M d Metadata (e.g., date range, author, recipient, etc.) ( d h i i )
© 2011 Winston & Strawn LLP
34
E‐Mail Threads • 70% of production is e-mail and of that nearly 65% or more are part of e-mail threads
Less Time
Less Errors The Problem:
eMail Threads – Step 1
eMail Threads – Step 2
No N clear l method th d to t identify eMail threads
Group Gro p into eMail sets
Build tree structure Identify missing links
eMails are reviewed multiple times and inconsistently
Suppress duplicates Focus on inclusives
Extremely difficult to identify where missing eMails exist © 2011 Winston & Strawn LLP
Source: Equivio
35
Less Cost
Duplication and Near Duplication p p • 15% to 40% of document population are duplicates or near d plicates duplicates
Less Time Near-Duping – Step 2
The Problem:
Near-Duping – Step 1
No clear method to organize and allocate documents across reviewers
Group the near-duplicates Assign near-dupe sets for coherent review to Identify the differences reviewers among the near Reviewers prioritize and duplicates review only the differences
Documents are reviewed multiple times by different reviewers
Apply coding to entire near-dupe sets where appropriate
High risk of different coding among similar documents © 2011 Winston & Strawn LLP Source: Equivio
36
Less Errors Less Cost
Second Generation Search Technology gy
Concept search places a document or part of a document in this space. Results are returned in order of relevance.
higher score = closer document
• Document D t 1: 1 98 • Document 3: 92 • Document 4: 91
© 2011 Winston & Strawn LLP Source: K-Cura Corp.
37
Advanced Features of Relativity Analytics y y
Clustering – Automatic grouping (sorting) of a collection of documents into subsets based on their conceptual content Categorization – Categorize and organize documents in the entire database based on a small set of user example documents database based on a small set of user example documents Keyword Expansion – Select keywords within the document viewer and Relativity will produce a list of conceptually related terms Inline Concept Search – Utilize concept searching within Relativity’s innovative document viewer by right‐clicking on portions of a displayed document
© 2011 Winston & Strawn LLP
38
Keyword Expansion y p
© 2011 Winston & Strawn LLP
39
Best Practices ‐ Planning g
Plan the work and work the plan: Plan the work and work the plan:
Outside counsel eDiscovery Team Review team leader
Planning meetings:
Developing relevance criteria Developing a review plan
Deadlines must be established, staffing needs must be defined, Deadlines must be established staffing needs must be defined and review speeds must be estimated to provide budget predictability to the client and avoid cost and time overruns
© 2011 Winston & Strawn LLP
40
Best Practices ‐ Planning (cont’d) g( )
Before commencing review, decide on format of Before commencing review, decide on format of production and privilege log
Rule 26(f) conference Native vs. TIFF productions Privilege log: metadata vs. manual
Discuss potential exclusions, such as for post‐litigation outside Di t ti l l i h f t liti ti t id counsel documents
Logging of email chains Timing of privilege log production (with or after document productions)
© 2011 Winston & Strawn LLP
41
Best Practices ‐ Roles
eDiscovery Team eDiscovery Team
Administration of the data set (e.g., loading of data, managing index, running clusters) Assist Review Team Leader in developing relevance criteria Creation of load file
R i T Review Team Leader L d
Coordinate with attorneys regarding development relevance criteria Identify test data set Validate test data set Manage reviewers
© 2011 Winston & Strawn LLP
42
Best Practices ‐ Training the Review Team Best Practices Training the Review Team Substantial attention should be given to training. Substantial attention should be given to training Briefing materials should be prepared:
List of counsel involved in the Legal Matter List of key witnesses involved in the Legal Matter Summary of the claims and issues Summary of the request for production of documents and y q p responses Instructions regarding how to use the review tool Instructions regarding coding of the individual documents g g g Procedure for resolving questions or ambiguities Training on how to recognize and code privileged documents
© 2011 Winston & Strawn LLP
43
Best Practices – Conducting the Review g
Uniformity in review team analysis should be the goal Variations in review team analysis should be addressed each day and additional training provided as required Daily feedback heightens quality and attention to detail by reviewers, and knowledge transfer to reviewers Reduces possibility of accumulation of poorly reviewed documents in the population Review team should be broken down into sub‐groups (e.g., separate privilege review group) Computer equipment should foster speed and efficiency p q p p y Dual screens or wide screens, fast processing time Review tool should be configured to reduce need for scrolling and key strokes Procedures to address potential departure of review team members Procedures to address potential departure of review team members
© 2011 Winston & Strawn LLP
44
Best Practices ‐ Quality Control Q y
Review team leader is responsible for overall quality control of the review and should meet with review team extensively f h d h ld h l during review
Review questions Id tif Identify categories of documents that can be removed as N/R t i fd t th t b d N/R Prepare list of additional questions for litigation team
QC reviewers work by:
EExamining individual reviewer productivity and error rates i i i di id l i d ti it d t Designating QC reviewers to re‐review established percentage of documents to verify accuracy of coding
Type of sampling (random, systematic, stratified) yp p g( y ) Sample sizes Review rate Accuracy rate Hours worked Hours worked
© 2011 Winston & Strawn LLP
45
Other Best Practices Annotating and coding: Annotating and coding:
Determination is required as to extent of annotation and coding that would be beneficial Coding may include responsiveness, privileged, confidential, sensitive and relevant to issue codes
Review in native format Review in native format Review metadata if necessary Review hidden data if necessary Review hidden data if necessary
© 2011 Winston & Strawn LLP
46
Best Practices ‐ Privilege Review g
Create separate privilege review/privilege log preparation team Create separate privilege review/privilege log preparation team Segregate attorney ESI from the production Screen out potentially privileged documents:
Run key word searches of privileged ESI Run key word searches of privileged ESI (e.g., attorney, solicitor, privilege, work (e.g., attorney, solicitor, privilege, work product) URL analysis Hashing comparisons Clustering
Educate the review team on privilege coding and issues Clarify handling of email chains and email attachments Include protocol for compliance with FRCP 26(b)(5)(B) and FRE Include protocol for compliance with FRCP 26(b)(5)(B) and FRE 502(b) and (d) Additional pre‐production validation of privileged documents (on sampling basis) p g )
© 2011 Winston & Strawn LLP
47
Review for Privilege g
Return to Production Set
Review Strategies Privilege Filter
URL Search Terms Clustering
No
Review all Docs Review Positive Hits
Privilege (Y/N)?
Sample NonPrivilege Hits
Redaction Log g
Yes
Log
© 2011 Winston & Strawn LLP
48
QC via Sampling
Best Practices – Privilege Log g g
Automate log as much as possible
Leverage metadata as much as possible
Increases speed and efficiency (saves time and money) Decreases typos and inconsistencies Goal is to avoid manual data entry wherever possible
Be careful, though, of relying on metadata alone for email chains and PDFs unless the parties have agreed PDFs, unless the parties have agreed
Use drop‐down boxes where possible, having reviewers choose the one or more descriptions that best fit the document
© 2011 Winston & Strawn LLP
49
Washington, D.C. – g , Litigation Review Center g
© 2011 Winston & Strawn LLP
50
Staffing a Review g
Lower‐cost Lower cost staff that specializes in review staff that specializes in review
Properly recruited and screened Proper supervised Properly trained
Considerations:
Insource vs. outsource Domestic vs. international JD or barred JD or barred Use of non‐lawyers Use of technical experts Use o tec ca e pe ts
© 2011 Winston & Strawn LLP
51
Ethical Issues
ABA 08 0451 (A ABA 08‐0451 (August 2008) 2008)
U.S. lawyers are ethically permitted to outsource legal work, including lawyers or non‐lawyers work, including lawyers or non lawyers (both (both domestically and internationally) if they adhere to ethics rules requiring:
Competence Supervision Protection of confidential information Reasonable fees Not assisting unauthorized practice of law
© 2011 Winston & Strawn LLP
52
Ethical Issues
ABA 08 0451 (A ABA 08‐0451 (August 2008) 2008)
Minimal obligations:
Conduct reference checks and background investigations of Conduct reference checks and background investigations of lawyer or non‐lawyer service providers and any intermediaries Interview principal lawyers on a project, assessing their educational background and evaluate the quality and educational background, and evaluate the quality and character of any employees likely to access client information Review security systems Visit the premises of the service provider Visit the premises of the service provider
© 2011 Winston & Strawn LLP
53
Ethical Issues
If the provider is in a foreign country: If the provider is in a foreign country:
Determine whether the legal education system in that country is similar to that of the U.S., and whether professional regulatory systems incorporate equivalent core ethics principles and effective disciplinary incorporate equivalent core ethics principles and effective disciplinary enforcement systems Consider utilization of additional training, especially regarding p privilege g Determine whether the foreign legal system protects client confidentiality and provides effective remedies to the lawyer’s client in case disputes arise Some circumstances may require more rigorous supervision than others May be necessary to obtain a client's consent before engaging outside assistance
© 2011 Winston & Strawn LLP
54
Proper Supervision p p
Bray & Gillespie Mgmt. LLC v. Lexington Ins. Co., No. 6:07‐ Bray & Gillespie Mgmt LLC v Lexington Ins Co No 6:07 cv‐222‐Orl‐35KRS, 2009 WL 546429 at * 21 (M.D. Fla. Mar. 4, 2009)
Form of Production; Rule 37 Sanctions; Outside Counsel Changing Form of Production; Rule 37 Sanctions; Outside Counsel Changing Law Firms; Individual Attorney Sanctioned “While B & G, as the client, has the obligation to supervise its lawyers, the evidence establishes that B & G's outside counsel made the decision how to produce ESI Additionally B & G has made the decision how to produce ESI. Additionally, B & G has already spent considerable time and effort to reproduce some ESI in native format, although problems remain with the form of that production. Under these circumstances, I find that it is not appropriate to require B & G to pay the attorney'ss fees, costs, and appropriate to require B & G to pay the attorney fees costs and expenses . . . . Should B & G fail to monitor its counsel's actions going forward, however, it will subject itself to all available sanctions should additional problems occur.” ““ . . . blindly relying on outside counsel falls short of the duty he has as an officer of bli dl l i t id l f ll h t f th d t h h ffi f the court, as counsel of record, and as an advocate for his client.”
© 2011 Winston & Strawn LLP
55
Budgeting g g
Now possible to model the entire review process to Now possible to model the entire review process to establish a reliable budget Considerations:
E‐discovery costs Size of review Potential cull/filter rate Percent of e‐mail/threads P Percent of dupes and near‐dupes t fd d d Complexity of the documents
© 2011 Winston & Strawn LLP
56
© 2011 Winston & Strawn LLP
57
Questions?
© 2011 Winston & Strawn LLP
58
Contact Information Contact Information
John Rosenthal John Rosenthal Chair, eDiscovery & Information Management P ti G Practice Group Washington, D.C. 1 (202) 282‐5785
[email protected]
© 2011 Winston & Strawn LLP
Scott Cohen Scott Cohen Director of eDiscovery Support Services N Y k New York 1 (212) 294‐3558
[email protected]
59
