Desiderata in the Legal Realm

1
Desiderata in the Legal Realm

• Two-party
• Negotiated (not one-sided) information needs
• Recall-oriented
• Smoking gun detection completeness
• Explainable
• Quantifiable comparison to present best practice
• Affordable
• Minimize amount of human review on back end

2
Deconstructing Concept Search

Method
Features
Specification
Result
3
Representing Documents

• Content
• Count the words
• Weight the words
• Ascribe meaning to words
• Context
• Who said this?
• When was it said?
• Who did they say it to?
• Description
• What was said about it?
• Behavior
• What was done with it?

Features
4
Controlling the Search System

• Proactive
• Keyword query
• Boolean query
• Query by example
• Reactive
• Ranked list selection
• More like this query
• Category exploration
• Social network exploration
• Iterative
• Query refinement
• Search within query

Specification
5
Generating Results

• Result set
• Ranked list
• Classification
• Clustering
• Visualization

• Logic
• Similarity
• Probability

Method
Features
Specification
Result
6
Meet-and-Confer Alternatives
7
Incremental Disclosure Benefit
50 Topics, Title Queries, TREC-2005 Robust Track
Collection
8
Taking the Larger View
Jack G. Conrad, E-Discovery Revisited A
Broader Perspective for Researchers, DESI-1
9
E-Discovery as Sensemaking
Simon Attfield and Ann Blandford, E-Discovery
Viewed as Integrated Human-Computer Sensemaking,
DESI-2
10
Identity Resolution in Email

• Date Wed Dec 20 085700 EST 2000
• From Kay Mann ltkay.mann_at_enron.comgt
• To Suzanne Adams ltsuzanne.adams_at_enron.comgt
• Subject Re GE Conference Call has be
  rescheduled
• Did Sheila want Scott to participate? Looks like
  the
• call will be too late for him.

Sheila
WHO?
11
Easy References of Identity
Message-ID lt1494.1584620.JavaMail.evans_at_thymegt Da
te Mon, 30 Jul 2001 124048 -0700 (PDT) From
elizabeth.sager_at_enron.com To sstack_at_reliant.com S
ubject RE Shhhh.... it's a SURPRISE ! X-From
Sager, Elizabeth lt/OENRON/OUNA/CNRECIPIENTS/CN
ESAGERgt X-To 'SStack_at_reliant.com_at_ENRON'
Email Standards
Hi Shari
Hope all is well. Count me in for the group
present. See ya next week if not earlier
Liza Elizabeth Sager 713-853-6349
Email-Client Behavior
User Regularities
-----Original Message----- From
SStack_at_reliant.com_at_ENRON Sent Monday, July 30,
2001 224 PM To Sager, Elizabeth Murphy,
Harlan jcrespo_at_hess.com wfhenze_at_jonesday.com Cc
ntillett_at_reliant.com Subject Shhhh.... it's a
SURPRISE !
Please call me [phone]
Elsayed and Oard, CEAS 2006
Thanks! Shari
12
Extraction From Quoted Headers
----------------------- Forwarded by Sheila
Glover/HOU/ECT on 06/14/2000 0739 AM
--------------------------- Sheung Tam
ltSheung.Tam_at_msdw.comgt on 06/14/2000 073950
AM Please respond to Sheung.Tam_at_msdw.com To
Sheila Glover ltSheila.Glover_at_enron.comgt cc
Subject Re Enron India LLC (Delaware
Company) Hi Sheila, For the new account funds
will have to be in the account, before any trades
Being put on the account. Sheung
Address-NameAssociationc
sheila.glover_at_enron.com
932 (Main Headers)
14 (Quoted Headers)
sheila glover
13
Signature Salutation Detection
From sheila.glover_at_enron.com
Shane, Laurel and I met with Sara Shackleton,
Legal, yesterday. Sara provided the following
updates for Australia ISDA Agreements National
Australia Bank Exectued 1/3/2000 on our side,
received executed back from them 3/3/2000 West
Pac Our end done, sent to them. Susan Flynn to
follow-up to id where in process currently. ANZ
Bank We need to know the exact name of the entity
we want the agreement with so we can apply to
the credit group to approve gettingan agreement
in place. thanks, sheila Enron Broadband
Services, Inc. 1400 Smith, Suite
EB-4573A Houston, TX 77002
Sara, We want to open an account with Direct
Trading Corp., There service is an ECN, similar
to Instinet. I will send up the agreement to
you for your review. Kelly Templeton will drop
it off to you this afternoon. Please call with
questions, 3-3210. thanks-Sheila Enron Broadband
Services, Inc. 1400 Smith, Suite
EB-4573A Houston, TX 77002
The kiddies are going back to school already so
now would be a good time to Paul, Where are we
on this? Can I do anything to assist? Thanks.
Sheila Enron Broadband Services, Inc. 1400
Smith, Suite EB-4573A Houston, TX 77002
sheila.glover_at_enron.com
932 (Main Headers)
14 (Quoted Headers)
sheila glover
14
Nickname Detection
From sheila.glover_at_enron.com
Shane, Laurel and I met with Sara Shackleton,
Legal, yesterday. Sara provided the following
updates for Australia ISDA Agreements National
Australia Bank Exectued 1/3/2000 on our side,
received executed back from them 3/3/2000 West
Pac Our end done, sent to them. Susan Flynn to
follow-up to id where in process currently. ANZ
Bank We need to know the exact name of the entity
we want the agreement with so we can apply to
the credit group to approve gettingan agreement
in place. thanks, sheila Enron Broadband
Services, Inc. 1400 Smith, Suite
EB-4573A Houston, TX 77002
Address-NicknameAssociation
nickname
sheila.glover_at_enron.com
932 (Main Headers)
216 (Signature)
19 (Salutation)
14 (Quoted Headers)
sheila
sheila glover
15
Representational Model of Identity
77,240 non-trivial identity models
Elsayed and Oard, CEAS 2006
16
Who is that Sheila?

• Date Wed Dec 20 085700 EST 2000
• From Kay Mann ltkay.mann_at_enron.comgt
• To Suzanne Adams ltsuzanne.adams_at_enron.comgt
• Subject Re GE Conference Call has be
  rescheduled
• Did Sheila want Scott to participate? Looks like
  the
• call will be too late for him.

?
Sheila
17
Topical Context
Date Wed Dec 20 085700 EST 2000 From Kay Mann
ltkay.mann_at_enron.comgt To Suzanne Adams
ltsuzanne.adams_at_enron.comgt Subject Re GE
Conference Call has be rescheduled Did Sheila
want Scott to participate? Looks like the call
will be too late for him.
GE
Sheila
call

• Date Fri Dec 15 053300 EST 2000
• From david.oxley_at_enron.com
• To vince j kaminski ltvince.kaminski_at_enron.comgt
• Cc sheila walton ltsheila.walton_at_enron.comgt
• Subject Re Grant Masson
• Great news. Lets get this moving along. Sheila,
  can you work out GE letter?
• Vince, I am in London Monday/Tuesday, back Weds
  late. I'll ask Sheila to fix
• this for you and if you need me call me on my
  cell phone.

sheila.walton_at_enron.com
GE
Sheila
call
18
Social Context
Date Wed Dec 20 085700 EST 2000 From Kay Mann
ltkay.mann_at_enron.comgt To Suzanne Adams
ltsuzanne.adams_at_enron.comgt Subject Re GE
Conference Call has be rescheduled Did Sheila
want Scott to participate? Looks like the call
will be too late for him.
kay.mann_at_enron.com

• Date Tue, 19 Dec 2000 070700 -0800 (PST)
• From rebecca.walker_at_enron.com
• To kay.mann_at_enron.com
• Subject ESA Option Execution
• Kay
• Can you initial the ESA assignment and assumption
  agreement or should I ask
• Sheila Tweed to do it? I believe she is
  currently en route from Portland.
• Thanks,
• Rebecca

kay.mann_at_enron.com
Sheila Tweed
19
Contextual Resolution
Elsayed, Oard and Namantha ACL/HLT 2008
Context-FreeResolution
Sheila Tweed
jsheila_at_enron.com
social
social
Sheila Walton
Sheila
topical
topical
sheila
social
Sheila
topical
conversational
sg
20
Test Collections
Enron-all
Enron-subset
Sager
Shapiro
21
Which Context is the best?
Sager
Shapiro
Enron-sub
Enron-all