SureMail: Notification Overlay for Email Reliability

1
SureMail Notification Overlay for Email
Reliability

• Sharad Agarwal Venkat Padmanabhan
• Microsoft Research
• Dilip Antony Joseph
• UC Berkeley
• HotNets 2005

2
Silent Email Loss

• Silent email loss
• email vanishes without sender/recipient
  knowledge
• ? missed opportunities, misunderstanding, or
  worse
• Nontrivial problem
• anecdotal evidence
• measurement studies
• 0.69 loss rate LM 04
• 0.1-5 loss rate AB 05
• commercial offerings to address the problem
• e.g., Pivotal Veracity, Zenprise

3
HotNets air ticket confirmation We have sent it
through again. If you do not receive it with in
an hour or two, please let us know. Funding
proposal "No I never got and I never acked
it My last mail from you was on
3/10/2004. IMC 2005 decision notification I
recd reviews for one paper (X) but not that of
Y. IMAP server upgrade problems Some
unanticipated migration problems occurred that
may have caused some lost or delayed email.
4
Silent Email Loss

• Silent email loss
• email vanishes without sender/recipient
  knowledge
• ? missed opportunities, misunderstanding, or
  worse
• Nontrivial problem
• anecdotal evidence
• measurement studies
• 0.69 loss rate Lang Moors 2004
• 0.1-5 loss rate Afergan Beverly 2005
• commercial offerings to address the problem
• e.g., Pivotal Veracity, Zenprise

5
Silent Email Loss

• Why email loss?
• spam filtering big problem ? aggressive
  filtering
• MS 90 of emails discarded before hitting user
  mailboxes
• AOL 100 emails per month to maintain IP
  white-listing
• server failures and upgrades
• SMTP is not end-to-end reliable
• (Non-)Delivery status notifications
• compounds spam problem
• raises privacy concerns
• So email loss is often silent

6
Fixing the Problem

• Improve the email delivery infrastructure
• more reliable servers
• e.g., cluster-based (Porcupine Saito 00)
• server-less systems
• e.g., DHT-based (POST Mislove 03)
• total switchover might be risky
• Smarter spam filtering
• moving target ? mistakes inevitable
• non-content-based filtering still needed to cope
  with spam load

7
SureMail

• SureMail addresses the problem from the outside
• add separate notification overlay
• email delivery infrastructure left undisturbed
• users can benefit without operator cooperation
• Design goals
• minimize demands on infrastructure and users
• preserve asynchronous operation and privacy (no
  worse than it is today)
• maintain defenses against spam and viruses
• minimize overhead

8
Basic Operation
Request lost message
Sender S
Recipient R
GetNotifications
Notification server
9
Notification Overlay

• Decentralized
• limited collusion among the constituent nodes
• Efficient notification server lookup
• e.g., R ? H(R) in a DHT setup
• Agnostic to actual implementation
• end-host-based (e.g., always-on user desktops)
• infrastructure-based (e.g., NX servers)

10
Challenges

• Privacy
• information about users email habits could be
  leaked
• Notification spam
• spammers can spoof notifications and burden users
• annoyance attacks discredit notifications in
  general
• Even the notification infrastructure isnt
  trusted
• No universal PKI for email users

11
SureMail Goals

• Protect the recipients identity
• attacker shouldnt be able to retrieve Rs
  notifications or learn the volume of
  notifications intended for R
• Protect the senders identity
• attacker shouldnt be able to learn Ss identity
  or monitor the volume of notifications posted by
  S
• Block notification spam
• attacker shouldnt be able to spoof notifications

12
Assumptions

• No email eavesdroppers
• privacy is moot otherwise
• Limited collusion among notification nodes
• needed only to avoid leaking notification volume
  info

13
Key Mechanisms

• 1 Email-based handshake
• 2 Decoupled registration and notification
• 3 Email-based shared secret
• 4 Reply-based shared secret

14
1 Email-based handshake

• Goal prevent hijacking of Rs identity
• Only R can receive emails sent to R
• One-time operation for initial registration
• Send email to R to establish registration secret
  shared with the notification overlay
• R can then use registration secret to
  authenticate itself to the notification overlay

15
2 Decoupled registration notification

• Goal prevent snooping on recipient identity
• Limited collusion among notification nodes
• Registration at DregH(H(R))
• Notification posted at DnotH(R)
• R contacts Dnot to retrieve notifications for
  H(R)
• Dnot can find Dreg without knowing R
• Neither Dnot nor Dreg can associate notifications
  with R, unless they collude

16
3 Email-based shared secret

• Goal prevent snooping on sender identity
• Email Mold from S to R in known only to S
  and R
• H(Mold) could serve as implicit identifier of S
  to R
• But it doesnt quite serve as authenticator for
  S
• Dnot knows H(Mold), so it could spoof
  notifications from S
• even other attackers could do so by first sending
  Mspam purporting to be from S

17
4 Reply-based shared secret

• Goal block spoofing of notifications
• Users rarely have conversation with spammers
• R remembers (hashes of) recent emails from S that
  it has replied to
• If S receives a reply to Mold it had sent R, Mold
  can serve as a shared secret between S and R
• S could use H1(Mold) as an implicit identifier
• and H2(Mold) as an authenticator
• Hard for a spammer (even Dnot) to spoof

18
Putting it all together
Sender S
Recipient R
DnotH(R)
DregH(H(R))
19
Other issues

• Reply-detection
• in-reply-to header may not always help
• indirect checks based on text similarity
• Reducing overhead
• post notifications only for important emails
• hold off on posting notification in the hope of
  receiving an implicit ACK (reply) or NACK
  (bounce-back)
• First-time legitimate senders
• they are indistinguishable from spammers
• Mobility
• reply-based shared secret enables secure
  migration without state transfer

20
Status

• Ongoing measurement experiment
• Design being refined
• Implementation in the works

21
Discussion

• 1 Should the notification system be folded into
  the email infrastructure?
• Separation is advantageous
• provides failure independence
• keeps the notification layer simple
• small, fixed format notifications dont require
  the same kind of processing as virus-laden email
• provides engineering convenience

22
Discussion

• 2 Is there a social benefit to silent email
  loss because of the plausible deniability it
  provides?
• Any such benefit is far outweighed by the costs
• Should cars be slightly unreliable because of the
  excuse it would give people when they miss an
  engagement?
• It is the asynchronous nature of email that is
  key

23
(No Transcript)
24
Anecdotes
Funding proposal "No I never got and I never
acked it My last mail from you was on
3/10/2004. Response to self-managing networks
summit invitation "Yesterday's email did not
bounce back, wonder where it is! IMC 2005
decision notification I recd reviews for one
paper (X) but not that of Y. IMAP server
upgrade problems Some unanticipated migration
problems occurred that may have caused some lost
or delayed email.
25
Basic Operation

• Senders post notifications to overlay, in
  addition to sending emails as usual
• Intended recipients periodically download
  notifications intended for them
• A notification without a matching email suggests
  possible email loss

26
Putting it all together

• Registration
• R contacts DregH(H(R)) to register
• Dreg sends R an email to set up registration
  secret
• Posting notifications
• upon sending email M to R, S posts notification N
  to DnotH(R)
• N Encrypt(H2(Mold), H1(M)), H1(Mold)

27
Putting it all together

• Retrieving notifications
• R asks Dnot for the notifications corresponding
  to H(R) and presents evidence of registration
  secret
• Dnot contacts Dreg to verify evidence, before
  returning the notifications to R
• R uses H1(Mold) to identify Mold and compute the
  encryption key H2(Mold)
• R discards bogus notifications and checks for
  missing emails corresponding to the remaining
  notifications

28
1 Protecting the recipients identity

• Goal
• only R should be able to retrieve notifications
  intended for it
• attackers shouldnt be able to learn even the
  volume of notifications intended for R
• Key idea
• Email-based handshake
• prevents hijacking of Rs identity
• Decoupling registration from notification
• prevents bad DHT node from associating
  notifications with R

29
1 Protecting the recipients identity

• Registration
• R contacts Dreg H(H(R)) to register
• Dreg sends R an email to set up a shared secret
• Posting notifications
• Upon sending email M to R, S posts notification N
  H(M),S to Dnot H(R)
• Retrieving notifications
• R presents an authenticator to Dnot and asks for
  the notifications corresponding to H(R)
• Dnot contacts Dreg to verify the authenticator,
  before returning the notifications
• R checks if emails are missing and presents the
  corresponding S to the user

30
SureMail Goals

• 1 protecting the recipients identity
• 2 protecting the senders identity
• 3 blocking notification spam

31
2 protecting the senders identity

• Goal
• attackers shouldnt be able to learn Ss identity
  or monitor the volume of notifications posted by
  S
• clearly N H(M),S wont do
• Key idea email-based shared secret
• assuming no eavesdroppers, an email Mold from S
  to R in known only to S and R
• so H(Mold) could serve as an authenticator and
  identifier of S to R

32
2 protecting the senders identity

• Posting notifications
• Ss identity is made implicit in the notification
• N H(M), H(Mold)
• Retrieving notifications
• R stores hashes of emails received (recently)
  from various senders
• it searches for H(Mold) to identify S
• if H(Mold) cant be found, the notification is
  ignored

33
SureMail Goals

• 1 protecting the recipients identity
• 2 protecting the senders identity
• 3 blocking notification spam

34
3 blocking notification spam

• Goal
• prevent spammers from posting bogus notifications
  and burdening users with false reports of email
  loss
• A malicious DHT node could itself be a spammer

35
3 blocking notification spam

• Current scheme is vulnerable to attack
• Easy for malicious DHT node Dnot to generate
  spam
• Dnot has ready access to H(Mold)
• it could spam R with bogus notifications
  purporting to be from S
• Another spammer (say X) could also do so
• X sends R a message Mspam purporting to be from S
  
• X can then use H(Mspam) as the implicit
  identifier in notifications
• R may assume it is really missing an email from S
• Spammer gain indirectly by annoying R and S,
  thereby discrediting notifications in general

36
3 blocking notification spam

• Key idea reply-based shared secret
• users rarely engage in conversations with
  spammers
• so if S receives a reply to a message Mold that
  it had sent R, S could use H(Mold) as an implicit
  identifier
• hard for a spammer to spoof the identifier
• special construction to prevent spoofing by Dnot
• Notification format
• N Encrypt(H(Mold), H(M)), H(Mold)
• R uses H(Mold) to identify Mold and compute the
  encryption key H(Mold)

37
PKI-based Design

• R?DA RegisterRecipient(R,A)
• S?DN PostNotification(H(R),N)
• N H(M), TTL, E(Rpb,S), Sg(Spv,H(M),TTL)
• R?DN CheckNotification(H(R),A)
• DN find DA H(H(R))
• DN?DA AuthenticateRequest(H(R),A)
• DN?R ReturnNotifications()
• R identify notifications corresponding to
  missing email and notify user if S is trusted

38
Notation

• Sender S
• Recipient R
• Message M
• Notification N
• Crypto operations
• H hash()
• Sg sign(key,)
• E encrypt(key,)

39

• Overhead
• duplication of effort with respect to email
  delivery
• 3 preserve privacy of email content
• attacker shouldnt be able to learn email content

40
Silent Email Loss

• Silent email loss is a non-negligible problem
• silent loss no notification to sender or
  recipient
• imposes significant cost, degrades user
  experience
• Several causes
• spam filtering
• MS IT 90 of emails discarded off the bat
• failures and upgrades
• Measurement studies
• 0.5-1.0 silent loss (Lang 04, Afergan 05)
• ongoing measurement study at MSR
• quantify email delays loss across 25 domains

41
SureMail Overview

• SureMail adds separate notification system
• orthogonal to email delivery infrastructure
• email is still subject to checking by spam
  filters, virus scanners
• doesnt create backdoor for malware
• bounds worst case performance
• Asynchronous operation
• senders post notifications hash(message)
• recipients check for them
• preserves the privacy of email (unlike read
  receipts)

42
SureMail Design

• Reply-based shared secret
• A sends an email M to B (A?B)
• B replies to As email (B?A)
• A uses hash(M) to prove to B that it is
  legitimate
• shared secret is continually refreshed
• Reply-based shared secret helps
• avoid burdening users with notification spam
• maintain privacy of notifications