A walk through a Grid Security Incident

1
A walk through a Grid Security Incident

• HEPiX
• Vancouver, October 24,2004
• Dane Skow, Fermilab

2
Players

• Users
• Resource Administrators
• Sites
• Virtual Organizations
• Certificate Authorities
• Authorization Authorities

3
Risk Assessment for Grid

• What is the level of risk ?
• Grid is not (yet) widely used with limited
  resources
• Resources are typically made available through a
  gatekeeper
• Grid is an error amplifier on steroids
• An opportunistic Grid job is much like a worm
  spread to all available resources and perform
  some coordinated task
• Needs and desires are outpacing controls
• Participants in a Grid announce themselves
• What do the bad guys want ?
• Most attacks are automatic these days (limited
  worry about WarGames attackers, but sk attack is
  counter argument)
• Spread itself is the game
• Spread to pop up untraceable service
• Spread to setup some mass action

4
Vulnerabilities Authentication

• Authentication system can be compromised
• CA compromise
• User private key compromise
• Proxy (authentication token) compromise
• Protocols implemented to fail open
• Responsibilities
• CA operators have to follow CP/CPS
• Resource Administrators have to protect proxies
• Users have to protect their private keys.
• All parties need to perform due diligence on
  checks

5
Vulnerabilities Resources

• Software applications (including OS) can be
  attacked.
• DOS attacks
• resource attacks (eg. filez, crackers, )
• Parallel for a sniffer attack would be a proxy
  hijacker.
• Responsibilities
• Resource Administrators have to keep software
  current
• Application authors have to patch defective
  software
• Incident Response ?

6
Vulnerabilities Authorization

• Authorization (AuthZ) system can be compromised
• AuthZ authority compromise
• Authorization managers authentication compromise
• Policy statements (eg. ACL) compromise
• AuthZ token (attribute certificate ?) compromise
• Responsibilities
• AuthZ operators have to follow an acceptable
  operations policy (CP/CPS or equiv)
• AuthZ admins have to protect their identities
• Resource Admins have to protect policy enforcement

7
Private Keys

• Todays story focuses on user private keys.
• Whoever has access to the private key can assert
  the public identity to which it corresponds.
• FNAL does not accept user-held private key PKI
  for general access systems save for a few Grid
  test systems. Considering the future.
• FNAL PKI for user certs uses KCA.
• Fair test of incremental load of 3rd party
  authentication
• Debate over user diligence on protecting private
  keys tested by looking at site network file
  systems.
• Storing a private key on any network file system
  exposes the key to network and file system
  attacks.
• openssl is your friend and the Grid incident
  investigators Swiss army knife.

8
AFS and User Private Keys

• Many users have home areas in AFS.
• Many users do not understand how AFS access
  control lists work.
• ? It is easy for users to leave their private
  keys world readable in AFS space.
• Should one proactively create a .globus directory
  in all users HOME with the proper permissions ?
• What about SSH RSA keys, browser credential
  caches, PGP keys,

9
The Stats

• Of 18 directories, 14 were world readable. 11 had
  valid certificates.
• After 40 days, 8 had still not been revoked. 3
  directories were still readable. 1 new exposure
  had occurred.
• Distribution of sources
• 5 DOEGrids
• 5 DOESciencegrids
• 1 Princeton self-signed

10
The Timeline

• September 5, scanned all HOME areas for readable
  .globus directories.
• Found 14 of 18 directories were world readable
• 11 had valid certificates for the matching keys
• September 5, sent mail to all affected users
• Basic statement of problem
• telling them how to fix the AFS permissions
• recommending they get certificates revoked

11
Details of a user private key

• more /.globus/userkey.pemBag Attributes   
  friendlyName Dane Skow 995399's ID   
  localKeyID 53 E0 A1 4A 57 DE 10 E6 79 DF DD AF
  DA 7D 4F 94 AD 90 E3 51Key Attributes ltNo
  Attributesgt-----BEGIN RSA PRIVATE
  KEY-----Proc-Type 4,ENCRYPTEDDEK-Info
  DES-EDE3-CBC,DBAD807ACC792107JIgCtaZcD4f2gyeILox
  kzd6nlLbK6JxZD/9ZJVKnPddsu2jy972JjAYhPR9b7y123e
  jBMW4XhikXewhODlkSZD0lNVUtcWBSKEmyMnkBXoZmHfpxSTQ
  6MZvAWkBbHWZt44Zdsw2ICbpqozy7zwAaCYFWOtwoE5DwvpR5
  1koKVAUcAjZdaLKzpxFs4wLm1oVcA0ONjM6jRBjtf0qQWcMDU
  Ytn57xZZlXscptORP2VRYBRjMY9xDPewnWUcM6FWb0iGerfs
  435XyAIcWDx0VL5GI1l1d1GBYyyKsoLruTah2IVJpbssmrroUe
  qt0T6jvlBgZgD7uRNSvfhBddXUV1uyE5SjgURI1t0BGoUs03
  K7MQjvjsenUIwPM/LjJYgdx2ctWtPR6YgXE4YCoqi30PwWd5S
  eyJljM3Mp0H28V7425DiU21VHwvHJPXfu5NsO3Q8oYC90G8H4
  9UZQGh6aZotJJQboGB3qHpYwwu4bSf1Rj9aLqqdr2NVSSHTmTL
  8bfchMIb5gGBrSTku1jq10Itdpg5KOvcH8neRllqN4p/NEdbR
  qf4e6R99E3PEdSUnyMJ5yiduE6SLdb49E6Z/McpRcv7SAyom
  An4YkADs4Az3MGqQnnHHOFThHNcyCRcPmd/0JW2OQiEqnz5e
  CXIsOfx2YXUHuPRKmqO/RWHx2yGu3lGuNpqHeouGrfcQf3zYr
  BH5jgxeEd627w6cE2Ty0KVgCwVMJd0ULZDlHQ8pqmGTRDkBPaQ
  GC6liX9vRNarMZKIGQB9EqhskWsVe4WkQAowEumFlDYPqVP4n
  3wkgM5Ks0Myjg-----END RSA PRIVATE KEY-----

12
Details of a certificate

• more /.globus/usercert.pemBag Attributes   
  friendlyName Dane Skow 995399's ID   
  localKeyID 53 E0 A1 4A 57 DE 10 E6 79 DF DD AF
  DA 7D 4F 94 AD 90 E3 51subject/Odoesciencegrid
  .org/OUPeople/CNDane Skow 995399issuer
  /DCnet/DCes/OUCertificate Authorities/OUDOE
  Science Grid/CNpki1-----BEGIN
  CERTIFICATE-----MIIDFjCCAf6gAwIBAgICAMQwDQYJKoZIh
  vcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCg
  mSJomT8ixkARkWAmVzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSB
  BdXRob3JpdGllczEZMBcGA1UECxMQRE9FIFNjaWVuY2UgR3JpZ
  DENMAsGA1UEAxMEcGtpMTAeFw0wMjA1MTYxNjE2MjlaFw0wMz
  A1MTYxNjE2MjlaMEkxGzAZBgNVBAoTEmRvZXNjaWVuY2Vncml
  kLm9yZzEPMA0GA1UECxMGUGVvcGxlMRkwFwYDVQQDExBEYW5l
  IFNrb3cgOTk1Mzk5MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQ
  KBgQDiUmnCnjw/0cpL9JSzc6VEwUzVkB72eOuSjWVQ1NSJq
  IEm2f6q/7TA8xtSXaOm6wIMhqaN95iS0Klmavd3gaUa4e0rvk
  Kltt7dmqhVU7p9DI74TnpQfJLV/CICIKcWumfO3QBnsLPPUY
  y4n819LGhgSVjrXWOymQxODp9t31QIDAQABo2AwXjARBglghkg
  BhvhCAQEEBAMCBeAwDgYDVR0PAQH/BAQDAgTwMB8GA1UdIwQY
  MBaAFFQXiMoDwTkmuFWmxJn0KwKrvgDpMBgGA1UdEQQRMABD
  WRhbmVAZm5hbC5nb3YwDQYJKoZIhvcNAQEFBQADggEBAGKmQ5
  aNxFzCHlnfhttEQqBH1lz/3n1Whfd4iyyYzZqlcOb8U7L3/ow
  vB4O4zOAt5v4echSrQxxEmp8O8T0ZZlN9ybOwUBhy5usTczOpT
  mdokuCuNUsICqIBY6iaQlaq5yDoILCjt30yslDoToinJfx
  OsOHltscNWCzNUju65cnz3HjqmdHqGWOLOozmrOvoRc44y0Br
  oJ5TF79YKl7wbH8JbiwL4EOI85EOzPkQC9ZGoo2adZFa5zY/
  OFk5ekiXvk5mTMnXbTQF8kdTi5pG7QwF9KL7f4Nctux85OgnPo
  F5VZWgFTjyrRYuiKHaZEJaczAj4JXIjsAqzH800/U-----E
  ND CERTIFICATE-----
• Human readable version available via openssl
  x509 text in /.globus/usercert.pem

13
Timeline (contd)

• September 6-8, had first round of followup with
  users and their management to explain the problem
  and why removal from the gridmapfile is
  insufficient.
• September, discussed with CA admins their policy
  on certificate revocation. No proof of
  association nor compromise.
• September 25, Tested SAZ revocation of site
  access for compromised certificates
• Oct 15 , raised issue with LCG Security Group on
  appropriate response.
• Agreement that site blocks until CRL issued may
  be prudent.
• Some concern about only triggering on real
  exposures.
• Agreement to distribute lists of compromised
  certificates to collaborating sites in the
  grid.
• Complaint from at least one CSIRT about noise and
  unknown expectations.
• Concerns about DOS and spoofed reports.

14
Timeline (contd)

• Oct 15,
• repeated scan
• 3 directories remained open
• 1 new exposure had occurred
• Tested CRLs
• 2 certificates had been revoked
• 1 was stuck in process

15
Certificate Revocation Lists

• Where do you find them ?
• Supposed to be referenced in the certificate
• List of CAs useful reference page
• http//marianne.in2p3.fr/datagrid/ca/ca-table-ca.
  html
• Guess what they look like ?
• -----BEGIN X509 CRL----- MIIBmTCBgjANBgkqhkiG9w0BA
  QQFA
• -----END X509 CRL-----
• Need to use tools to compare contents
• Certificates identified by serial number only
• Case of hex serial number not standard

16
Followup Issues

• What constitutes a private key compromise ?
• To prove it, one has to crack the private key
  encryption.
• Do we run GridCrack on our filesystems regularly
  (ala passwd/shadow checks) ?
• If anything else, how does one establish trust
  between the CA and the reporter ?
• Correct assessment of exposure
• Correct association of key to certificate

17
Followup II

• What coordination between resource providers,
  VOs, users, is necessary ?
• Learn of suspected compromised identities
• Trusted communication chain
• Agreement on compromise
• Determine appropriate scope of response
• Is disable everywhere overkill ?
• Investigate the problem
• Coordinate forensics investigations
• Present conclusions and summarize confidence
• Remediate the problem
• Issue the all clear
• Agree on followup responsibilities

18
Followup III

• Incident Response
• How does the case of compromise of a host/service
  private key differ from this ?
• Are there restrictions on types of access ?
• Are there differences in service to service
  transactions ?
• How does case of application hole exploit differ
  from this ?
• Does the grid contain its own advertisement (ala
  NIS) ?

19
Followup IV

• Authorization handled by gridmapfile for each
  resource.
• Think of a gridmapfile as an /etc/passwd file on
  a host
• Authorization done by DN (Distinguished Name)
  only
• How to deal with replacement certificates with
  same DN ?
• Maintenance of gridmapfile either manual or
  disconnected from incident response teams.