OpenSSH Monitoring and Analysis for OpenSSH 5.x

1
OpenSSH Monitoring and Analysisfor OpenSSH 5.x

• Scott Campbell
• NERSC/LBNL

2
Overview

• What does the instrumented SSHD do?
• Why are we doing this?
• Overall design considerations.
• Overview of
• sshd core and related software
• transport
• bro policy analysis
• Modifications for version 5.x.

3
Purpose of Project

• Gain insight into user activity using ssh, user
  actions are opaque. We would like to fix this.
• Leverage experience and analysis from clear text
  era.
• Augment process accounting which is excruciating
  and difficult to process in a timely manner.
  Limitations such as no command arguments in AIX.

4
Design Considerations

• Focus on not impacting our users
• Minimize performance impact.
• Identical user experience.
• Minimize changes to existing software.
• Avoid additional SSH failure modes or security
  exposure.

5
More Design Considerations

• By minimizing impact on users and system admins
  we ease transition path and get buy in. Very
  important.
• Make SSHD end as bullet proof as possible put
  more fragile analysis and smarts away from
  users.

6
Functional Goals

• Record session/keystroke data.
• Record user login information.
• Identify and record SSH events such as channel
  creation and command execution.
• Track scp and sftp data movements and commands.
• Integrate PSC performance mods.
• Analyze data stream for hostile content.

7
Stuff not done

• Data is a text stream there is no notion of
  process accounting or system log level detail.
• No cleaning of data meta data such as colors
  etc still show up as junk.
• No application reassembly vi and emacs provide
  a somewhat murky picture.

8
Design Overview
SSHD
Stunnel
Stunnel
Log Parser
Bro
Analysis System
Client System
9
SSHD Design Overview

• Mechanics startup, server id and client id.
• Instrumentation when, where and defensive
  strategies for users accidents.
• Passwords the 98 solution.
• Decoupling the write.
• PSC Mods while we were in there

10
Startup

• At startup several things happen to log main
  server process.
• Message sent that sshd has started
• sshd_start 127.0.0.12222/tcp 4102_spud_2222
• Creation of server session id composed of
• ltprocess-idgt_lthostnamegt_ltportgt
• This id will be used to identify any further
  activity from this sshd instance.

11
Startup Client Connect

• When a client connects, it is assigned a session
  id equal to the pid of the child process.
• Process mechanics avoid collisions. If one
  occurs (race condition, logging failure etc), bro
  will restart counters for client ie not
  pathological.

12
Login Information
Logging on bro end looks like 1192854558.838989
2 - ssh_connection_start 127.0.0.132812/tcp
gt 127.0.0.12222/tcp 1192854559.428210 2 -
auth_ok scottc publickey 127.0.0.132812/tcp gt
127.0.0.12222/tcp 1192854559.481470 2 -
new_session SSH2 1192854559.848240 2 0
new_channel_session pty-req 1192854559.860270 2
0 new_channel_session shell
Channel Number
Bro Session ID
13
Instrumentation

• After user logs in, we begin keystroke logging in
  earnest. Most data encapsulated in data_server
  and data_client events.
• Clean any non-ascii data into the record to avoid
  binary interpretation issues.
• Bro sanitizes its end as well.

14
User Games

• Copying all data is expensive and provides a
  diminishing return.
• Limit volume of data from client/server in a
  single line.
• Limit number of lines before summary byte count.
• Count reset by client data.
• Observe most attacks happen in the beginning of
  the typing stream ie not
• ./././././././././././././././..n./../tmp/attack
  
• In particular, reading the output from a three
  hour make or gtcat big-binary-file seem within the
  bounds of experience. Plan defensively.

15
Passwords!

• The decision for a site to record password
  information can be a protracted mess.
• Simple test for passwords/phrases matching on
  pass(wordphrase) tags the next user response
  line as a credential. Optionally log it.
• Logging is a configure time option and is opt-in.
• Auditing covers both login and in line activity
  so additional logins to other sites will have
  their password data recorded as well. User
  education is important!

16
Decoupling the Write

• Failure of downstream networking or software can
  not effect the users ssh experience. Our design
  will lose security data before slowing down our
  users.
• Use of non-blocking write in sshd for sending
  data to a local stunnel socket.
• Stunnel socket has aggressive timeout to avoid
  buffering issues on the sshd side.

17
PSC Modifications

• PSC has a set of patches for OpenSSH which
  provide for significant performance improvements
  in data transfer rate.
• Benefit from performance gain greater than
  increased exposure to new security
  vulnerabilities.
• More info
• http//www.psc.edu/networking/projects/hpn-ssh/

18
Stunnel
SSHD
Stunnel
Stunnel
Log Parser
Bro
19
Notes on Stunnel

• Generic configuration.
• Data sent to logger on server end.
• Initial implementation tolerates snakeoil
  certificates only the central collector is
  required to be official.
• Cross authentication is planned for near term.

20
Log Parser
SSHD
Stunnel
Stunnel
Log Parser
Bro
21
Log Parser Design

• The log parser is a libBroccoli based converter
  for translating well structured text into bro
  events which can be (optionally) transmitted
  across the network.
• Changes made to process long strings containing
  spaces and odd characters.

22
General Idea

• Takes formatted text as input
• event_name count0,5 addr0,1.2.3.4 uristringfoo
• Converts the data types
• Event event_name
• int tmp_1 atoi(5)
• uint32 tmp_2 (uint32)inet_addr(1.2.3.4)
• bro_string_init(foo)
• Then turns it into a bro event
• event event_name(count i, addr a, string s)

23
History brosocket
(1) Start with Syslog message
Mar 10 183941 2.3.4.5 Message forwarded from
host609 sshd74334 Accepted password for user1
from 1.2.3.4 port 45674 ssh2
(2) Parse into structured text
ssh_login double1108696916 addr1.2.3.4
addr2.3.4.5 stringuser1 stringpassword
(3) Bro Generic Client reads text and
communicates event to Bro
(4) Bro acts on communication
24
Design Overview
SSHD
Stunnel
Stunnel
Log Parser
Bro
25
Bro Overview

• Continuing the tradition of separating agnostic
  events and larger policy, all security related
  decision making is concentrated with the bro
  instance.
• General design strong emphasis on auditing and
  connection details.
• Once you get that right, you have the scaffolding
  in place to make security decisions. Without,
  they are ad-hoc.

26
Full login example

• 511 ssh_connection_start 128.55.19.25334141/tcp
  gt 0.0.0.022/tcp
• 511 auth_ok scottc publickey 128.55.19.25334141/
  tcp gt 0.0.0.022/tcp
• 511 new_session SSH2
• 5110 new_channel_session pty-req
• 5110 new_channel_session shell
• 5110 data_server Last login Fri May 2
  133550 2008 from red.nersc.gov PDT
• 5110 data_server FreeBSD 6.2-RELEASE-p7
  (GENERIC) 1 Thu Aug 9 221943 PDT
• 5110 data_server
  
• 5110 data_server
  NOTICE TO USERS
• 5110 data_client ls
• 5110 data_server ls
• 5110 data_server 1 bro.tar
• 5110 data_server NODE development
• 5110 data_server bin hot.bro
• 5110 data_server block-server incidents
• 5110 data_client exit
• 5110 data_server exit
• 5110 data_server logout

27
Event Breakdown

• For a remote execution we see
• ssh_connection_start
• auth_ok
• new_session
• new_channel_session
• ssh_remote_exec_pty
• ssh_remote_do_exec
• channel_exit
• ssh_connection_end

28
Interactive Sessions

• For interactive sessions the data stream is
  monitored by several mechanisms.
• Watch server response to clients typing to avoid
  translating kahhls into ls.
• Augment login.bro for a more contemporary hostile
  command list.

29
General Detection Schemes

• Patterns/command for immediate notification. Pass
  the server_data stream through the login
  analyzer.
• In addition, look for a series of suspicious
  commands which by themselves are not clearly
  hostile, but in total represent something
  interesting.

30
Changes to 5.x

• Heartbeat moved to server.
• Key fingerprint now recorded for key based
  logins.
• String values now uriencoded.
• Made native output simpler.
• Non-TTY activity now logged - all data recorded
  for window, then test to continue.

31
Questions?

• Contact
• Scott Campbell
• NERSC/LBNL
• scampbell_at_lbl.gov