Carol Sabbar and Jim Walsh

1
Towards a Better September Controlling Residence
Hall Computing

• Carol Sabbar and Jim Walsh
• Carthage College

2
Surviving September?

• Can it really get better?
• Do we have any control at all?
• Who we are Lowly IT people from a liberal arts
  college with a pretty limited budget and about
  1,200 resident students whose computers are all
  infected

3
Fall 2003 Blaster!

• What happened?
• Blaster and Welchia infected nearly every student
  computer on campus
• DoS attacks shut down the core switch
• We shut down whole residence halls to protect the
  core
• We thought that students could help themselves
  or not
• We turned off ports for hundreds of rooms
• We cleaned and patched hundreds of student
  computers

4
We survived but

• We resolved to never let that happen again
• We had to figure out something
• Identify what happened and why
• Figure out how to prevent it
• Figure out what we could afford

5
Determining the Causes
Vlan110 is up, line protocol is up Hardware is
Cat5k Virtual Ethernet, address is 0008.7c6d.d800
(bia 0008.7c6d.d800) Description Hedberg User
VLAN Internet address is 10.7.0.1/16 MTU 1500
bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set ARP type
ARPA, ARP Timeout 040000 Last input 000000,
output 000000, output hang never Last
clearing of "show interface" counters 2d00h
Input queue 1/75/1139/55 (size/max/drops/flushes)
Total output drops 0 Queueing strategy
fifo Output queue 0/40 (size/max) 5 minute
input rate 27000 bits/sec, 37 packets/sec 5
minute output rate 149000 bits/sec, 25
packets/sec 13871279 packets input,
2992610801 bytes, 0 no buffer Received
2214895 broadcasts, 0 runts, 0 giants, 170
throttles 0 input errors, 0 CRC, 0 frame, 0
overrun, 0 ignored 10781241 packets output,
3862081410 bytes, 0 underruns 0 output
errors, 0 interface resets 0 output buffer
failures, 0 output buffers swapped out
6
Identifying the Culprits

• Seeing the traffic
• Understanding how worms work
• Finding infected computers
• How many are patched
• How many have anti-virus
• The patch is not the fix, and anti-virus wont
  clean these
• Finding the secondary problems
• Spyware
• Peer-to-peer

7
Looking forward to 2004

• Containing outbreaks
• Can we isolate them to a building?
• To a room?
• Preventing infections
• Can we mandate patches?
• SP2 good or bad?
• Can we mandate anti-virus?
• Chasing down infected machines faster and easier
• Can we identify infected machines?

8
What we couldnt do

• Some proven solutions just wouldnt work for us.
  These included
• Perfigo or Bradford software too expensive
• Anything requiring an agent on a student computer
  too many installations to do
• A Packeteer for ResNet users only too expensive
• Broadcast storm control anomaly related to our
  wiring plant in res halls

9
Isolating Outbreaks

• Subnets
• Already in place but only the base
• ACLs (details on next screen)
• Isolation to the building
• Moving them out to the edge switches
• Required new hardware
• Required outside expertise
• Storm control on ports
• Problematic in our environment
• Maybe fall 2005

10
ACLs on Cisco Switches

• Extended IP access list 180
• deny icmp any any (92 matches)
• deny udp any any eq tftp
• deny udp any any eq 135
• deny tcp any any eq 135
• deny udp any any eq netbios-ss
• deny tcp any any eq 139
• deny tcp any any eq 445
• deny udp any any eq 445
• deny tcp any any eq 4444
• permit ip 10.12.128.0 0.0.127.255 host
  10.2.3.4
• deny ip 10.12.128.0 0.0.127.255 any (1410
  matches)
• permit ip any any (23199006 matches)

11
Preventing infections Part 1

• Distribution of Symantec anti-virus in summer
  2004
• Changing our Symantec licensing to make it free
• Mailing out the CDs, Update before you get
  here!
• Handing them out at check-in
• We do not yet check for its existence before
  network access

12
Preventing infections Part 2

• Patching
• Is SP2 really recommended?
• In November, we decided yes
• Working on PatchLink for on-campus computers, but
  requires agent for student computers
• We do not yet require any specific patches for
  network access

13
NetReg a Tool for the Hunt

• Required registration in fall 2004
• Decreases identifying infected machines by
  several steps
• Turned off rooms posted on our web site
• Need to have someone well-versed in Linux to set
  it up

14
Fall 2004 Any Better?

• No dorms totally shut down in Sept
• We cleaned less than 20 computers in September
  and October
• Infections seldom traveled from building to
  building
• Infected machines were identified and ports
  unplugged the same day
• A different story in November started in an
  administrative building with no ACLs

15
New Problems

• With the elimination of the bulk of virus-related
  outages, we experienced other problems
• Rogue wireless/wired routers
• More spyware
• Issues with Windows settings like connection
  bridging and 802.1x
• Some education issues related to NetReg

16
Looking forward to fall 2005

• Do all the same as last year
• Increase functionality of NetReg?
• Use a product like Cisco Network Access Control?
• Deploy more switches that can discard DHCP
  response packets
• Deploy our own wireless in res halls
• Were open for suggestions