Network Operations Centers Training Programs

1
Quilt NOC Workshop Panel Discussion CENIC
Indiana University/Global NOC WiscNetPacific
Northwest GigaPopOctober 3, 2007
Network Operations CentersTraining Programs
2
October 3, 2007
NOC Training Programs WiscNet Kika Barr
3
Who We Support

• 53 Higher Education Institutions
• Over 300 k12s
• 12 CESAS Cooperative Education Service Agency
• 18 Technical Colleges
• 13Library Systems
• 36 Government Institutions
• 1 museum
• I hospital
• BOREAS-Net

WiscNet
4
Staff Workload

• 100-200 tickets per month

WiscNet
5
Our Staff

• Size
• 2 full time tech support specialists
• 1 ¾ time tech support specialist
• 1 Network Architect
• Mix
• The Tech Support Specialist are strong
  generalists
• Yet, the staff tend to divide itself on the level
  of the expertise by service (Jack of all trades,
  master of some)

WiscNet
6
Recent Training Content

• Training offered to staff in the last 6 months
• Seminars about Security
• Fiber training
• Fiber Optics training
• Firewall training
• Cisco related training
• Juniper training

WiscNet
7
Training Resources

• Recent training was designed and provided by
• Vendors
• Cisco
• Packeteer
• Engineers on staff

WiscNet
8
Training Budget

• We allocate training budget from the organization
  budget
• Try to stay local to save on expenses
• Take advantage of vendor training opportunities

WiscNet
9
Training Logistics

• Methodology for scheduling training
• When training is out of the office, we normally
  just send 1 tech support member
• When training is in house, our NOC is made aware
  that staff is on training. Tech Support
  specialist have their laptops on with IM so NOC
  can reach us if there is an emergency
• Voice mail to Tech Support goes to
  support_at_wiscnet.net mail list
• Where do you train?
• In house using conference rooms. No training
  rooms available
• travel

WiscNet
10
Lessons Learned

• If you dont use it, you lose it
• Consistency and relevance
• Delicate balance between overdoing it and not
  doing it enough

WiscNet
11
Training Program Example

• WiscNet Support Staff if Tier 2
• Training as needed
• Have been discussing with engineers about
  internal training and a more structured planned
• Competencies
• Troubleshooting
• Ticketing
• Tools
• Customer Services
• Service Expertise
• Optical networks around the corner

WiscNet
12
Demonstrated Competencies

• Excellent phone skills
• Excellent understanding of WiscNet services
• Technical proficiency in several areas of
  technology router configuration, DNS,
  troubleshooting, firewall configuration, WAN
  infrastructure, and Optical networks

WiscNet
13
Keeping the Program Current

• Challenge for us because of time constraints
• Hoping to work with WiscNet engineers to design a
  more predictable training train

WiscNet
14
October 3, 2007
NOC Training Programs WiscNet Kevin Schmidtke
15
Who We Support

• Number and type of sectors your organization
  supports
• UW Madison Campus network backbone, wireless
• WiscNet statewide educational network
• BOREAS Net regional optical network, research
  focused

WiscNet
16
Staff Workload

• of incidents opened per month
• 300 Clarify cases opened on average
• WiscNet cases 68
• UW Madison cases 29
• BOREAS Net cases 2

WiscNet
17
Our Staff

• Size of Staff
• 10 staff
• 7 Professional Senior
• 3 Professional Junior
• 1 Process Procedure coordinator
• Staff Mix
• All Junior staff scheduled with a Senior staff
  member. Seniors perform in a lead capacity.

WiscNet
18
Content

• Examples of training offered to your staff in the
  last 6 months
• Infinera DNA application training
• Infinera hardware configuration training
• Modified WiscNet process and procedures

WiscNet
19
Training Resources

• Who designed the content and provided the
  training for your staff in the last 6 months
• Most training was developed in-house
• Most training was delivered by subject matter
  experts on staff

WiscNet
20
Funding Sources

• Two Sources
• New network support contracts are expected to
  deliver training to all staff as part of the
  service rollout and acceptance
• Training budget allocation for each staff.
  Traditionally this is used for specific
  professional development

WiscNet
21
Training Logistics

• Scheduling training
• Training is typically presented in a minimum of
  two sessions to ensure all staff can attend.
  Rarely do we have an all-staff training session
• Training facility
• Several training rooms are available in data
  center complex

WiscNet
22
Training Program Example

• Knowledge Database Repository
• All formal training results in an article or a
  series of articles
• Many departments have instances of the knowledge
  database and articles can be shared from one
  instance to another
• Only the originating instance administrator can
  modify the content of the article

WiscNet
23
Training Program Example

• Knowledge Database instance administrator
• Creates all new knowledge database articles or
  recruits a subject matter expert for content
• Reviews all articles originating from another
  department
• Is responsible for making all NOC staff aware of
  new process or procedure
• Ensures all staff review and accept
  responsibility for content of articles

WiscNet
24
Training Program Example

• Knowledge Database Lifecycle
• All articles start out in draft form
• All articles are set to team review
• All staff members sign off on content of the
  article
• All articles are reviewed at a minimum of 6
  months for accuracy and relevance
• Articles are reviewed by Instance Administrator
  and/or subject matter expert

WiscNet
25
Lessons Learned

• Transition from hardcopy to online knowledge
  database required trust in materials.
• All articles are keyword searchable. Review
  failed searches to determine better ways to
  define articles
• Keep a copy of knowledge database on stand alone
  server on separate UPS

WiscNet
26
October 3, 2007
NOC Training Programs Pacific Northwest
GigaPop Linda Hornung
27
Who We Support

• PNWGP
• University of Washington Campus
• University of Washington Medical Centers
• Washington State K-20 network
• TransitRail (4/2007)

PNWGP
28
Staff Workload

• Our NOC opens 900-1000 tickets per month across
  the various customer bases we handle

PNWGP
29
Our Staff

• Full time staff in the NOC and Computer
  Operations, who provide 2nd and 3rd shift
  coverage.
• Four tier 3
• Seven tier 2
• Five tier 2 for off-shifts.
• New NOC students 6 students work 4pm-8pm M-F,
  and 9am-6pm Sat/Sun. Considered tier 1.

PNWGP
30
Recent Training Content

• Several of our staff have been to Cisco router
  classes and Cisco ONS classes.
• Several others attended Juniper ONS classes
  provided on the UW campus.
• Professional development courses such as
  leadership, communication, project management,
  team building.
• Many presentations by Network Engineering and
  Network Architecture teams on new services and
  network design that have recently been
  implemented.

PNWGP
31
Training Budget

• No defined budget, but this past year has seen a
  tremendous increase in training opportunities!
• We have been able to have training from outside
  companies, our vendors, the UW Professional
  Development program, and many in-house experts.

PNWGP
32
Training Logistics

• Because of the different shifts and the need to
  provide continuous coverage, we will always offer
  at least two different training sessions for NOC
  and other interested staff.
• Usually train in local conference rooms.
• Vendor classes have been held in larger
  facilities on campus. Always in well-connected
  facilities. ?

PNWGP
33
Lessons Learned

• The staff that usually does our NOC training must
  have taken a Train the Trainer course.
• Have defined training goals.
• Use mixed media dont just talk.
• Engage the audience through occasional questions
  or scenarios.
• Provide take-away reference material.

PNWGP
34
Training Program Example

• Tier 1 staff the focus is on escalation within
  the NOC and being able to gather appropriate
  initial information.
• Competencies
• Troubleshooting
• Ticketing
• Tools

PNWGP
35
Training Topics for Tier 1 Staff

• Architecture overview
• Services overview
• Customer information
• Monitoring
• Opening trouble tickets
• Handling and issuing outages
• Escalation

PNWGP
36
Demonstrated Competencies

• Basic phone skills (answer, hold, transfer)
• Searching tickets and email for previous
  information
• Using tools to find out information on equipment
  location, customer contacts, circuit ID, etc.

PNWGP
37
Keeping the Program Current

• Continuous project to keep the content updated
  and accurate
• Reliance on an internal wiki for procedure and
  information

PNWGP
38
October 3, 2007
NOC Training Programs CENIC Cindy Abercrombie
39
Who We Support

• CalREN Community (-approx site count)
• California Community Colleges - 112
• University of California - 11
• California State University - 25
• California K-12 System - 74
• California Private Institutions - 9
• Associates and sites outside California - 11
• NLR
• Layer 1 Services
• Support equipment / deploy services across 250
  sites
• Pwave, CIIX, TransitRail, AARNet
• Support equipment and services

CENIC
40
Staff Workload

• CalREN Monthly Ticket Stats
• Avg 670 tickets opened
• Community requests for support and outages
• Deployments, upgrades, maintenance
• Avg 375 support tickets resolved
• Community requests for support and outages
• NLR Monthly Ticket Stats
• Avg 105 tickets resolved
• Deployments, outages, maintenance

CENIC
41
Our Staff

• 15 NOC Engineers
• 1/3 of our team have gt 5yrs. Network support
  experience
• 2/3 of our team have 2-3 years combination
  LAN/WAN and end-user technical support, with 1-2
  years large-scale network support
• Other Network Operations Support Resources
• Our senior engineering team works with us daily
  on all types of request
• Our Sys Admin team responds to NOC needs quickly
  and provides support to our community too

CENIC
42
Content

• Examples of Training Provided
• Cisco 15454 MSTP Test and Turn-up
• Configuring BGP on Cisco routers
• Cisco CRS-1 Essentials
• DNS in Practice

CENIC
43
Training Resources

• Cisco
• Global Knowledge
• CENIC Senior Engineers
• Outside Specialty Organization

CENIC
44
Training Funding Logistics

• Methodology for scheduling training
• 2 training sessions on same topic, sometimes
  weeks apart or months apart
• Reduce upgrade, maintenance and deployment
  activities
• Reduce NOC coverage and include senior
  engineering resources
• Purchase bulk training hours and use Cisco
  Learning Credits

CENIC
45
Lessons Learned

• Discussing training opportunities during
  recruiting process
• Coordinate training with at least two people at a
  time, the more the better
• Purchasing bulk training hours is more cost
  effective than a couple of courses at a time
• If course feedback for a particular trainer is
  good, request the same trainer for future
  courses
• Make training and professional development a
  regular part of communications and schedules

CENIC
46
Day1 Training Program

• Start of a Shift
• Tools
• Process
• Escalations
• Schedule Maintenance
• Outages
• Announcements
• Contact Information
• Where
• Updates and requests for info

CENIC
47
Non-Technical Training Topics

• High visibility sites
• Education acronyms
• Our role in other network support organizations
• Preparing turn-over work for the next shift
• Standard signatures and greetings
• How we do lunch
• Program your cell phone
• How to report you are delayed or absent

CENIC
48
Demonstrated Competencies

• Using tools to find out information on equipment
  location, customer contacts, circuit ID
• Searching tickets and email for previous
  information
• Document and articulate the known facts of the
  current issue
• Know how to use our vendors processes

CENIC
49
Keeping Network Operations

• Everyone owns network operations
• Sharing is caring keep wiki current, send
  email
• Ask questions to understand b/c theres too much
  to memorize

CENIC
50
October 3, 2007
NOC Training Programs IU/Global NOC Steve Peck
51
Who We Support

• 17 Campuses, University System, StateNet,
  GigaPoP, Regional Aggregation point, National
  Backbones, International Connections (total
  includes separate networks supported by the
  national backbones)

IU/Global NOC
52
Staff Workload

• 906 incidents a month

IU/Global NOC
53
Our Staff

• Size of Staff
• 25 Service Desk
• 17 Network Engineering
• 7 Systems Engineering
• Staff Mix - Service Desk
• Shift Supervisors (Day Night shifts)
• Senior Technicians (at least one on every shift)
• Technicians (including hourlys)
• Off Front-Line support personnel (5 people)

IU/Global NOC
54
Our Staff

• Staff Mix
• Network Engineering
• Network Engineering Team
• - Day-to-day problem assessment / resolution,
  - After Hours on-call rotation.
• Network Planning Team
• - Longer term network planning tasks.
• Systems Engineering
• Software Developers
• System Administrators

IU/Global NOC
55
Content

• Service Desk
• New Internet2 Network
• Connecticut Education Network
• Automated reporting tool
• IU Core network transition to GRNOC Eng
• Indiana GigaPoP changes
• CIC OmniPoP follow up
• NLR L2 VLAN / Circuit determination
• New database circuit schema

IU/Global NOC
56
Content

• Engineering
• Internal training Ex Ciena CoreDirector
  training from resident in-group expert.
• Training for new staff as necessary Ex
  commercial Cisco BGP training for a new Network
  Engineer.
• Systems Engineering
• One-on-one training of new staff
• Ad-hoc training from in-group experts on various
  technologies and systems.

IU/Global NOC
57
Training Resources

• Service Desk
• Senior staff picks topics and focus, utilize
  internal expertise to provide and design the
  content
• Engineering
• Internal formal and informal training sessions by
  in-house experts.
• Commercial training for individual engineers as
  needed.

IU/Global NOC
58
Training Resources

• Methodology for scheduling training
• Training Facilities
• Service Desk
• Definitely a challenge! Once a month from 400 pm
  to 700 pm two techs man the SD use conference
  room meetings are video recorded

IU/Global NOC
59
Lessons Learned

• If you have access to excellent internal training
  resources within your organization, you HAVE to
  utilize them!
• Must have a clear picture of staff
  responsibilities to ensure that appropriate
  training takes place in a prioritized manner.

IU/Global NOC
60
Training Program Example

• From I-Light Service Desk Training Session
• There are three main components to what
• we call I-Light
• Layer 1 fiber optic network
• Layer 2/3 network on the fiber
• Copper based circuit network

IU/Global NOC
61
Training Program Example

• From I-Light Service Desk Training Session
• The I-Light NOC will be monitoring and supporting
  TDM border routers at universities, and taking
  calls from their primary technical contacts.
  However, calls will sometimes be from end users.
• The dmarc for all of I-Light is the customer
  facing port on the last piece of equipment
  I-Light owns. TDM customers are used to IHETS
  helping troubleshoot beyond that dmarc.

IU/Global NOC
62
Training Program Example

• From I-Light Service Desk Training Session
• Most of the universities have Cisco 7206 VXRs,
  Cisco 2620 XM routers, or possibly 1700 series
  routers. These routers use several different
  versions of IOS. Some have QOS, some don't.
  However, all commands should be uniform.

IU/Global NOC
63
Panel Contact Information

• Linda Hornung
• PNWGP
• lwright_at_cac.washington.edu
• Steve Peck
• Indiana University/Global NOC
• speck_at_iupui.edu
• Kevin Schmidtke
• WiscNet
• kevin.schmidtke_at_doit.wisc.edu

• Cindy Abercromie
• CENIC
• cindy_at_cenic.org
• Kika Barr
• WiscNet
• barr_at_wiscnet.net