BP 401 - Admin Zero to Hero in 60 Minutes

1
BP 401 - Admin Zero to Hero in 60 Minutes

• The question is no longer, "How can we?"
• The question now is, "How should we?"
• Andrew Pollack, PresidentNorthern Collaborative
  Technologies

2
Language Note

• I realize that for some of you, English is not
  your primary language, and for others, my accent
  is not the same as yours.
• If you are having trouble understanding me during
  this talk, please raise your hand and I will try
  to slow down and speak more clearly.
• Thank you.

3
Wireless Devices

• Wireless device noises are rude in any language.
  Please take a moment to turn off any of the
  following
• Cell Phones
• Scheduler Devices
• Pagers
• Alarm Clocks
• Pacemaker low-battery warning alarms
• Anything else you are carrying on or about your
  person which may make noise during this
  presentation.

4
About this Presentation

• A "best practices" session is different
• This is not a list of product features.
• This is a practical 'field guide' of which ones
  to use, and why.
• Focused on What and Why, pointers to resources
  for how.
• Designed for re-use
• These are not empty bullet points.
• The details you need are in this text.
• The Goal of this Presentation
• Provide an overview of what you should be
  thinking about as an administrator
• Provide a trail map for finding out more, and
  implementing the ones you find of value
• Help you start thinking in terms of the big
  picture rather than being constantly swamped by
  the details

5
Agenda

• Who am I to be telling you anything?
• The Scenario Setup
• Server Stability Management
• Security Management
• Mail Management
• Database Management
• Client Software Management
• End User Support

6
Who am I To Tell You Anything?

• Andrew Pollack
• President, Northern Collaborative Technologies
• 2003 IBM Lotus Beacon Award Winner
• 1999 Lotus Beacon Award Finalist
• Administrator Developer since version 2.0
• Member of the Penumbra Group
• Firefighter Cumberland, Maine!
• Lieutenant of Engine 1, Ladder 7, Heavy Rescue,
  RIT, Special Operations
• In firefighting, just like Server Administration
  it's all in the planning
• Why We're Here
• To learn and grow as human beings
• The question has changed, now it isn't "How Can
  we," it's "How Should We"
• Also, I'm here because it makes the phone ring
  more

7
A Typical Environment

• Three Offices
• Southeast The Home Office
• Mid Sized, easy to get to, excellent net
  connection
• Southwest A Production Facility
• Mid Sized, easy to get to
• Northeast RD
• Small Office
• Terrible Airport Access
• Heavy Ground Traffic
• Weather Power Issues
• Expensive Travel Costs
• Then theres you
• The new Domino
• Administrator

8
Server Load Hardware Choices
9
Clustering vs. Giant Boxes

• Benchmarks are just statistics, and we know how
  much we should trust those.
• Would you really put 12,000 users on one server?
  20,000? More?
• Domino clusters do not shared any hardware or
  part of the same operating system. They are fully
  redundant.
• Balance the load across all the servers in the
  cluster, but make sure that if one goes down, the
  others can handle the load without crashing.
• A performance drop is acceptable for a brief
  period in most shops.

10
Clusters Provide High Availability, Low Cost
11
Domino Clustering is REALLY Easy

• Put databases on both servers
• Make sure they replicate, and have proper access
• Select the servers in the directory
• Click "Add to Cluster"

12
Considering Peak Loads

• We think of number of users dont do that.
• Think number of concurrent users.
• If you run three shifts, and only one shift is
  active at a time, you may be able to use smaller
  hardware.
• Think total disk space.
• Disk usage is critical on the server, even if it
  isn't in use it costs the server resources to
  keep indexes and run checks.
• In some customer sites, mailbox size dictates
  server count because of drive space limitations
  and the cost of massive storage networks.
• For more information about clustering
• JMP102 An Introduction To All Things IBM Lotus
  Domino Clustering -- Gabriella Davis

13
Software Version Management
14
Operating System Choice

• Which operating system is the best?
• Avoid politics, religion, and operating system
  preference discussions at the dinner table
• Either choose an OS that your staff knows well,
  or send them to school
• All operating systems need to be patched and
  updated. Keeping up with these is required for
  stability
• Make a choice that is not unique in your company
• Test, Test, Test
• Watch out for case sensitivity when moving off
  Win32
• Debugging can be very difficult because the
  initial hit to a resource is case sensitive, but
  once the object is in the cache, it may not be.
• BP403 Best Practices IBM Lotus Domino for Linux
  -- Daniel Nashed

15
Remote Server Administration

• No matter what tool you use, always use
  encryption
• Many tasks you might think you need remote
  control software for, can be done with the Web
  Administration Tool and the Lotus Domino
  Administration Client
• Editing the NOTES.INI on the server
• Starting and Stopping Windows Services
• Use the Server Controller and Java Console
• These can restart even crashed servers remotely
• Start the server with "jc"
• Stat the console from the Notes program directory
  "jconsole.exe"

16
Remote Control Software

• Make sure it is set to lock the console
  automatically if your connection drops
• Make sure it requires encryption for connections
• Keep up with the vendors patches and updates for
  the server side
• Security patches could be critical
• These ports are scanned constantly

17
ADMINP is your best friend

• Properly configured, this will do a lot of the
  hardest and most tedious work for you
• Distribution of new databases to multiple servers
• User move, add, or change requests
• This becomes more and more important with each
  new version of the IBM Lotus Domino server
• Each server should have a replica of the
  "ADMIN4.NSF" from the administration server
• For more information
• ID113 Maximize the Power of AdminP in IBM Lotus
  Domino -- Kathleen McGivney, Susan Bulloch

18
Local Staff

• Nothing is better than local staff
• Before doing any kind of remote access work,
  compile a list of local contact staff with phone
  numbers and availability
• Have someone check the cdrom trays you do not
  want to reboot to a setup disk
• Nothing is worse than local staff
• Control access to the sever
• More on this topic when we talk security

19
Monitoring and Event Handling

• Use Events Be the First to Know
• Easy to set up
• Know about problems before your phone rings
• Fix problems before the boss calls you
• Make sure to log them, so he knows what you do
• Event notices make great justification tools for
  new servers!
• For more information
• BP407 What are Your Servers Trying to Tell You
  Now The (Even) Easier Route to IBM Lotus Domino
  Reporting Logging -- Gabriella Davis

20
Power-off Recycle Devices

• When all else fails, sometimes you need to power
  cycle a machine from 3000 miles away
• Inexpensive power modules can be commanded to
  recycle power with a 5 second power down pause
• Controlled through serial port
• Include "watchdog" software
• Many devices on the market
• Some include remote shell access
• Some include Web browser control

Heres what I use http//www.cpscom.com/gprod/ipn.
htm
21
Developer Management

• Sir, please step away from
• that Designer Client.

22
Deployment Policies

• These are a good thing, and you should have some.
• Questions to answer with your deployment
  policies
• Who decides when a database has been tested
  enough?
• Who will be called when a problem is reported?
• Do you have a contact number for this developer?
• How will you know when the database is no longer
  in use?

23
More Deployment Policy Questions

• How big is the database expected to get?
• What servers does it need to be on?
• Is external replication required?
• How volatile is the access control going to be?
• What kinds of agent code will be running at the
  server?
• Server side java agents? Agents that call COM
  objects?
• File System Access? ODBC or Connector LSX Use?
• API Calls?

24
Do Not Modify the Domino Directory

• Nothing impacts performance more than changes to
  the Domino Directory
• There are two critical view indexes in the Domino
  Directory
• ServerAccess
• Users
• If the indexer is busy doing other things in that
  database, these updates will take longer
• If these indexes are not up to date,
  authentication and access rights may not be
  granted to users

25
Java Agents Must Be Tested at Full Scale

• Multi-threading is so powerful, you can shoot
  yourself in both feet at once
• Very easy and common mistakes in Java agents can
  kill production servers easily
• Unlike LotusScript, when writing Java agents
  programmers must call "recycle()" on every object
  you instantiate, or their parent document
• In test, it is frequently possible to get away
  with simply recycling the "session" object when
  the agent terminates
• In production, this kills servers when the agent
  handles a large number of documents in a loop,
  among other things
• Yes, I know this from bitter experience

26
Restricted vs. Unrestricted Agents

• Unrestricted agents can do to things outside the
  scope of the agent itself
• Access the operating system
• Access files on the server important ones
• Reboot or shutdown the server
• If someone needs to run an unrestricted agent,
  you need to understand why

27
Security Management
28
The Five Pillars of Security

• Physical Server Security
• Operating System / File System Security
• Lotus Domino Server Access
• Certificates Cross Certification
• Public / Private Key Certification
• Cross Certification
• Server Access Settings
• Database Access The ACL
• Document Access Reader Names

29
Notes Client Side Security

• Guard Your Certifier
• Dealing with a compromised certifier
• Assume Users have Designer
• It's easy to get
• Obscurity is not Security
• Encrypt Workstation Data
• Escrow ID Files
• Preventing Workstation Copies
• Third Party Tool dotNSF Tools noCopy
  www.dotNSF.com
• Client to Server Communication Encryption

30
Browser Access Security

• Obscurity is not security!
• This is the 1 issue on Web sites
• URL Hacking
• NoteID Crawling
• Common Word Crawling
• /database.nsf/knownViewName/ltinsert word heregt
• SSL Preventing Man in the Middle Attacks
• Creating an SSL Key Ring
• Obtaining an SSL Certificate
• An authority unto yourself Are you trusted?
• Buying an SSL Certificate
• Deploying an SSL Key Ring to Domino

31
Securing the Other Protocols

• Understand your ports
• If your server faces the internet, put a firewall
  in front of it
• Many of the server tasks listen on a port,
  understand them or dont load them. Particularly,
  LDAP and SMTP can give away a lot of valuable
  information if improperly configured
• If you dont need a protocol, shut it down
• If nothing is listening on a port, that port is
  secure. Well, mostly.

32
Password Guessing isnt Just Browsers!

• User's "Internet" passwords are frequently less
  complex than their Notes ID Passwords Use the
  tools to enforce complexity
• It is now very common for hackers to "Name Guess"
  via POP3, SMTP, and even "Harvest" names from Web
  sites, e-mail addresses, and open LDAP ports
• Once a name is guessed or harvested, POP3 or
  other protocols are used to guess passwords
• With a name and password, spammers can use your
  server using an authenticated username

33
Mail Management

• This is probably why many of you
• came here in the first place.

34
Notes Mail Routing

• Servers on the same Notes Named Network
• Should be able to find each other "by name"
  without connection documents with TCPIP, this
  would be DNS
• Servers on the same "named" network route mail
  automatically no connection document is needed
• This is a "least cost" indicator to Domino's
  routing cost matrix
• Use this to your advantage
• Set up your named networks to reflect your
  network's faster and slower links. Put only
  servers that have excellent connectivity on the
  same "Named Network"

35
Connection Documents

• Connection documents tell servers which are not
  on the same "Notes Named Network" how to find
  each other
• They're also used for replication, but we'll get
  to that later

36
Internet Mail Routing -- Turning off SMTP inside
the Network

• If you turn off the SMTP Inbound Listener, local
  Windows clients which have been infected with a
  virus, worm, Trojan horse, or spy-ware
  application cannot send mail through your
  servers.
• This also eliminates accidental or deliberate use
  of your internal servers for spam routing.
• Even if you require password access for SMTP mail
  sending, password guessing is now quite common.
• If you disable SMTP Outbound on your servers, it
  will force the mail to route through your single
  gateway. In many cases this is a more secure
  method and provides greater traffic control on
  your network.

37
Using a Single Internet Mail Gateway

• Server Documents (all but the server that will
  route smtp)
• Set "SMTP Listener" to Disabled
• Set "Routing Tasks" to "Mail Routing" but not
  "SMTP Mail Routing"
• Create a "Foreign SMTP Domain" Domain Document
• Route . to "OurFakeName"
• Create a Connection Document
• Type SMTP
• Source Server The domino server with smtp
• Destination Server MAKE UP a name
• Destination Domain "OurFakeName"
• Routing Task SMTP Mail Routing
• This method means you dont even need TCPIP as a
  protocol on your other Domino servers, because
  the routing all happens using Notes RPC protocols
  to the one server with SMTP capability.

38
Single Internet Mail Gateway -- What Really
happens?

• All the servers where SMTP Mail Routing is not a
  task, look for a route to send the mail.
• These servers see that . goes to the domain
  "OurFakeName"
• That's the SMTP Domain Document's Job
• The router task on the servers see that one
  Domino server has a connection to the
  "OurFakeName" domain so they route the messages
  to that server
• That's the connection document's job
• The server which is SMTP Mail Routing Enabled
  receives the mail in its INBOX and knows how to
  send SMTP mail directly, so it does.

39
Standardizing on a Mail Template

• Beware of Customized Templates
• Prevents Update Bug Fix
• Look at the update lists in each point release
  and note how many related to small fixes in the
  mail templates.
• Serious Performance Issues
• More views means more view indexing work for the
  server.
• Limiting Design Access to Mail Files
• People are most likely to make "quick" (untested)
  updates to the design of their mail file,
  considering it their own problem if they cause a
  problem. These people can take down your server.
• If you want additional features, look for
  "Packaged" alternative mail templates which are
  properly supported.
• openNTF.org has a very popular one, for example.

40
Managing Mail File Size SCOS

• Single Copy Object Store has been a feature for
  many years.
• It DRASTICALLY reduces disk usage by keeping one
  copy of each file no matter how many different
  people have it in their mail files.
• It's significantly better than it was, and with
  "Transaction Logging" and Domino clustering can
  be much more reliable than ever before.
• It's still a single point of failure if you do
  have a problem, everyone is affected by the
  problem.

41
Managing Mail File Size (continued)

• Take Advantage of Archiving
• Archiving can be easily set up and managed
  through policies
• Put Archives on different server, they're less
  frequently accessed and have different load
  characteristics
• Impose Realistic Limits with Quotas

42
Managing Unwanted Mail

• Don't be a Relay
• In the "Configuration" document for your server
  not the Server document, on the
  "Router/SMTPRestrictions And ControlsSMTP
  Inbound Controls" Tab
• Deny messages from the following internet hosts
  to be sent to external internet domains( means
  all) Set to ""
• This is the Default on all recent Domino versions
• Hold Undeliverable Mail
• Don't send bounce messages Frequently, the mail
  never even originated on your site and you're
  only adding to the problem
• Fighting unwanted mail is much more complex than
  this
• BP405 Controlling Spam Mail In Your Organization
• BOF509 Keeping Up with the Spammers with IBM
  Lotus Notes and Domino

43
Don't Give Away Address Information

• Verify that local domain recipients exist in the
  Domino Directory
• Pros
• Stops inbound SMTP messages send with dictionary
  style drops and name guesses from clogging your
  router
• Can make your site less attractive to spammers
  who get credit for "delivered" messages
  accepted by your server
• Cons
• Makes it easy for spammers to test for valid
  names on your server
• Consider using this if you have another tool that
  can detect multiple failed attempts from the same
  source and ban those sources at the firewall.

44
Other Message Filtering Considerations

• Using Black Lists (aka Real-time Black Hole or
  RBL)
• Many "black lists" exist that you can use
• (e.g. bl.spamcop.net sbl-xbl.spamhaus.org)
• Not 100 accurate
• Read the lists website to understand their
  criteria for listing
• Using White Lists (aka "Known Good" addresses)
• Most mail you get, is from people you've
  communicated with already
• New to version 7 of Lotus Domino, but part of
  several 3rd party tools for some time

45
Mail Filtering Tools

• Third Party Tools
• User-Interactive Products like spamJam can be
  excellent because each user decides individually
  what's wanted and what's not
• Appliance Solutions can be inexpensive and
  effective, but less user-specific
• My Recommendations
• spamJam because users really like being able to
  interact with it
• Barracuda for simplicity and price, this device
  works very well
• ASSP Open source proxy, good but scale is
  uncertain

46
Signed Mail

• Signed mail to Notes users
• Your Public Key
• Use "Files-Security-User Security" to get it or
  copy it from your Domino Directory person
  document
• Signed Mail to Internet users
• X.509 Certificates The modern standard for
  authentication
• Self Certifying
• If you create your own certificate authority,
  everyone will always have to decide accept it as
  trusted
• Excellent alternative for internal company use
• Buying Certificates or Certification Rights
• Free Certification Network

47
Importing Your X.509 Certificate

• If you obtain a personal x.509 certificate, you
  can import it into your person document in the
  Domino Directory
• Open your Person Document
• Select "Actions Import Internet Certificates"
• Once this is done, you can "sign" mail to be sent
  to users with Internet addresses

48
Verifying Signed Mail

• From Notes Users
• The Lotus Notes Public Key
• You must have their public key in your address
  book
• Verifying Signed Mail from Internet Users
• Accepting a Cross Certificate
• Do this the first time you get signed mail from a
  user
• Call the user, make sure its them sending the
  message

49
Adding a Sender's Public Key to Your Personal
Address Book

• While viewing, use "Tools Add sender to address
  book"
• Advanced tab, check to add "x.509 certificate"

50
Mail Encryption

• The Recipients Public Key is required
• The Public Key is used to create a one-way cipher
  that can only be read with the private key and
  only the user has the private key, it's in their
  Notes ID file (or other file if a non-Notes user)

51
Obtaining a Recipient's Public Key

• Notes Mail users in your domain already have it
  in their "Person" document in the Domino
  Directory.
• Notes Mail users in other domains must send it to
  you. They can copy it from their record in their
  Domino directory, or use the options in "Files
  Security User Security" to get it.
• Users can also simply send you a "Signed"
  document, and you can "Cross Certify" them when
  you receive the mail. (You'll be prompted.)

52
Adding a Sender's Public Key to Your Personal
Address Book

• While viewing, use "Tools Add sender to address
  book"
• Advanced tab, check to add "x.509 certificate"

53
Database Management
54
Deployment Policies

• Limit Designer Manager Access
• On the fly changes cause most problems
• Use Database Access Groups to Delegate Control
• Create Groups that a database owner can manage
• Example "SalesTools.NSF Editors"
• Set the database owner to be the owner of that
  group

55
The Connection Document for Replication

• A connection document is required for replication
  even on the same "Notes Named Network"
• A common error on the connection document is not
  changing the schedule to work around the clock.
  Default is 8am-10pm.
• Keep in mind that following replication, the
  indexer may be very busy. Consider having
  replication occur prior to the start of the
  normal business day.

56
Database Deployment Policies

• Track Database Usage Ownership
• Every Database must have an Owner
• Every Database must have a Review Date
• Remove Outdated or Unused Databases
• Even unused databases can load the server
• Old data represents a security, accuracy, and
  legal risk

57
Replication Topologies

• Avoid "Everyone Replicates with Everyone"
• Map Network Choke Points

58
Creating a Redundant Hub Spoke

• Two distinct local area networks or well
  connected individual networks
• One high bandwidth connection between the two
  clustered hubs
• Reduces traffic across the expensive long haul
  network

59
Client Software Management
60
Common Policy Settings

• Use policies to define ECL (Execution Control
  List) settings
• Use policies to make sure users have the right
  replicas on the local workstations
• Policies in version 7 can be much more rigidly
  enforced

61
Client Version Update Rollout

• Excellent for ROI No more touching the desktop
• Reduces support due to version/template
  incompatibility
• BP404 Best Practices in IBM Lotus Notes Client
  Deployment -- Steve Sterka, David Via
• ID117 IBM Lotus Notes Deployment Made Easy --
  Jeff Mitchell, John Paganetti

62
Handling User Support
63
Delegating Admin Roles Safely?

• Version 6.x added granularity to "Administrator"
  access
• Allows you to delegate specific areas of
  responsibility without giving complete control to
  junior administrators.
• Using the administrator task, you can allow area
  managers to register users without giving them a
  certifier.

64
Admin Roles in Version 6.x

• Full Access administrators
• Able to leap tall ACLs impervious to
  Reader-Names
• Administrators
• Use all the power of the administrator tool, but
  subject to database and document controls
• Database Administrators
• Manage databases, but not the server itself
• Full Remote Console Administrators / View-only
  Administrators
• System Administrators
• No database controls, but plenty of server setup
  access
• Restricted System Administrators
• Restricted System Commands

65
Limit Use of Full Access Administration

• Full Access Administration should only be used
  rarely, when a need to override ACL or
  ReaderNames is required.
• Grant this only to specific ID files. Make the
  administrator switch to this ID file when needed.
• Create an "Event" notification to notify
  management any time this level of access is
  granted.
• Use encryption on database you dont want full
  access administrators to read.

66
In summary

• It's no longer a question of whether or not
  something can be done, it's a question of which
  is the best way to do it and why.
• This presentation serves as a guideline, not a
  bible.
• This has been a high to medium high level look at
  the features you should be using, with pointers
  to where to find more detailed information.

67
Thank you for playing!

• Were all Lotus professionals here, please ask
  your questions so others can here the answers.
  You may also contact me directly if you like.
• Please fill out your evaluations
• The latest copy of this presentation will also be
  available at my website http//www.thenorth.com

• For those playing the home game, direct questions
  comments to

Andrew Pollack andrewp_at_thenorth.com http//www.the
north.com