Colleague System Maintenance

1
Colleague System Maintenance

• File System Administration Maintenance Part
  Two

Cinda Goff Chuck Hauser 2005-10-30
2
Presentation Conventions

• Names (files, users, daemons) are usually in
  bold/etc/syslog.conf
• System dependent or variable items are usually in
  italics /var/sadm/patch/patchnumber/log
• File entries and output are in mono-spaced
  typegt root 8036 c Tue Apr 26 235900 2005
  lt root 8036 c
  Tue Apr 26 235959 2005
• Ä marks a line wrapped to fit on the slide mv
  Solaris_8_Recommended_log
• ÄSolaris_8_Recommended_log.yyyymmdd
• ð marks a horizontal tab (09 hex)
• Reference OE is Solaris 8
• Reference UniData is 6.0

3
Unix File System Maintenance

• Log File Maintenance
• Cleaning the File System (files that do not need
  to be retained)
• File System Maintenance (fsck UFS Logging)

4
Solaris 8 (and Earlier) Automatically Maintained
Logs
5
Solaris 9 System Log Rotation

• Starting with Solaris 9, logadm is used to handle
  log rotation for
• /var/adm/messages
• /var/cron/log
• /var/log/syslog
• /var/lp/logs/lpsched
• /var/lp/logs/requests

6
Some Logs That Need Manual Maintenance

• These files will grow forever
• /var/adm/authlog
• /var/adm/loginlog
• /var/adm/sulog
• /var/adm/wtmpx
• /var/sadm/install_data/Solaris_8_Recommended_log
• var/sadm/patch/patchnumber/log
• /var/spool/lps/adm/activity_log (EasySpooler)

7
Trimming Logs

• Most logs can be cleared by copying /dev/null or
  standard out to the logcp /dev/null gtlogfil (or
  simply gtlogfile)
• Or save the file, then clearmv logfile
  logfile.save.nameÄ gtlogfile
• Or save the latest entries in the log with the
  tail commandtail -50 logfile gtlogfile.tmp Ä
  mv logfile.tmp logfile
• To stop EasySpooler logging, remove the log
  file rm /var/spool/lps/adm/activity_log

8
Patch Cluster Logs Recommendation

• If a Solaris_8_Recommended_log file already
  exists, the next cluster installation will append
  new entries at the end of the existing log. May
  want to save each log individuallymv
  Solaris_8_Recommended_log
• ÄSolaris_8_Recommended_log.yyyymmdd

9
Maintaining wtmpx

• Can be maintained by hand or scriptcp
  /var/adm/wtmpx /var/adm/cis.wtmpx.2004cp
  /dev/null gt/var/adm/wtmpx
• (Or simply gt/var/adm/wtmpx)
• Note /var/adm/utmpx is information about who is
  currently logged into the system. Used by who,
  whodo, w, users, and finger commands. Dont mess
  with this file.

10
Files That May Need Manual Removal

• core files
• preserve files
• backing-store snapshot files

11
Core Files

• Core files are images dumped to a disk of a user
  process terminated by certain signals
• coreadm command will configure or show settings
  (but wont prevent core dumps)
• To prevent core dumps, use Bourne Shell ulimit
  command with c set to zeroulimit c 0

12
Removing Core Files

• As root, find and remove all core filesfind /
  -name core type f exec rm f \(Note Be
  sure to use the type f option patches for
  third-party apps such as Apache sometimes have a
  directory named core.)

13
Crash Dump Files

• A crash dump file is a disk copy of physical
  memory of the computer at the time of a fatal
  system error.
• Usually not a problem unless the machine has
  crashed.
• The dump is usually saved in a swap file
  partition, then on reboot the savecore command
  copies the dump to the savecore directory.

14
Removing Crash Dump Files

• dumpadm will show crash dump settings, including
  the savecore directory dumpadm
• Dump content kernel pages
• Dump device /dev/dsk/c2t0d0s3 (swap)
• Savecore directory /var/crash/cis
• Savecore enabled yes
• Check the savecore directory for a crash dump
  file (vmcore.n), or use a find command with
  name vmcore..

15
Preserve Files

• Should the vi editor (or the system) crash, a
  copy of the users file is stored in
  /var/preserve/username.
• vi r filename allows recovery of workfile.
• Unlikely there are any files there, but may want
  to check and removefind /var/preserve exec rm
  f \

16
Snapshot File Cleanup

• If the fssnap command is used without the unlink
  option, the backing-store file will still exist
  on the system and need to be manually removed
  after issuing the fssnap d filesystem command.
• If there are no active snapshots (check with
  fssnap i), either
• rm /backup-store-path/snapshot
• Or
• find /backing-store-path name snapshot
  Äexec rm \

17
Unix Disk-Based File Systems

• Disk drives consists of slices (partitioned using
  the format command).
• Each slice is either raw or contains a file
  system (constructed by newfs, front-end to mkfs).
• Disk file systems consists of so many blocks
• The default Solaris disk file system is UFS (Unix
  File System)

18
VTOC (Volume Table of Contents)

• The first cylinder of the disk contains a VTOC
  describing the disks slices slice number, tag,
  starting sector, size, last sector.
• Use the prtvtoc command to save the VTOC of a
  disk in case the entire disk fails
• prtvtoc /dev/rdsk/ctd0s2 Ägtctd0s2.vtoc
• Use this file with the fmthard command to
  recreate the VTOC on a replacement disk
• fmthard -s ctd0s2.vtoc Ä/dev/rdsk/ctd0s2

19
UFS File System Structure

• Formatting a UFS file system divides the disk
  slice into cylinder groups.
• Cylinder groups have four basic types blocks
  bootblock, superblock, inode, and storage (or
  data block).
• For further info, see man page fs_ufs (4)
• Structure is documented in /usr/include/sys/fs/ufs
  _fs.h

20
Cylinder Group Layout
Graphic from Sun Microsystems, Inc.
21
UFS Areas and Block Types
22
Boot Block

• Used for booting system, holds bootstrap programs
• Only appears in first cylinder group (blocks 0
  -15)
• Left blank if file system isnt used for booting
• Firmware boot program loads and executes bootblk,
  which then loads and executes /ufsboot.
• Installed by installboot command.

23
Superblock

• Critical data about the file system a copy is
  replicated before each cylinder group
• Sync command forces all file systems superblocks
  to be written to disk
• The shutdown command calls sync.
• Structure is documented in /usr/include/sys/fs/ufs
  _fs.h
• Superblock contains flags about file system
  state, including fs_clean.

24
Superblock fs_clean Flags
25
Some Inode (or I-node) Fields

• File type (regular, directory, link, etc.)
• File mode (read/write/execute permissions)
• Hard link count
• UID of owner, GID of group
• Size (number of bytes)
• Dates times created, last accessed, last
  modified
• Array of 15 disk-block addresses

26
File Addressing
Graphic from Sun Microsystems, Inc.
27
Boot Block Problems

• If The file just loaded does not appear to be
  executable message appears when booting, then
  the hard disk boot block is corrupted.
• Boot Solaris Software 1 of 2 CD and install a new
  boot block on the boot disk
• okgt boot cdrom s
• //
• installboot /usr/platform/uname Ä-
  i/lib/fs/ufs/bootblk /dev/rdsk/ctd0s0

28
Causes of File System Inconsistencies

• Unclean shutdowns
• StopA executed
• System turned off without proper shutdown
• System unplugged or power failure
• Disk with mounted file systems removed
• Software error in kernel
• Hardware
• Failing disk or controller
• Other major component fails

29
Check Repair File Systems fsck

• fsck is only a file-system checker does not
  handle data-integrity by checking contents of
  regular file data blocks.
• Runs automatically at boot, can be run manually
• Never use fsck on a mounted file system! (Unless
  you want to cause a panic and test your crash
  dump settings )

30
fsck Superblock Checks

• File system size versus number of inodes and
  number of blocks used by superblock
• Checks free blocks blocks marked as free should
  not be claimed by any files
• Count of summary block free inodes is compared to
  actually count of free inodes

31
fsck Inode Checks

• Inodes are checked sequentially starting at first
  true inode (2).
• Inodes are checked for
• Format and type
• Link count (directories compared to inode)
• Duplicate block
• Bad block numbers (blocknumber gt first block
  lt last block number)
• Inode size (actual number of blocks compared to
  inode size field)

32
fsck Directory Block Checks

• Inode number in directory points to unallocated
  inode
• Inode number in directory entry points beyond
  inode list
• First entry in directory list must be . entry
  referencing itself
• Second entry must be .. and equal to inode of
  parent directory
• Directory must be linked to somewhere in the file
  system.

33
fsck and fs_clean

• The fsck command uses the fs_clean flag to
  determine whether a file system needs checking.
• To view a file systems fs_clean flagfstyp v
  /dev/rdsk/ctds Ä grep fscleanfilesystem
  state is valid, fsclean is 2
• (be patient fstyp v returns a lot of
  information)

34
fsck at Boot Time

• At boot time the /sbin/rcS script checks /, /usr
  and /var file systems (if /usr /var are
  separate systems).
• When the system boots, fsck runs in preen mode
  inconsistences consist with an unorderly shutdown
  are repaired, file systems are checked
  sequentially using the fsck pass field in
  /etc/vfstab.
• If the system cannot be repaired interactively,
  /sbin/rcS will print a either a warning or fatal
  message that states Run fsck manually.
• After non-fatal errors are repaired, the system
  will continue booting. After fatal errors, the
  system will reboot.

35
/ (Root) and fsck

• If the root file system isnt mounted FSCLEAN or
  FSTABLE, then at boot time / is mount read-only
  and flagged FSBAD.
• To fix, need to boot from alternate device such
  as CD, then use fsck on CD to repair.
• If /usr is hosed, will usually also need to boot
  from CD.

36
UFS Logging

• UFS logging first appeared in Solaris 7
• Previously (2.6 and earlier) file system logging
  required Solaris DiskSuite and a separate logging
  partition.
• UFS logging writes all metadata changes first to
  the logging space, then actual data blocks are
  written. Metadata is the directory and Inode
  information
• In other words, details of all changes to the
  file system are recorded in a log before the
  changes are actually written to the disk.

37
How to Log File Systems

• To log a file system, either mount the system
  with the logging optionmount o logging
  myfilesystem or change the option field in
  /etc/vfstab to logging
• device device mount FS
  fsck mount mount
• to mount to fsck point type
  pass at boot options
• ...
• /dev/md/dsk/d4 /dev/md/rdsk/d4 /vol1 ufs 1
  yes logging
• Using mount without arguments will include
  logging status/vol1 on /dev/md/dsk/d4
  Äread/write/setuid/intr/largefiles/logging/onerro
  r Äpanic/dev1540004 on Sat Oct 29 141421 2005

38
UFS Logging Space

• The logging information is stored in the file
  systems free blocks.
• Log size is 1MB per 1GB of file system space, up
  to 64MB.

39
UFS Logging Advantages

• Reduces risk of file system inconsistencies if
  data blocks arent written, metadata changes are
  rolled back.
• Increases boot speed fsck doesnt waste time on
  system where the fs_clean flag is FSLOG.
• Most file system operations are significantly
  faster

40
Solaris Disk Suite (up to OE 8) andSolaris
Volume Manager (OE 9 and later)

• If using Solaris Disk Suite (SDS) or Solaris
  Volume Manager (SVM), periodically use the
  metastat command to check status of metadevices
  and hot spare pool.
• Automate checking by using shell and awk scripts
  run by cron Check status of disk metadevices
  0 4,16 /opt/local/sbin/dsmon.sh

41
dsmon.sh Shell Script

• !/bin/sh
• _at_()dsmon.sh 1.1 _at_()
• Uses an awk script (dsmon.awk) to parse output
  of DiskSuite
• metastat command in order to report errors.
• DSMON_SCRIPT/opt/local/sbin/dsmon.awk
• DSMON_OUT"/tmp/dsmon..out"
• DSMON_STATUS0
• RECEPIENTS sendmail alias of who to notify
• RECEPIENTSsysadmin.list
• trap "rm -f DSMON_OUT exit 1" 1 2 3 15
• if metastat awk -f DSMON_SCRIPT gtDSMON_OUT
• then
• else
• DSMON_STATUS?

42
dsmon.awk Awk Script

• DiskSuite Status Monitor
• _at_()dsmon.awk 1.1 _at_()
• BEGIN
• STATUS0
• /State /
• if (2 ! "Okay" )
• if (prev /d/)
• print prev, 0
• STATUS9
• prev 0
• END
• exit STATUS