Htar

1
Htar
Hpss Tape Archiver Client API-based interface
written by Mike Gleicher Originally
commissioned for LLNL in 2000 Now
available as part of the HPSS Offering A File
Bundling and manipulation tool that conforms to
standard tar requirements
Bundles small files on a platform into large
files in storage. The Htar is
created in the storage system, each
file is transferred directly into the Htar in
storage.
2
Why use Htar
Powerful command set that makes it easy to
recursively archive files to storage
whole directories, files in lists, etc.with
one command. A users directory structure is
maintained in the htar file. Multiple files
transfer faster with htar HTAR cvf
htar.tar ./mydir is optimized for simultaneous
transfer HSI put r
./mydir is recursive.. But transfers files one
at a time .
3
Htar_at_NERSC
Authentication Htar uses the same .hsipw file
as HSI for authentication to NERSC
storage systems. If you currently have a .hsipw
file Htar will use it. If you dont
you will be prompted for your dce
combo, and if its successful a .hsipw will be
created in HOME. Availability
Installed on Seaborg /usr/common/mss/bin and
on the PDSF /usr/local/bin
4
Htar Definitions
Archive File - The large tar file that Htar
creates out of files Member File - one of the
files contained in the Htar Archive File. Index
File - a catalog of the Member Files contained in
the Archive
File Consistency File - the file at the end of
an Archive File used to
verify the consistency of the Archive File and
the Index File.
5
Instorage
Htar -cvf myhtar.tar creates the following in
storage the archive file called
myhtar.tar the index file called
myhtar.tar.idx a consistency file
inside the archive file (looks
like a member file but is treated differently)
called /tmp/HTAR_CF_CHK.. This
consistency file can be used to verify the
consistency of the index file and the archive
file. It will not be extracted unless explicitly
specified using the Htar command. (it would be
extracted using the tar command)
6
Htar behavior
Absolute path names pathnames beginning
with / can be used when creating the
htarwhen files are extracted from the htar the
leading / is removed. All extracted
files use pathnames that are relative to
the current working directory. Htar makes use
of the TMPDIR environment variable when creating
temporary files. If TMPDIR is not set in
the environment then /tmp is
used. There is no default device for Htar the
-f flag must be used.
7
Htar and tar
Tar Similarities Htar conforms to standard tar
requirements. An Htar Archive File can be
manipulated using the tar utility. File and
directory name lengths and file sizes must
conform to tar restrictions (POSIX 1003.1
USTAR). Buffer path to 155 chars, name
buffer to 100 bytes. Member file size lt
8GBs. Tar Differences Htar Cant append a new
Member File to an existing Archive File.
Htar Cant update/replace Member Files in
an existing Archive File. Htar creates an
index .idx file in the same dir as the archive
file. and a consistency check file inside
the Htar archive file
8
Htar examples
Htar -cvf files.tar file1 file2 writes
file1 and file2 into a new archive called
files.tar in the current hpss home
directory. Htar -cvf /nersc/user/homedir/htar.tar
./dir1 writes all the files in the
current dir1 directory into a new archive
file called htar.tar in the hpss directory
/nersc/user/homedir Htar -tvf htar.tar
list files in the archive file htar.tar in the
current hpss homedir.
9
Htar more examples
Htar -xvf files.tar extract all the files
from the archive file files.tar into the
current directory. Htar -xf proj1.tar
project1/src extract all files in
project1/src directory of the archive file
proj1.tar into the current directory Htar -xf
file1 proj1.tar extract the individual
file file1 from the archive file proj1.tar
10
Htar extensions
Htar -X -f /nersc/user/hpsshomedir/tarfile
creates an index for a tarfile in hpss. This
file can now be accessed using
htar. Htar -X -E -f localtarfile creates an
htar index for a local tar file. Htar
-K -f htartest.htar use the consistency file
to verify the index and the archive file.
11
Recommended Usage
Htar should replace all current tar transfers
piped to storage. Htar should be used with data
that is not changing. Htar makes two
passes thru the data first to generate the index
file, second pass to transfer the data.
If files change before they are transferred
and after the index is created a warning is
issued and the file is truncated or
padded to match the index. Htar should be used
to bundle small files Htar should be used to
archive files that are not frequently
referenced (purely archival)
12
Usage for Performance
Htar should be used with collections of data
that will reaccessed as a collection.
These Htar files should be lt 6GBs. Read access
to files in storage 1 second 86400
accesses/day at XXMBs/sec 30 seconds 2880
accesses/day at XXMBs/sec 60 seconds 1440
/etc The Htar will
travel in the archive as one file to and from
disk and tape. Access to any file in the
Htar will stream the Htar to disk
improving all subsequent accesses to the data
to 1 second or less.