Using the High Performance Storage System (HPSS)
The HPSS is a large multi-terabyte storage system consisting of
2 Terabytes of hard drives in the front and 500 Terabytes of
tape storage in the back end. Actually, there may be more storage
at the backend by the time you read this.
You should use the HPSS for storing large data that you
don't access often.
HPSS is provided by UITS,
and their storage group is responsible for
the management and handling of it. HPSS data is not backed up
(after all, it's already on archival storage) but UITS duplicates
it at IUPUI and IUB, providing a form of backup in case Bloomington
disappears from the face of the earth.
The following presumes you will use HSI, the direct interface to
HPSS. Two of the other methods, ftp and pftp, involve sending your
password in the clear over the network and are not recommended.
Another mechanism is to use
DFS (distributed file system) software, by which data is moved to/from the
HPSS tape storage system automatically. If you use DFS, beware the
difference between "mirrored" and "archived" filesets; the latter
does not allow you to access your files except through DFS.
In any case, I recommend using HSI. It requires just a single
client to be installed. If you want strong encryption of passwords,
you can also install kerberos - well, actually, you have to have
UITS install it. They do not trust users to handle that. A kerberos
client requires a fixed IP address and other restrictions which make
it unsuitable for using from a laptop or behind a firewall.
HSI is installed already on all of the CS department solaris and
Process for getting and using an HPSS account:
If you want to use other mechanisms (pftp, CFS) then go to the
starter kit for MDSS.
Two significant notes about HPSS:
- Request a Mass Data Storage account. Instructions for this are at
- Get your DCE credentials started. It may be that UITS has already
mapped your netpass password to the DCE cell, but you can check/change
the password at
- Download the necessary client from http://storage.iu.edu/hsi.html
The RedHat 7 client will still work under RedHat 8. It does not work
under RedHat 9. The client is called "hsi"
- Execute the "hsi" command and you should be prompted for a
"DCE Principal", which is just your username. Then you'll be
prompted for your password. This
is weakly encrypted and sent to the HPSS front end. At that time,
you can do a pwd and get something like
where the /k/a/ will always be from the first two letters of your
login name, and of course your login is kaddiddlehopper
- Now you are ready to use the HSI commands detailed at
Those are similar to ftp but with more flexibility. Common ones:
put, delete, ls, get. This is much like a shell, so you can move
files, rename them, use wildcards, etc.
- Since it is a tape storage device HPSS is better suited for small
numbers of large files than for large numbers of small files.
So tar your directories before shipping them over.
According to UITS, "optimal applications will store data in files
that are typically larger than 50MB".
- The metadata (file names, directory structure) is immediately
visible when you login to HSI, because it is stored on hard drive.
The actual data is written to tape, so it can take up to 2 minutes
to access a file. You will get a prompt that indicates the scheduler
has scheduled the retrieval, then that it is retrieving the files,
and finally a progress bar when "get" a file that has been on HPSS
for some time.