|
This document is intended as a quick reference on basic
questions about the SAM-QFS file system on the HPCVL
Sun Fire cluster. It includes information on disk quota,
our tape library backup system, scratch space and
archiving schemes.
How is the /home file system structured?
Are there disk quota?
What to do with files that I use infrequently?
What do I do if I need a lot of disk space?
How is scratch space handled on the cluster?
Which files are backed up, which aren't?
I lost data, how can I get them back?
How do the SAM file systems work
and how are they configured on the Sunfire cluster?
How can I find out which of my
files are on disk, and which ones are on tape?
How can I manage my files manually?
Answers:
How is the /home file system structured?
The /home file systems (there is actually two of them, but they are
both called /home) is the main area where users of the Sun Fire
cluster keep their data. Each user's home directory resides there,
and is called /home/hpcXXXX where hpcXXXX denotes the
user name.
Physically, this file system resides in 3 racks containing approximately
160 TB of RAW disk space in the form of 46 Sun StorageTek 3410 FC units.
The building blocks of the file system consist of many RAID 5 logical
volumes later mapped as LUN's to the SAM-QFS host. This configuration
is designed to tolerate the failure of multiple disks without the loss of
data or disruption in service. This is achieved with the implementation of
a "global spare" pool of disks. When a member drive from any of the logical
drive fails, the global spare drive joings the logical drive and automatically
starts to rebuild.
Logically, the file system is a SAM-QFS file system which which
is a feature of the Sun Performance Suite (QFS) and the Sun
Utilization Suite (SAM-FS) storage management packages. The former
provides fast and efficient file access on clusters, while the
latter offers dynamic archiving capabilities in the form of a
so-called "Hierarchical Storage Management" environment that allows
a close integration of our StorEdge L1400 tape library with
the disk file system. This allows the automatic archiving of files,
and the dynamic management of various copies of user files residing
on disk and on tape (see below, question
6).
The present version of the QFS files system is 4.5.
Back to Top...
Are there disk quota?
Unlike it predecessor (3.5), the 4.5 version of QFS supports
disk quota. The increased number of HPCVL users on the system
has made it necessary to introduce quota. At present these
quota have been fixed at 100 GB per user for /home,
which is enough for 9 out of 10 of our users. These quota
pertain to the /home file system only. The /u1 file system
has a quota of 2 TBytes to avoid uncontrolled fill-ups ("sanity
quota"). Data that exceed these limits have to be moved,
preferably off the system (see below question 4).
Note that disk quota are enforced without exception. Once a
user exceeds these quota, no further data can be written to the
file system by that user, making it impossible to log in in
some cases. If this happens, you need to contact us and
arrange for freeing up disk space.
Files in /home and /u1 are automatically managed by the
SAM-QFS file system. This includes backups. Users do not
have to do anything for these activities occur.
Back to Top...
What to do with files that I use infrequently?
Experience shows that users who need large amounts of disk space
use a great portion of their files rather infrequently. An example
would be the trajectory files of a molecular dynamics run, or the
results of large fluid dynamics simulations. Files that contain
information which is only rarely accessed may be moved away from the
/home file system into an alternative area denoted /u1. This
file system offers almost 28 TB of storage and is not subject to disk
quota.
Data residing in /u1 are backed up by default.
/u1 should be used by creating a directory with the user-ID as
directory name, and then restricting access to this directory
to the owner only.
mkdir /u1/work/hpcXXXX
chmod 700 /u1/work/hpcXXXX
Of course, hpcXXXX stands for the user-ID and has to be
substituted. The structure of the files and directories below
/u1/work/hpcXXXX is left to the user.
SAM-QFS offers the possibility of "releasing" files,
i.e. they will still be accessible under /u1/work/hpcXXXX, but
whenever it is accessed it will be retrieved from tape, since
the file does not physically reside on disk anymore (see below
question 6). At present, we do not
release files.
Back to Top...
What do I do if I need a lot of disk space?
If you need more disk space than the disk quota on /home allows,
you should consider the following options, preferably in that
order:
- The easiest way to keep data safe and with fast access is to
keep them in /home. Our disk quota are comparably high, and the
presence of more than 100 GB of data should lead a user to
consider "cleaning up" the /home area. We strongly
encourage all our users to download their permanent data and back
them up on external media such as multiple drives, CD-Rom, Tapes,
or DVD. This is the safest way to make sure your data cannot
be lost. In many cases the only reason why disk space usage
approaches the limit is that unprocessed data are kept around
unnecessarily. We do not supply individual backups or
archiving facilities for longterm storage. This is the
responsibility of the user.
- If the data are only temporary (eg, they are
results that serve as input for other computations, but can
then be discarded), they might be written out on scratch
space. See question 5 below for
further details about scratch space. Data on scratch space
will not be archived or backed up, and and should be removed
as soon as they are not needed anymore. We have at this
point approximately 12 TB of scratch space available on the
system. This is only a solution if the data will be removed
soon (within days) after being generated.
- If the data are needed for a longer period of
time, but can eventually be deleted, they can be brought to
the afore-mentioned /u1 file system (see previous question). Data of this kind
will be backed up for the (rather unlikely) event of a
multiple disk failure. We recommend to consider this
solution for data that need to be available for several
weeks, but cannot be kept in the /home area to avoid
exceeding the quota. Note that a system-wide size limit for
backing up a single file is 25 GByte. Files larger than that
will not be backed up.
- Finally, if you need to keep large amounts of data
that are accessed rather frequently, exceed the disk quota,
and need to be backed up, you need to contact us
and make special arrangements. Such arrangement might imply
Category 2 membership in HPCVL, or even necessitate
re-negotiating the membership conditions.
Back to Top...
How is scratch space handled on the cluster?
Scratch space is supplied in the /scratch area
of the file system. This space is intended for transitory
data that are generated during a calculation and are usually
deleted shortly after the calculation has finished. However,
it is worthwhile to consider keeping intermediate results
that are only needed for a few days on scratch space if
there is a danger of exceeding disk quota in /home.
Note that our scratch space is now global,
i.e. accessible from all nodes. While this implies somewhat
slower access than local scratch, it allows data to be used
from different nodes within a program run (e.g. of an MPI
program), and it simplifies maintanance.
Scratch space is accessed via the
/scratch directory. To use scratch space, do the following:
- Create a directory with your user-ID as a name under
/scratch on all nodes:
mkdir /scratch/hpcXXXX
- Change the file permissions on the directory to
allow access only to you:
chmod 700 /scratch/hpcXXXX
Here, XXXX stands for the numbers in your user-ID. To use
the scratch, you will often have to set an application
specific environment variable, which can then be given the
name /scratch/hpcXXXX to work on all nodes, eg. for
the quantum-chemistry code Gaussian, one would set
(in a csh): setenv GAUSS_SCRDIR /scratch/hpcXXXX
Note that the above settings are automatically applied when
you are given a user account.
Back to Top...
Which files are backed up, which aren't?
HPCVL maintains backups and short-medium-term archives for the
purpose of securing user data, not for permanent storage
or external use. This means that it is the responsibility of the
individual user to remove data that are to be kept permanently
from the cluster and store them on external media, such as disks,
tapes, CD's or DVD's.
User data that reside in the /home file system are backed
up on a 15 minute cycle for the first copy, and on a 60 day
cycle for the second, i.e. keep two backup copies for the
case of disk failure. Backup data kept on our L1400 tape
library. Since /home is using a SAM-QFS file system,
this archival happens automatically as soon as a file
appears on the files system. Whenever a file changes, the
change will be comitted to the archive as well.
Data that reside in /u1 are also backed up. The
scanning period for the first copy there is 1 hour, while
the one for the second copy is 60 days, just as for the
/home file system. No data residing either in
/home or /u1 are released at this point.
All data outside these two areas are not backed
up. This holds specifically for /tmp and /scratch.
Back to Top...
I lost data, how can I get them back?
The general answer is contact
us. The system administrator will be able to retrieve
the lost data from the regular backup on the L1400 tape
library. Keep in mind that changes that you made to the data
before the loss occured might be lost since the copy of your
file might be outdated. If the loss is the consequence of a
general disk failure, the part of the file system that was
affected will be restored from safety backups, and it is not
necessary (nor useful) to contact the administrator for the
retrieval of individual files. In that case, you will have
to wait until the file system is restored to normal. This
may take several days in the case of a severe failure.
Back to Top...
How do the SAM file systems work and how are they configured on the Sunfire
cluster?
The SAM file system was designed to optimize the utilization
of limited disk space and at the same time provide a dynamic way
to do archives and backups. To this end they provide an interface
between the files that reside directly on disk, and a tape
library such as our L1400.
The files system will automatically
create copies of files that are freshly written to disk or that
are altered. This is to safeguard from disk failures and serves
as a standard backup. Additionally, the files system offers the
possibility of making further copies on tape, and to keep files
that have not been used for an extended period of time on tape
altogether.
The latter process is called releasing
files. Data that have not been accessed for a certain time period
may be copied to tape (preferably, more than one copy
is made for safety), and then the copy on disk is replaced by a
stub, which serves a similar function as inodes in Unix
file systems, i.e. it contains information about where the data
reside, their size, the date they have been accessed last,
etc. However, the data themselves do not reside on disk anymore,
and if they are to be accessed they will be retrieved
automatically from tape. Often, the user will prefer to retrieve
them manually before attempting to access them.
Retrieving released data back to disk is called
staging, and is either done automatically when the
data are accessed, or preferably, by issuing a command
beforehand. The staging process can take between minutes for
a few small files that are on the same tape, when the
archiving system is not busy, to several hours, when large
data spread over several tapes are required at a time when
several requests for tape operations are queued.
The HPCVL Sunfire cluster has at this time two SAM file
systems: /home (more accurately, /home1 and /home2), and
/u1. Neither of these two systems is currently
releasing any files. Here are the backup cycles for these areas:
- /home: All files residing in the /home area, ie,
in the user home directories, are handled by SAM. A first
copy of the files is made every approximately 15
minutes after the file is created or altered. A second
copy is produced every 60 days. No releasing
takes place on /home, i.e. the data are always kept on
disk. Note that /home is subject to disk quota of 100
GByte per user to make this policy possible.
- /u1: Additional disk space is provided in
the /u1 area. Note that data need to be placed in a specific
directory: /u1/work/hpcXXXX. Data in
/u1/work/hpcXXXX are handled as follows: The first copy is
backed up on a one-hour cycle, whereas the second copy is on
a 60-days cycle. No data are released at
present. Note that a 2 TByte "sanity quota" is in effect on
/u1.
Note that the configuration of the SAM file systems
might change in the future. Check this FAQ to keep up-to-date.
Back to Top...
How can I find out
which of my files are on disk, and which ones are on tape?
None of the files on our file system is currently subject
to releasing, i.e. all file physically reside on
disk.
Back to Top...
How can I manage my files manually?
Many file operations such as releasing and staging
are done by the SAM file system automatically, and
have been configured by the system
administrator. Certain basic settings (such as
releasing files within some file systems) cannot be
overridden by users, to maintain the integrity of the
file system.
There are commands that let the user control their
own files to some degree. However, at present, these
commands cannot be accessed from the login zone. This
may change in the future. Please check this FAQ file
occassionally for updates.
If you are unsure about the state of your files, contact us
.
Back to Top...
|
