Archer:NFS HOWTO
From Grid-Appliance Wiki
Contents |
Sharing files using NFS
Each Archer VM appliance is capable of transparently exporting and mounting NFS file system(s) from other appliances. This can be done easily by users themselves, without any administrator intervention.
Basics of NFS
Network File System (NFS) is a distributed file system widely used in Unix-based operating systems. NFS allows one or more clients to mount an entire file system exported by a server. Applications on the client side can access file system objects (files, directories, links) as if they were stored in a local disk, when in fact they are brought in transparently from a server. An important feature of NFS is that such transfers are done on a block basis (an NFS block typically has 8KB-32KB) and on-demand. If you use a Unix machine in a local area network, and your data is stored in a shared server, chances are you use NFS.
NFS in Archer: when and why
NFS is useful in Archer for two main reasons:
- It allows users to export large, sparsely-accessed files for efficient access when their jobs run remotely through Condor.
- When you submit jobs through Condor, they need access to your files. Some jobs may require very large files as inputs - for example, a Simics simulation with a checkpointed Linux O/S image may require several GBytes of data. Rather than transferring the entire dataset to all nodes running your job, NFS allows transfer on-demand of only the blocks that are needed. Because applications and benchmarks only "touch" a subset of the checkpointed state, the on-demand transfer of NFS often results in a much more efficient system.
Exporting an NFS file system in Archer
The Archer appliance is pre-packaged with NFS client/server and the system is designed to make it very simple for end users who might never have managed NFS to export and mount file systems. Essentially, all files/directories that you put into the following directory within the appliance:
/mnt/local
are automatically exported (with read-only permissions) to the rest of Archer
Mounting a file system
To mount a file system in Archer, all you need to find out is the hostname of the machine running the network server. The hostname has the form Cxxxyyyzzz (where xxx, yyy and zzz are numbers corresponding to the three least significant bytes of the appliance's virtual IP address).
To access the read-only NFS file system exported by server Cxxxyyyzzz, simply access files within the directory:
/mnt/ganfs/Cxxxyyyzzz
Archer will look up the server name from this path, and NFS-mount this directory automatically for you.
Example
Check the hostname of your appliance with the command:
hostname
Let's assume your hostname is C111222333. Let's create some test files to your NFS export folder:
cd /mnt/local mkdir mydir cp /home/griduser/examples/dht/bget.py . cd mydir cp /home/griduser/examples/dht/bput.py . touch /mnt/local/mydir
Now check that these files are available over NFS. From your own machine, you can check this with the commands:
ls -l /mnt/ganfs/C111222333 ls -l /mnt/ganfs/C111222333/mydir cat /mnt/ganfs/C111222333/bget.py
Using NFS file systems in Condor jobs
It's easy to access your files over NFS in Condor jobs. Just follow these guidelines:
- In your wrapper scripts or Condor submit files, reference the file you want to be accessed by its full path - e.g. /mnt/ganfs/C111222333/bget.py
- Make sure you do *not* list these files in the list of input files Condor should transfer
Here's an example of a Condor submit job configuration that you can use to test your NFS server, assuming you've been through the steps above. It will simply copy a file from the NFS server (bget.py) to another file that is transfered back to Condor. Replace C111222333 by your hostname.
universe = vanilla executable = /bin/cp Log = test.log Error = test.err Output = test.out
arguments = /mnt/ganfs/C111222333/bget.py copied_bget.py should_transfer_files = yes when_to_transfer_output = ON_EXIT transfer_output_files = copied_bget.py queue
Handling datasets larger than the default 32GB virtual disk
The Archer appliance by default comes with a 32GB growable virtual disk where your files can be placed. If you need to export a dataset larger than 32GB, there are two options you can consider.
Make sure you start from a "fresh" Archer appliance, otherwise the following instructions may not work properly.
Create and attach an additional "export" virtual disk
This option is currently only available for VMware users.
- Use VMware's interface to create a blank VMware SCSI virtual disk (say export.vmdk) with enough capacity to hold your files
- Use VMware's interface to attach this new disk as the 4th SCSI disk in your appliance
- Boot up the appliance; the disk will be /dev/sdd in the appliance.
- As root, use fdisk to create a Linux partition for the disk:
fdisk /dev/sdd
- type "n" for a new partition
- select primary partition number 1
- use defaults for first cylinder and number of cylinders
- use "w" to write and exit
- As root, create an ext3 file system for this disk with:
mkfs.ext3 /dev/sdd1
- mount this disk under /mnt/local
mount /dev/sdd1 /mnt/local
- Now /mnt/local points to the new disk you created, instead of the default disk that comes with the appliance
Replace the home.vmdk virtual disk
- Use your VM's interface to create a blank virtual disk with enough capacity to hold your files. Call this disk home.vmdk
- You will need to use another Linux-based VM temporarily (let's call it temporary VM) to prepare a file system in this disk
- Add your home.vmdk disk to your temporary VM
- Boot up the temporary VM
- As root, use fdisk to create a Linux partition for the disk. eplace XXX by the disk device corresponding to home.vmdk - e.g. hdb or sdb; this will vary depending on which VMM you use and which device home.vmdk is attached.
fdisk /dev/XXX
- type "n" for a new partition
- select primary partition number 1
- use defaults for first cylinder and number of cylinders
- use "w" to write and exit
- As root, create an ext3 file system for this disk with:
mkfs.ext3 /dev/XXX
- Shut down the temporary VM
- Replace the home.vmdk file in the Archer appliance with the home.vmdk file you just created
