View source

== Condor tips ==
=== Configuration needed to set up a Condor job - example cache simulation ===
You need the following files to run a Simics batch job using Condor in the Grid Appliance:

* A wrapper script. This script is the first thing that is executed when a job submitted through Condor finds a destination host to run. There are many ways to create such a wrapper script, and this is dependent upon the application you are running. The following is a template using Bash that works with Simics (simics_wrapper.sh):

 #!/bin/sh
 
 mkdir new-workspace   
 cd new-workspace
 tgt_wrk_spc=`pwd`                 # create a workspace directory and store path in tgt_wrk_spc
 cd /opt/virtutech/simics-3.0.31/bin
 ./workspace-setup $tgt_wrk_spc    # setup workspace
 cd $tgt_wrk_spc
 cp ../setup-gcache.simics .       # copy Simics configuration files into workspace
 cp ../batch_script.simics .
                                   # Run Simics; -no-win and -batch-mode are required
                                   # for batch execution, and -stall for cache simulation
                                   # Follow commands from batch_script.simics and write standard out to screen_dump.out
 ./simics /mnt/ganfs/C001001250/homework2/xen-dom0-domu -no-win -batch-mode -stall batch_script.simics > ../screen_dump.out

* Simics configuration file. This is provided to Simics to drive what commands and simulation models Simics should execute on the remote machine. Here is a sequence of commands for the simulation of Nbench for 2 billion cycles using the cache model setup by the setup-gcache.simics file, and print out cache statistics at the end (batch_script.simics):

 run-command-file setup-gcache.simics
 con0.input "cd /root/nbench-byte-2.2.3; ./nbench -cCOM.DAT\n"
 c 2_000_000_000
 cpu0_l1_uc.statistics
 quit

* A condor submit file. This file tells Condor which files to transfer before and after running a job, which binary to execute, and requirements for the machines that will run the job. Here is a template Condor submit file for this exercise (condor_script):

 # specify the executable and job type (Simics is always vanilla in Condor)
 universe = vanilla
 executable = simics_wrapper.sh
 
 # specify requirements for resource running job. In this case, resource must
 # have memory > 512MB and Simics installed
 requirements = HasArcherSimics == TRUE && Memory >= 512
 
 # Provide names for the output files. $(Cluster) and $(Process) are substituted
 # by unique Condor IDs
 Log = simics.$(Cluster).$(Process).log
 Error = simics.$(Cluster).$(Process).err
 Output = simics.$(Cluster).$(Process).out
 
 # Tell Condor which files to transfer: send the Simics input files, retrieve output
 should_transfer_files = yes
 when_to_transfer_output = ON_EXIT
 transfer_input_files = setup-gcache.simics, batch_script.simics
 transfer_output_files = screen_dump.out
 
 # Submit job
 queue

=== Submitting a sample job ===
* In the appliance, create a folder named "condor1" and untar /mnt/ganfs/C001001250/homework3/hw3_cache.tgz into this folder

 mkdir condor1
 cd condor1
 tar -xzf /mnt/ganfs/C001001250/homework3/hw3_cache.tgz

* To submit the job:

 condor_submit condor_script

* To track the progress of the job:

 condor_q

* It will take around 10-20 minutes for this job to finish (if there are resources immediately available). When the job finishes executing, you should see the screen_dump.out file with the outputs of this simulation.

=== Submitting multiple jobs ===
The instructions above show how to run a single job. In several realistic scenarios, however, you need to run many simulations. This is where Condor is particularly useful, as it ensures that your jobs are queued for execution if there are no resources available, deal with failures by retrying, among other features. But one needs to be careful when preparing multiple jobs to avoid files being overwritten and to keep appropriate bookkeeping.

Given the way the Condor and Simics configuration files are setup, one approach here is to create multiple directories, one per configuration, and submit your jobs from these subdirectories. For example, to set up two different configurations where nbench runs on dom0 and domU:

 mkdir nbench-dom0 nbench-domU
 cp /mnt/ganfs/C001001250/homework3/hw3_cache.tgz nbench-dom0
 cp /mnt/ganfs/C001001250/homework3/hw3_cache.tgz nbench-domU
 cd nbench-dom0
 tar -xzf hw3_cache.tgz
 cd ../nbench-domU
 tar -xzf hw3_cache.tgz 
 # edit nbench-dom0/batch_script.simics to set up con0.input appropriately
 # edit nbench-domU/batch_script.simics to set up con0.input appropriately
 cd ../nbench-dom0
 condor_submit condor_script
 cd ../nbench-domU
 condor_submit condor_script

The output files will be stored in the corresponding subdirectories at the end of the job execution.

=== Changing the cache size ===
The architecture of the cache that is being simulated can be changed by modifying the setup-gcache.simics file. This file is shown below, annotated with explanation of the different parameters

 #########################################################
 # To set up 1 level of unified cache
 
 @from configuration import *
 
 $tsc = cpu0->ia32_time_stamp_counter
 @SIM_set_configuration([
 OBJECT("cpu0_l1_uc", "g-cache",
        cpus = OBJ("cpu0"),                - This specfies the cpu to which this cache is connected
        config_line_number = 512,          - The number of entries in the cache
        config_line_size = 64,             - The size of each cache line in bits
        config_assoc = 2,                  - Associativity of the cache
        config_virtual_index = 0,          - Specifies whether the cache is indexed and using virtual address or physical address.
        config_virtual_tag = 0,            - Specifies whether the cache is indexed using virtual address or physical address.
        config_replacement_policy = "lru", - Cache replacement policy
        penalty_read = 0,                  - Read/write latencies
        penalty_write = 0,
        penalty_read_next = 0,             - Latencies for read/write communication with the next unit in the memory hierarchy
        penalty_write_next = 0)])
 
 # plug the hierarchy
 @conf.cpu0_mem.timing_model = conf.cpu0_l1_uc
 
 # Send instruction fetches to the cache - NOTE MUST START IN STALL MODE
 cpu0.ifm "instruction-fetch-trace"
 cpu0->ia32_time_stamp_counter = $tsc
 #########################################################

By changing the line number, the size of the cache can be varied.

=== TLB simulations ===
Simulations for different TLB configurations require additional files to be sent along with the job. The following files provide templates for the execution of TLB simulations of 20 billion instructions for openAFS, with TLBs configured with 64 and 256 entries, respectively:

 mkdir openafs-64
 cd openafs-64
 tar -xzf /mnt/ganfs/C001001250/homework3/hw3_tlb64.tgz
 # edit batch_script.simics to set up con0.input appropriately
 condor_submit condor_script
 cd ..
 mkdir openafs-256
 cd openafs-256
 tar -xzf /mnt/ganfs/C001001250/homework3/hw3_tlb256.tgz
 # edit batch_script.simics to set up con0.input appropriately
 condor_submit condor_script

=== Summary: con0 input strings for running benchmarks ===
==== nBench ====
* In dom0:
 con0.input "cd /root/nbench-byte-2.2.3; ./nbench -cCOM.DAT\n"
* In domU:
 con0.input "ssh 10.10.0.14 'cd /root/nbench-byte-2.2.3; ./nbench -cCOM.DAT'\n"
* In both dom0 and domU:
 con0.input "cd /root/nbench-byte-2.2.3; ./nbench -cCOM.DAT &\n"
 con0.input "ssh 10.10.0.14 'cd /root/nbench-byte-2.2.3; ./nbench -cCOM.DAT'\n"

==== OpenAFS make ====
* In dom0:
 con0.input "cd /root/openafs-1.4.7; make\n"
* In domU:
 con0.input "ssh 10.10.0.14 'cd /root/openafs-1.4.7; make'\n"
* In both dom0 and domU:
 con0.input "cd /root/openafs-1.4.7; make &\n"
 con0.input "ssh 10.10.0.14 'cd /root/openafs-1.4.7; make'\n"