Gato-docs.its.txstate.edu
Texas State University: star.tr.txstate.edu
Introduction
Prologue:
The star system is composed of 36 nodes or hosts, 32 of which are composed of 2 Xeon E5-2450 central processing units (CPU's) and each CPU contains 8 cores. The host node, star.tr.txstate.edu, is dedicated to administration and is the primary node to which all users will login. Ordinarily, a user (star_user, say) is set up by staff and the user logs in with their Texas State netID and password. Their home directory, /home/star_user, is mounted on all nodes simultaneously and is accessible from any node. star is normally accessible by ssh (secure shell) login. The accompanying ssh commands, scp(secure copy) and sftp(secure ftp(file transfer protocol)), are the preferred method of access and datafile transfer from other desktops or laptops.
Note: ssh, sftp and scp are normally installed and available in any Mac or Linux OS(operating system). In Windows, Putty is a free software source and Qvt-Term is a licensed to Texas State faculty, staff and student source of these clients.
The blade compute nodes, locally named compute-1-x where x varies from 01 to 32, are the preferred execution and compilation hosts as the software libraries are primarily stored on them. All of the nodes are running a Red Hat Linux OS with rocks cluster administration and use the Univa Grid Engine for batch queue scheduling. Nodes compute-2-1 and compute-2-2 are the “fat” nodes and compute-3-01 and compute-3-02 are the “gpu” nodes.
When star_user logs in to star.tr.txstate.edu, the user is placed in a shell environment, that is a terminal window into which shell commands are typed. Typing the command
echo $SHELL
followed by pressing the enter/return key will initially return
/bin/bash
indicating the bash login shell. Typing
echo $PATH
and typing enter/return will show the directories which are searched for commands. The echo command is found in /bin/echo. If the PATH environment variable is not set correctly, ordinarily one could use the full pathname /bin/echo for execution. The set or env command will list the environment variables associated with the current shell. Command lines will initially be prefaced with
[star_user@star ~]$
whose format is set by the PS1 environment variable. Echoing $PS1 will show its current setting. These and other settings are controlled by system login files and personalized by files in your home directory. The command
ls -a
will list .bashrc and .bash_profile text files whose contents serve this purpose. You can use cat
cat .bashrc
or less
less .bashrc
to display these files (press the space bar to continue and q to quit when using less). Help can be found in the man pages. For instance,
man ls
or
man echo
or
man less
will display descriptions of the commands using a less-like context. You will ordinarily need to use one of the available text file editors. The command
man -k editor
will list a selection( man man shows you that this is equivalent to the command
apropos editor
and
man apropos
will give the description “apropos searches a set of database files containing short descriptions of system commands for keywords and displays the result on the standard output”). A few more quick notes, the command
apropos editor | less
will pipe (the name of the vertical bar key in unix systems) the textual output of the apropos command into the less command as if that output were a file. The command
apropos editor > editorlist
will create a file called editorlist (assuming you have write privileges in the current directory) from the output of the apropos command which you can peruse with an editor or show with less,
less editorlist
The command
ls -al
will show (the option l implies long, option a implies all files (i.e., including those starting with dots)) the permissions of the current directory, designated by a dot, the parent directory, designated by two dots and the rest of the files. The chmod command is used to control these permissions if you have permission to use it.
Use the command syntax
export ENV_VARIABLE=”text_here”
in a bash or bash-like shell and
setenv ENV_VARIABLE “text_here”
in csh or a csh-like shell. The manual (man command output) pages for some of these commands may be inside the shell man pages. To find an export description for the bash shell, issue the command
man bash
and type
/export
and the page will immediately move to the first occurrence of the string “export”. Typing
/
repeatedly will move the page to following occurrences of the previous search string. You will eventually find the page which contains
export [-fn] [name[=word]] ...
export -p
and descriptions.
Using the Parallel Environment:
The
qstat -g c
command lists the available queues, all.q, bigmem.q, blades.q and gpu.q. The queue, all.q, is the default queue, accesses all nodes and has a time limit of 15 minutes. Nominally you can choose from 1 to 616 cores, but the diverse set of nodes makes using all of them effectively in any particular job relatively rare. Thus the qstat -g c command shows a bigmem queue for accessing compute-2-1 and compute-2-2, each of which has 40 cores and 1TB of memory. The gpu queue access compute-3-01 and compute-3-02, each of which is loaded with the cuda environment and 2 M2090 Nvidia GPU's. If you feel you have an unusual job case please contact us via the contact page. The qsub command is the preferred entrance into the SGE scheduling system. A submit example is shown below.
#!/bin/bash
#
#$ -cwd #current working directory
#$ -N srtest #job name
#$ -j yes #merge stderr with stdout
#$ -S /bin/bash
#$ -q blades.q #shell
#$ -pe orte 12 #parallel environment and number of processes
#$ -M star_user@txstate.edu #mail address for job results
#$ -m as #mailing circumstances
#$ -R yes #reserve (place) in queue
#
#
. /etc/profile.d/modules.sh #setup modules
#
module load openmpi-x86_64 #load openmpi environment
#
#mpicc -O2 c_coded_file.c #source of a.out file
mpirun -np 12 ./a.out #mpi start and run command
These lines will be in a file, call it jobfile.qsub, and submitted with the command
qsub < jobfile.qsub
The
man qsub
command will show the description of the options. The file a.out is the default output of the mpicc compiler command and the -O2 option is a typical gcc optimization choice. You must use the mpicc wrapper instead of gcc in order to most easily access the mpi libraries. This script assumes that the file, c_coded_file.c and its compiled result a.out, exists in the current working working directory which is specified with the dot above. You must be aware that 16 cores exist on each node filesystem. Thus parallel jobs whose cores use separate files must use unique filenames on the same node in the same filetree. Home directories are shared as is among all hosts/nodes.
Available modules are shown by the
module avail
command, not all of which may be active. These modules will undergo changes as we adapt our library and compiler environment. We have the the Intel compilation environment available, as seen in the modules list, to better optimize execution.
Please let us know if you have software and/or environment requests that do not fit in the present framework.
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.