1. Subject



Helix - HPC/SLURM Tutorial Cont..28/09/2017Prepared for the Centre of System GenomicsCreated by Bobbie ShabanCentre for System GenomicsThis document outlines the material for the tutorial. This tutorial will also be recorded and uploaded to the Genomic Databases Resource Hub (COGENT) as a webinar for download: This tutorial assumes that you have an account. If you don’t have an account please contact your group leader to give you access.1. SubjectAnswer questions from previous tutorialHPC Cluster (Helix) Cont..Screen and TmuxSinteractive with tmuxNohupScreen and Tmux references2. GlossaryTmux: tmux allows you to run multiple terminal command lines at the same time side by side much like putting multiple windows next to each other so you see both simultaneously. It stands for "terminal multiplexer", a fancy term for something so logical. Here are the basics to get you started with tmux quickly.Screen: The screen program allows you to use multiple windows in Unix.Shebang: Line which calls the interpreter for the script to be executed with. 3. Questions from previous tutorial.1) Set up demo/guest account with melbourne bioinformaticsAnswer: Ask for access from Group Leader is first option for internal users. I’m in contact with Melbourne bioinformatics regarding guest accounts.2) which node to log on interactively?Answer: Run sinteractive and let the scheduler choose which node to log you on. You can reattach using screen or tmux which I will go into this tutorial.3) Examples of logging into sinteractive.Answer: I had some troubles last time but I have figured out what is required.Examples: sinteractive --ntasks=1 --mem-per-cpu=10000 --time=10:00:00Sinteractive –ntasks=1 (1 core)--mem-per-cpu=10000 (~10gb of RAM per cpu) *note a cpu may be dual core.--time=10:00:00 (Session will last 10 hours ie, hours:minutes:seconds)3) can you use output with s interactive and give example of job finishing info?Answer: Yes.Via srun: srun --ntasks=1 --mem=4096 --time=0-1:0:00 --output=output.txt --account=SG0009 cat –helpVia sbatch: [bshaban@snowy-sg1 ~]$ cat test.sbatch#!/bin/bash# Created by the Melbourne Bioinformatics job script generator for SLURM# Tue Sep 26 2017 10:51:03 GMT+1000 (AUS Eastern Standard Time)# Partition for the job:#SBATCH -p main# The name of the job:#SBATCH --job-name="testBobbie"# The project ID which this job should run under:#SBATCH --account="SG0009"# Maximum number of tasks/CPU cores used by the job:#SBATCH --ntasks=1#SBATCH --cpus-per-task=1# The amount of memory in megabytes per process in the job:#SBATCH --mem=4096# Use this email address:#SBATCH --mail-user=bshaban@unimelb.edu.au# The maximum running time of the job in days-hours:mins:sec#SBATCH --time=0-1:0:00#test output file#SBATCH --output=output.sbatch.txt# check that the script is launched with sbatchif [ "x$SLURM_JOB_ID" == "x" ]; then echo "You need to submit your job to the queuing system with sbatch" exit 1fi# Run the job from the directory where it was launched (default)# The job command(s):cat –help3. HPC continued. Extra compute1) log into helix: ssh helix.vlsci.unimelb.edu.au2) type in sacctmgr list cluster3) log into barcoo: ssh barcoo.vlsci.unimelb.edu.au4) Check node info: sinfo --format "%n %20E %12U %19H %6t %a %b %C %m %e"What does this tell us?Do we have permissions to use barcoo and merri?3. Screen and tmuxScreen taken from () Best to run screen on head node and sinteractive from within screen if you so choose.If your local computer crashes or you lose the connection, the processes or login sessions you establish through?screen?don't go away. You can resume your?screen?sessions with the following command: screen -rIn some cases you may have to manually "detach" your?screen?session before resuming it. For more information, see the Knowledge Base document?Using screen, why can't I re-attach to my session after a lost connection?The?screen?program creates multiple processes instead of multiple Unix login sessions, which means that it is resource-efficient.You can cut and paste between different screens without using a mouse. Thus, you don't need to be on a computer with a windowing environment such as?Mac OS,?Mac OS X, Windows, or the?X Window?System.It has a block copy feature which is similar to the kill rectangle feature of?Emacs.You can copy and paste more than one page at a time, which you cannot do with some clients. You can scroll up more than one page, depending on how many scrolling lines you have set with the?-h?option.Using the detach feature, you can save?screen?processes when logging out and resume where you left off, saving the trouble of restarting them.Starting screenTo start?screen, enter the following command: screenGeneral commandsNote:?Every?screen?command begins with?Ctrl-a.Ctrl-a?cCreate new window (shell)Ctrl-a?kKill the current windowCtrl-a?wList all windows (the current window is marked with "*")Ctrl-a?0-9Go to a window numbered 0-9Ctrl-a?nGo to the next windowCtrl-a?nToggle between the current and previous windowCtrl-a?[Start copy modeCtrl-a?]Paste copied textCtrl-a??Help (display a list of commands)Ctrl-a?Ctrl-\Quit?screenCtrl-a?D?(Shift-d)Power detach and logoutCtrl-a?dDetach but keep shell window openPress the Spacebar or?Enter?to end a command.To copy a blockTo get into copy mode, press?Ctrl-a?[?.To move the cursor, press the?h,?j,?k, and?l?(the letter l) keys. The?0?(the number 0) or?^?(the caret) moves to the start of the line and?$?(the dollar sign) moves to the end of the line.?Ctrl-b?scrolls the cursor back one page and?Ctrl-f?scrolls forward one page. To set the left and right margins of copy, press?c?and?C?(Shift-c). The Spacebar starts selecting the text and ends selecting the text. To abort copy mode, press?Ctrl-g.To paste a blockTo paste the copied text to the current window (as many times as you want), press?Ctrl-a?].Other commandsTo run a program or execute any Unix command in a new window, at the Unix prompt, enter: screen unixcommandAbove, replace?unixcommand?with the appropriate command name.To automatically start several windows when you run?screen, create a?.screenrc?file in your home directory and put?screen?commands in it.To quit?screen?(kill all windows in the current session), press?Ctrl-a?Ctrl-\.The?man?pages for?screen?are quite readable and make a good tutorial. At the Unix prompt, enter: man screenExample: (go through further) more detailed[bshaban@barcoo ~]$ screen –hScreen (to create new terminal)Type test 1Press Ctrl-a dScreen (to create new terminal)Type test 2Press Ctrl-a dType screen -lsType screen –r <screen id>Press Ctrl-a n (switch between screens)Close terminal and open againOpen terminal and loginScreen –list (View screens)Screen –r <screen id> (to log in)Ctrl-a Shift-S (Split screen)Tmux on next page:TMUX ()WHAT ABOUT SCREEN?tmux?is a lot like screen, only better. The short answer for?how?it’s better is that tmux is better designed to perform the same functions. Screen gets you there (kind of) but does so precariously.Here are a few of the key advantages of tmux over screen:Tmux is built to be truly client/server; screen emulates this behaviorTmux supports both emacs and vim shortcutsTmux supports auto-renaming windowsTmux is highly scriptableWindow splitting is more advanced in tmuxEnough about that. Use tmux.BASICSNow is a good time to mention that there is a universal tmux shortcut that lets you quickly perform many tasks.THE?TMUX?SHORTCUTBy default, tmux uses?Ctrl-b?as its shortcut activation chord, which enables you perform a number of functions quickly. Here are a few of the basics:First you hit:$?Ctrl-bINVOCATIONRight then. Let’s start by running?tmux. You want to do this from the system that you want to detach and re-attach to—which for me is usually a remote server.$?tmuxSimple enough. You now have a tmux session open that you can disconnect from and come back to later.SHOW SESSIONSSince the idea of tmux is having multiple sessions open, and being able to disconnect and reconnect to them as desired, we need to be able to see them quickly.# Via shortcut (by default?Ctrl-b)$?Ctrl-b?s# Via tmux command$?tmux?lsEither way you get the same thing:0: 1 windows (created Thu Nov 28 06:12:52 2013) [80x24] (attached)DETACHING FROM A SESSIONYou can detach from an existing session (so you can come back to it later) by sending the?detach?command.$?tmux?detachOr you can use the shortcut.$?Ctrl-b?dCREATE A NEW SESSIONNow we’re going to create a new session. You can do this with just the?newcommand, or by providing an argument to it that serves as the session name. I recommend providing a session name, since organization is rather the point of tmux.$?tmux?new?-s?session-name# Without naming the new session (not recommended)$?tmux?new**detach screen** If you try to start a new session within a session you will get this errorsessions should be nested with care. unset $TMUX to force.ATTACHING TO AN EXISTING SESSIONSince we’re going to be creating sessions with names, and we may have more than one, we want to be able to attach to them properly. There are a couple ways of doing this.You can simply type?tmux a?and it’ll connect you to the first available session.$?tmux?aOr you can attach to a specific session by providing an argument.$?tmux?a?-t?session-nameKILLING A SESSIONThere are times when you’ll want to destroy a session. This can be done using the following syntax, which is much the same as attachment:$?tmux?kill-session?-t?session-name[?NOTE: You can kill windows the same way, but using?kill-window?instead. You can also kill tmux altogether with?killall tmux. ]CONFIGURATIONAs with most things in tech, you can get pretty silly with your tmux config. The common things to tinker with are:The primary?tmux?shortcutYour status barYour various keyboard shortcutsI went pretty Spartan with mine.# Set a Ctrl-b shortcut for reloading your tmux configbind r source-file ~/.tmux.conf# Rename your terminalsset -g set-titles onset -g set-titles-string '#(whoami)::#h::#(curl plain;echo)'# Status bar customizationset -g status-utf8 onset -g status-bg blackset -g status-fg whiteset -g status-interval 5set -g status-left-length 90set -g status-right-length 60set -g status-justify leftset -g status-right '#[fg=Cyan]#S #[fg=white]%a %d %b %R' One thing worth noting here is that I use to get my current WAN IP4 WAN address instead of icanhazip as most other tutorials have. It’s just faster and less prone to error, from my experience.[ My current, updated configuration can be found?here?if you’re interested. ]ADVANCEDThat covers how I usually use?tmux, but I do often make use of some of the more powerful features.WINDOWS AND PANESOne of these features is the ability to break your session into more discreet components, called windows and panes. These are good for organizing multiple varied activities in a logical way.Let’s look at how they relate to each other.Nestingtmux?sessions have windows, and windows have panes. Below you can see how how I conceptualize them—although if anyone has a more authoritative or useful hierarchy I’ll happily embrace it.Sessions?are for an overall?theme, such as?work, or?experimentation, or?sysadmin.Windows?are for?projects?within that theme. So perhaps within your?experimentation?session you have a window titled?noderestapi, and one titled?lua sample.Panes?are for?views?within your current project. So within your?sysadmin?session, which has a?logs?window, you may have a few panes for access logs, error logs, and system logs.It’s also possible to create panes within a session without first creating a separate window. I do this sometimes. i.e. ctrl-b %4. sinteractive with tmux1) Enter tmux2) Type in sinteractive command: sinteractive –ntasks=1 –mem-per-cpu=1000 –time=01:00:003) wait for resources4) detach tmux5) close terminal/log out6) open terminal log back into helix7) type tmux ls8) log into session5. nohupWhen you execute a Unix job in the background ( using &, bg command), and logout from the session, your process will get killed. You can avoid this using several methods — executing the job with nohup, or making it as batch job using at, batch or cron command.In this quick tip, let us review how to make your process running even after you logout, using nohup.Nohup stands for no hang up, which can be executed as shown below.nohup syntax:# nohup command-with-options &Nohup is very helpful when you have to execute a shell-script or command that take a long time to finish. In that case, you don’t want to be connected to the shell and waiting for the command to complete. Instead, execute it with nohup, exit the shell and continue with your other work.Explanation about nohup.out fileBy default, the standard output will be redirected to nohup.out file in the current directory. And the standard error will be redirected to stdout, thus it will also go to nohup.out. So, your nohup.out will contain both standard output and error messages from the script that you’ve executed using nohup command.Instead of using nohup.out, you can also redirect the output to a file using the normal shell redirections.Example: Printing a counter#!/bin/bash COUNTER=0 while [ $COUNTER -lt 10 ]; do echo The counter is $COUNTER sleep 1; let COUNTER=COUNTER+1 doneExecute the script without redirection$ nohup sh custom-script.sh &[1] 12034$ nohup: ignoring input and appending output to `nohup.out'$ tail -f nohup.outThe counter is 0The counter is 1The counter is 2The counter is 3The counter is 4The counter is 5If you log-out of the shell and login again, you’ll still see the custom-script.sh running in the background.$ ps aux | grep bshaban bshaban 12034 0.0 0.1 4912 1080 pts/2 S 14:10 0:00 sh custom-script.sh6. screen and tmux referencesSHORTCUT REFERENCE (TMUX)Now a?Ctrl-b?options reference:Basics??get helpSession managements?list sessions$?rename the current sessiond?detach from the current sessionWindowsc?create a new window,?rename the current windoww?list windows%?split horizontally"?split verticallyn?change to the next windowp?change to the previous window0 to 9?select windows 0 through 9Panes%?create a horizontal pane"?create a vertical paneh?move to the left pane. *j?move to the pane below *l?move to the right pane *k?move to the pane above *k?move to the pane above *q?show pane numberso?toggle between panes}?swap with next pane{?swap with previous pane!?break the pane out of the windowx?kill the current paneTmux Quick referenceSCREEN Quick Reference ()Top of FormBottom of FormGetting instart a new screen session with session namescreen -S?<name>list running sessions/screensscreen -lsattach to a running sessionscreen -xattach to session?namescreen -r?<name>the “ultimate attach”screen -dRR?(Attaches to a screen session. If the session is attached elsewhere, detaches that other display. If no session exists, creates one. If multiple sessions exist, uses the first one.)Top of FormBottom of Form ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download