Operating Systems Technology



System Administration

Course Notes #17

Shell Scripts

You have already examined shell scripts in various labs. A shell script is a program that, when run, is interpreted in the Bash shell. This means that the program is not compiled, but instead each instruction is read by the shell interpreter and executed as if you were typing the commands in from the command prompt. Shell scripts can be as basic as a sequence of Linux commands or as complex as Java programs with input and output, loops, nested if-else clauses, variables (including arrays but not objects), functions (methods) and parameters. You will find some aspects of Bash shell scripting to be easy since you already know how to program, but you will most likely find the syntax very awkward (the Bash shell is based on Algol syntax, a programming language from around 1960!)

Why would you write a shell script? There are many tasks that you might face as a system administrator that you want to automate. By capturing the sequence of commands in a script, you can run the script any time you want (or you can invoke it automatically using at or crontab). You might, for instance, write a script that takes a textfile of people’s names and their classes for the semester and automatically generate class accounts for every student on a Linux machine. Or, you might write a script that searches all of the files in /home to see if any have bad permission settings (such as 000 or 777). Or, you might write a script that searches some of the log files for unusual activity (such as a number of failed log in attempts which might indicate someone trying to obtain unauthorized access).

Before we get into the specifics of shell scripting, note that you can write scripts in many different languages such as php, Javascript, Ruby or perl. The language you are using is that of the Bash shell, so we might call them Bash scripts. If you were using a different shell, your script would be in a different language with different syntax. The csh (c-shell) for instance uses syntax that looks like C.

Each script will be placed in its own file. To run the script, you must make sure that the script’s permission is executable. To run it, you use the syntax ./name where name is the filename containing the script. If the script permits parameters, you can list them after the name as in ./name file.txt –c or something like that. All Bash scripts must start with the line #!/bin/bash which tells the Bash interpreter that what follows should be executed by the Bash interpreter, which is located in /bin. Note that # is used for comments, so any comment you want to add will start with #.

Here, we will cover many of the things you will need to know to write your own scripts. These set of notes will be used over two sessions. You should read the entire set of notes before starting either of the two labs, and then have the notes accessible while you work through both labs and the accompanying homework.

Variables

Just as in other programs, you can have variables in a shell script. There are two differences between a script variable and a variable as found in languages like C or Java: first, you do not declare your variable but instead just assign it when you are ready to use it, and second, when you want to access the value stored in a variable, precede the variable name with a $. For instance:

I=0 // initializes I to 0

J=$I // sets J to equal the value stored in I

E=$((C+D)) // when using a variable in an expression, you surround the

// expression with $((…))

X=$((X+1)) // here, we increment the value in variable X

In addition to variables that you define in your program, you can also use variables that have been defined in the shell that you are operating in. For instance, $USER and $PWD as defined in your .bashrc, or a variable that you define at the command line prompt. However, to use one of these variables, it must first be exported. An example is shown later.

Examine your .bashrc to see other variables defined. If you define a variable at your command line prompt, it is also available for any shell scripts you run before you close that shell or log off.

Output Statements

The basic output statement is echo, much as you used early in the semester. The echo statement can combine literal text with variables and the results of commands. For instance, you might have the following code:

Name=Richard

echo Hello $Name, the date and time are `date`

Notice the use of $ for Name, and surrounding date with `` as we did earlier in the semester.

Here is an example script, we will call it script1:

#!/bin/bash

X=0

Y=1

Z=$((X+Y))

echo $X $Y $Z

Once we write script1, we need to change its permissions to be executable, so from the command line, I might do the following:

$ chmod 755 script1

$ ./script1

0 1 1 // the script’s output

$

Now consider the following modified version of script1:

#!/bin/bash

X=0

Y=1

Z=$((X+Y))

Q=$((Z+A))

echo $Z $A

A has not been defined in our script. We will define it from the command line.

$ A=5

$ ./script1

1 1

Why did A output 1 when we assigned it 5? Shouldn’t the output be 1 6? Actually, no, A in the shell script is not the same as A from the command line because we didn’t export A. So we continue by doing:

$ export A

$ ./script1

1. 6

And now we get the right output.

Here is another sample script, we will call it script2:

#!/bin/bash

echo Hello $USER, you are currently at $PWD

Now, we do the following from the command line:

$ chmod 755 script2

$ ./script2

Hello zappaf, you are currently at /home/zappaf

$

Input Statements

The input statement is read. As with other programming languages, you specify the variable(s) after the read statement so that the input is stored in the accompanying variable(s). Typically, you will only read in a single value in your read statement, so you would use read name, where name is the name of your variable. If you enter a numeric value, the variable will store the number, if you enter something comprised of any other characters, the variable will store a string (even if the input starts with numbers). As the read statement merely waits for you to enter something via keyboard, you will often want to precede your read statement with a prompting message using echo. Here are two examples:

echo Enter a positive number

read num

echo Enter your first name

echo fname

To use num or fname, recall that you will need to add a $ before the variable name as in $num.

Assignment Statements

A script assignment statement will look like this var=expr where var is a variable and expr is some mathematical expression. However, recall that if a variable is referenced in the expression, it is preceded by a $. If the expression contains more than just a variable, you must use $((…)). You can also use $[…] if you prefer. Here are some example assignment statements:

I=$J // sets I to the value stored in J

J=0 // sets J to 0

I=$((I+1)) // increment I

avg=$((sum/count)) // compute the average of count values that total to sum

z=$(((x+y)/(x-y))) // expression requires its own ( ) to force order of operations

z=$[(x+y)/(x-y)] // another way to write the previous instruction

NOTE: variables in the Bash Shell can only store integers or strings, but not reals/floats. So, for a division operation, you only get the whole portion of the quotient. To obtain the remainder, use % (the mod operator as found in Java or C). For instance, if x=12 and y=5, then x/y is 2 and x%y is 3 (12/5 has a quotient of 2 with a remainder of 3).

Conditions

If our shell scripts consisted of only input, output and assignment statements, it would perform the same way every time we would run it. Most programs require that we have control statements as well, to control what instructions get executed and how many times. We use selection statements (if-then, if-then-else, nested if-then-else) to select which statement(s) to execute, and iteration statements or loops (while, for) to control how many times statements get executed.

When using a while loop or if-then statement, you will want to specify a condition. Conditions are placed inside of [ ] marks. There are three general forms of conditions:

• Numeric comparisons, which use –eq, -ne, -gt, -lt, -ge, -le for equal, not equal, greater than, less than, greater than or equal to and less than or equal to respectively

• String comparisons, which use = or != for equal to and not equal to respectively

• File testing, which allows you test if a file exists, is readable, etc (as shown below)

-d – is the item a directory?

-e – does the file already exist?

-f – does the file exist and is it a regular file?

-h (or –L) – is the item a symbolic link?

-r – does the file exist and is it readable?

-w – does the file exist and is it writable?

-x – does the file exist and is it executable?

Examples: [ $x –gt 0 ] // is x greater than 0?

[ $Name = Frank ] // does the name store “Frank”?

[ -d $i ] // is the value of $i the name of a directory

The syntax is tricky as the interpreter requires a space before and after each item in the [ ]. So for instance, you must use [ $x –gt 0 ], not [$x –gt 0] and you must use [ -d $i ] instead of [-d $i] or [ -d$i ].

You can combine conditions using AND, OR and NOT. For not, use ! at the beginning of the condition as in [ ! –e $f ] for does this file not exist? If you want to use NOT for –gt, just change your condition to –le, and to use NOT for =, just use !=.

For AND and OR, use && and || (this is the same as in C or Java). But, if you are going to use one of these, you MUST wrap your condition inside an additional set of [ ] as in [[ -r $f && -x $f ]] which tests to see if file $f is both readable and executable or [[ $x –le 100 || $x –ge 1 ]] to test to see if x is between 1 and 100.

The if-then Statement

Syntax: if [ condition ]; then action(s); fi

Or: if [ condition ]

then

action(s)

fi

The syntax can be tricky. Notice that you can omit the ; after the condition if you place each item on a separate line (other than the if and condition). The same is true of the actions. If there are multiple actions, separate them by ; on the same line, or just put each on separate lines. The statement must end with a fi. Forgetting the fi will most likely lead you to other errors.

Here is an example written two ways:

if [ $FOO –ne 0 ]; then echo $FOO; fi

or

if [ $FOO –ne 0 ]

then

echo $FOO

fi

NOTE: the indentations are strictly to enhance readability and not necessary, but useful.

Two additional examples are:

if [[ $FOO –gt 0 && $FOO –lt 10 ]]; then echo $FOO is within bounds; fi

if [ $FOO –gt 0 ]; then FOO=$((FOO-1)); echo $FOO; fi

It is recommended that you separate the words then, the if clause, and fi so that you do not have to worry about where to place your semicolons.

We can use if statements to help with error checking. Earlier, we used division in some assignment statements, but if we divide by 0, we will get an error. So we could modify our code as follows:

if [ $count –ne 0 ]; then avg=$((sum/count)); fi

The if-then-else Statement

In our previous example, what happens if count is 0? Do we want an alternate action? If so, we use the if-then-else statement instead of the if-then statement.

Syntax: if [condition ]; then action(s); else action(s); fi

Or: if [ condition ]

then

Action(s)

else

Actions

fi

This is almost identical to the if statement except that after the last then statement, you add the word else and list one or more statements for your else clause.

if [ $FOO -gt 0 ]; then FOO=$((FOO-1)); else FOO=1; fi

Or

if [ $FOO –gt 0 ]

then

FOO=$((FOO-1))

else

FOO=1

fi

We could rewrite our previous average example as

if [ $count –ne 0 ]; then avg=$((sum/count)); else avg=0; fi

or

if [ $count –ne 0 ]

then

avg=$((sum/count))

else

avg=0

fi

Here is an example that we might actually use as system administrators. This instruction will test to see if the file is executable and if not, will change its permission to be executable (by the owner only)

if [ -x /home/foxr/foobar.txt ]; then echo file is already executable; else chmod 700 foobar.txt; fi

Alternatively, using ! we can simplify the above statement:

if [ ! –x /home/foxr/foobar.txt ]; then chmod 700 foobar.txt; fi

The Nested if-then-else Statement

What if we have more than just two possible outcomes (true or false) such as if we want to assign a letter grade based on the student’s score being >= 90, between 80 and 89, between 70 and 79, between 60 and 69, or < 60? We place if-then-else statements inside of other if-then-else statements. The resulting structure is known as a “nested” if-then-else statement.

Syntax: if [condition ]; then action(s); elif [ condition ]; then actions(s); elif [ condition ]; then actions(s); else actions(s); fi

The alternative syntax will look like this:

if [ condition ]

then

action(s)

elif [ condition ]

action(s)

elif [ condition ]

action(s)

…

else

action(s)

fi

You can have as many elif clauses as you like. There is only one fi statement for the entire structure.

Here is a simple example:

if [ $FOO –lt 0 ]; then echo cannot take square root; elif [ $FOO –eq 0 ]; then echo 0; else echo too lazy to compute square root do it yourself; fi

Or

if [ $FOO –lt 0 ]

then

echo cannot take the square root

elif [ $FOO –eq 0 ]

then

echo 0

else

echo too lazy to compute square root do it yourself

fi

Here is how we can assign a letter grade based on a student’s score.

echo Enter student score:

read score

if [ $score –ge 90 ]

then echo Student Grade is A

elif [ $score –ge 80 ]

then echo Student Grade is B

elif [ $score –ge 70 ]

then echo Student Grade is C

elif [ $score –ge 60 ]

then echo Student Grade is D

else echo Student Grade is F

fi

It is common for a student to solve this as follows instead:

echo Enter student score:

read score

if [ $score –ge 90 ]

then echo Student Grade is A

elif [[ $score –ge 80 && -lt 90 ]]

then echo Student Grade is B

elif [[ $score –ge 70 && -lt 80 ]]

then echo Student Grade is C

elif [ $score –ge 60 ]

then echo Student Grade is D

else echo Student Grade is F

fi

However, we do not need to use the && and compound conditions above because, to reach the first elif statement, score must not be >= 90, so we don’t need to test –lt 90.

The for Loop

Syntax: for var in list; do statement(s); done

Or: for var in list

do

statements

done

Again, indentation is not needed but helpful. The var is any variable name you wish to use. You may reference the variable in your statement(s) by using $var. For instance, for i in 1 2 3 4; do echo $i; done will output each number from 1 to 4 on different lines. As with the if statements, after the word do, you can list multiple statements. If placed on a single line, the statements must end with ; so that the bash interpreter knows when the next statement begins. If you separate them onto other lines, they do not need the ;.

While we can use the for loop as shown above to work through a list of values, we will more commonly use it to work through either a list of files or through a list of values input. We will explore the latter later in this news.

To iterate through a list of files, we can specify a directory in the in statement, as in for x in /home/foxr do. Or, if we just specify *, it works through the current directory. We can also use *.txt, or some other regular expression. The following example will print all of the .txt files in the current directory. It is, in essence, doing ls *.txt.

for x in *.txt

do

echo $x

done

Lets enhance the above code also output the number of .txt files that are writable. How do we do this? We first have to add a counter variable to the program. We will initialize it to 0. We will call it count. We initialize it before the for loop (if we initialize it to 0 in the for loop, then count will become 0 every time we go through the loop). Next, we need to add an if statement to test if the current file is writable. If it is, we add 1 to count. Finally, we need to output count, but we will do this after the for loop or else it will output the message every time through the loop. Here is our enhanced script:

count=0

for x in *.txt

do

if [ -d $x ]

then

count=$((count+1))

fi

echo $x

done

echo The number of .txt files found in this directory that are writable is $count

A common set of code found in various shell scripts is to first test to see if a file exists, before doing something with or to that file. Below is code that will work through the current directory to see which files are not regular.

for x in *

do

if [ ! –f $x ]

then

echo “$x is not a regular file”

fi

done

This set of code changes any non-executable files in your directory to have 700 permission (rwx for owner, no access for anyone else)

for x in *.txt

do

if [ ! –x $x ]

then chmod 700 $x

fi

done

While loop

Syntax: while [ condition ]; do statement(s); done

This loop will test a condition to determine whether to execute the statements (the loop body) or not. The loop continues to execute the loop body while the condition is true. Once the condition becomes false, the loop ends and the next instruction in the script is executed. If the condition is never true to begin with, then the loop body is never executed. Note that the for loop allows you to iterate through a list such as the contents of the current directory, but the while loop doesn’t permit that feature. Here is an example of using the while loop that uses the read statement.

echo Enter a filename, quit to exit

read file

while [ $file != quit ]

do

ls –l $file

echo Enter a filename, quit to exit

read file

done

Notice here that we repeat the echo and read statements. Why? Without the read inside of the loop, we would have an infinite loop. The echo inside of the loop instructs the user what to enter otherwise, after outputting the ls –l of the file, the user would see the cursor blinking without knowing what to do next or why the script is waiting.

We will most likely use a while loop to coincide with input (from the user or a disk file) whereas the for loop will be used to iterate through a list of values instead.

Case statement

Usually, if we have multiple conditions/actions, we will use a nested if-then-else structure. But in some cases, when we want to test a variable against a list of specific values, we can use the case statement. For instance, if the user inputs a choice between 1 and 4 and I want to do a different action if the input is 1, 2, 3 or 4, I can use the case instead of the nested if-then-else. Here is what the case statement looks like:

Syntax: case $var in value1) statement(s); value2) statement(s); value3) statements; esac

And a simple example:

case $FOO in

1) FOO=$((FOO+1)); echo FOO is now $FOO;;

2) FOO=0; echo FOO is now $FOO;;

3) echo FOO is 3;;

esac

Notice the peculiar way that we list the test values (value followed by close paren) and the way we end each group of statements with an additional ;. You can use if-elif-else statements instead of case.

Using Parameters

A shell script can receive parameters, just like a normal Linux command. For instance, if you do rm /home/zappaf/myfile1.txt, the /home/zappaf/myfile1.txt is a parameter passed to the rm program. You can write your script to receive any number of parameters. You can also test to see what values they are, for instance if you want to write a script to receive parameters like –a or file names or other types of things.

Inside the shell script, a parameter is accessed by using $n where n is the numeric placement of the parameter. The first parameter is $1, the fifth parameter would be $5 and so forth. $# gives you the number of parameters and $@ returns all of the parameters in a list. The following script would output each parameter on a separate line.

for i in $@

do

echo $i

done

If this was in a script called foo and you invoked it as ./foo a b c d, the output would be a, b, c and d on separate lines. The following code would perform an ls –l on each filename provided.

for name in $@

do

ls –l $name

done

Imagine that a shell script required at least one parameter. You can test to see if any parameters were passed to the script by using

if [ $# -gt 0 ]

The condition is true if $# (the number of parameters) > 0. So you might have code like this:

if [ $# -gt 0 ]

then

… // do whatever operations are needed

else

echo Illegal use of this script, at least one parameter is expected!

fi

Let’s write a script that will add two numbers together and output the sum.

if [ $# -eq 2 ]

then

echo $((1+2))

else

echo Error in input, expecting two parameters and received $#

fi

The following code will output whether each of a list of file names is of an existing, regular file.

if [ $# -eq 0 ]

then

echo Error, no parameters passed

else

echo The following are names of files that exist and are regular

for i in $@

do

if [ -f $i ]

then echo $f

fi

done

fi

Accessing Files

There are a couple of ways to access information in files from a script. First, you can pass the filename as a parameter and then use the cat instruction to obtain each string in the script and process them all in a for loop. We can also access all items in a file using the while read statement (see the next section).

As an example, assume a file, numbers.txt, consists of a list of integer numbers. We write the following script, called counter, and run it with ./counter numbers.txt

Notice that the script makes sure that the file is valid, and also makes sure that the file has some numbers in it.

#!/bin/bash

sum=0

count=0

if [[ ! –f $1 || ! –r $1 ]]

then

echo $1 is not a valid or readable file

else

for value in `cat $1`

do

sum=$((sum+value))

count=$((count+1))

done

echo Number of values is $count

echo Sum of values is $sum

if [ $count –gt 0 ]

then

echo Average of values is $((sum/count))

else

echo File empty, cannot compute average

fi

fi

The while read Statement

If the file being input consists of different types of values, we want to use a different approach. Assume we have a file that consists of records (rows) where each record is a student’s name, major, GPA and number of hours earned to date. We want to compute the average hours for all CSC and CIT majors. The while read statement can be used. The syntax for while read is while read field1 field2 field3 where field1, field2, field3 are the names of the values we will read in. In our example here, these will be name, major, gpa and hours. The following code will compute the average for us.

#!/bin/bash

totalStudents=0

totalHours=0

while read name major gpa hours

do

if [[ $major = CIT || $major = CSC ]]

then

totalHours=$((totalHours+hours))

totalStudents=$((totalStudents+1))

fi

done

if [ totalStudents –gt 0 ]

then

echo Average hours of CIT/CSC majors is $((totalHours/totalStudents))

else

echo There were no CIT or CSC majors

fi

If this script was called avg_hours and the file was class_list.txt, we would invoke this as ./avg_hours < class_list.txt.

What if we wanted to do the same for the average GPA for our students? It would seem sensible just to change hours to GPA and totalHours to totalGPA, but remember that the Bash interpreter cannot handle real (float) values, so you would wind up with run-time errors.

As another example, imagine that we have a list of usernames in a file and we want to find out which ones have directories in /home (as opposed to say /home/CIT370/ or some other location). How do we do this? We use the while read to get each name from the file and then we test to see if /home/$name exists using –d. The code follows.

#!/bin/bash

while read name

do

if [ ! –d /home/$name ]

then

echo $name

fi

done

If the shell was called directory_checker, and our file of students is students.txt, we would invoke the shell as ./directory_checker < students.txt.

What if we wanted to see if the student listed in the file has an account and does not have a /home directory, and if so, create that directory? Well, this is a little trickier. We can test to see if $name has a directory in /home as shown above. But how do we know if $name actually has an account? Recall that all users’ username are listed in /etc/passwd. We can use grep on $name to see if it exists in /etc/passwd. But then what? The grep command will either return the line where it found the entry, or an empty line. What if we piped the result of grep to wc –l to count the number of responses? It should either be a 0 (if $name doesn’t exist in the file) or 1 (if $name does exist). So we add the following prior to the if statement in our script

exists=`grep $name /etc/passwd | wc -l`

We then alter our if statement to test to see if exists is 1 and ! –d /home/$name is true. If both are true, then $name is a real account but without a directory in home. The if statement will look like this

if [[ $exists –ne 0 && ! –d /home/$name ]]

Finally, instead of doing echo $name as our action, what we want to do is create a directory for $name in /home, so our statement changes from echo $name to mkdir /home/$name. We may also want to state the operation that took place so that the system administrator knows. But imagine that we run this shell script every week using crontab. Instead of outputting the result to the screen, lets output the resulting action to a log file called home_dir_new_users.txt. Our final script will look like this:

#!/bin/bash

while read name

do

exists=`grep $name /etc/passwd | wc –l`

if [[ $exists –ne 0 && ! –d /home/$name ]]

then

mkdir /home/$name

echo “$name directory created in /home”

>> home_dir_new_users.txt

fi

done

Using Advanced Linux Operations

Here, we look at combining shell scripting with the awk command. Consider for instance that you want to compute the sum of the size of all files that are readable in a directory. How can we do this? We can output the names of all readable files by using for i in * combined with if [[ -f $i && -r $i ]] then echo $i. But how do we get the file sizes? If we use echo `ls –l $i` in place of echo $i, then we get a long listing of the file information, but still this does not sum up the values we want. Recall that awk can return (print) a particular field from the input it matched. We can use awk to do a {print $5} since the file sizes are the 5th column of an ls –l. The awk statement will look like this: awk ‘{print $5}’ but rather than specifying a filename, we want to pipe the ls –l $i to the awk statement. So now, our statement will look like this: `ls –l $i | awk ‘{print $5}’`. Notice how this ends, with ’ to end the awk statement and ` to end the “this is a Linux command to execute”. Now what do we do with the value returned from awk (which is the size)? We add it to a running total. Here is the code that will perform this entire operation:

#!/bin/bash

sum=0

for i in *

do

if [[ -f $i && -r $i ]]

then

temp=`ls –l $i | awk ‘{print $5}’`

sum=$((sum+temp))

fi

done

echo The readable files in this directory are $sum in size

Note that we did not need temp but instead could have added the statement in ` ` into the assignment statement as sum=$((sum+`ls –l $i | awk ‘{print $5}’`)) but that syntax is very ugly and awkward. We can also use grep (or egrep) to test for patterns. For instance, if we want to find all files whose pattern is –rwxrwxrwx and place the file names in a log file so that we can later tell owners to correct the permission, we might use code like this:

#!/bin/bash

for file in *

do

if [ -f $file ]

then

perm=`ls –l $file | awk ‘{print $1}’`

if [ $perm = ‘-rwxrwxrwx’ ]

then

echo $file >> bad_permission_files.txt

fi

fi

done

Here, we test each file to see if it is an existing file and if so, we obtain its permissions using awk. Now we compare this string against –rwxrwxrwx. If the permission matches, we echo the file name and append it to the file bad_permission_files.txt.

One thing to recall about awk is that it actually has a built-in loop. That is, it will iterate through all of the input for us. So we could actually accomplish the same task with

#!/bin/bash

ls –l | awk ‘/-rwxrwxrwx/ {print $9} >> bad_permission_files.txt’

The only flaw with the above code is that it doesn’t test to see if each file tested is truly a file (i.e., using –f as we did in the previous script). We have no way of including the condition –f here since awk is iterating through all files and not testing a specific file. So we can use awk in some cases, but not always.

Note that in both of these scripts, we are appending the output to a specific file that we hard-coded into the script. We do not necessarily have to limit a script to this action. Instead, we could allow the user to specify the filename by using >> at the command-line prompt when calling the script. If the filename is included as a parameter, we would use echo $file >> $1. We could write our echo statement like this:

if [ $# -gt 0 ]

then

echo $file >> $1

else

echo $file >> bad_permission_files.txt

fi

Alternatively, we could just leave the echo statement as echo $file with no redirection of the output and then the user can either see the output on the screen, or redirect the output to a file by using something like ./script >> bad_permission_files.txt.

Consider a script that calls for an input file to be used with a while read statement, and then the user wants to redirect the output. The syntax to invoke such a script would look like this:

./script < inputfile > outputfile

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download