Shell Scripting

Shells provide an interface for controlling programs and their input and output.

There are 4 different shells

• Bourne (sh)
• C (csh)
• Korn (ksh)
• Bash (Bourne-Again)

The Bourne shells are ideal for string and word manipulation where as C shell is best used for mathematical scripting. Korn shell was an attempt at merging these two shells at a point but was later dropped and Bash was created.

All of the above shells are line-oriented, with features dating back to the earliest UNIX shells for Teletype terminals. All shells operate in a loop:

1. Display a prompt
2. Ready input command line
3. Perform substitutions in command line
4. Execute command
5. Loop to step 1

The substitutions in step 3 may be of : variables, filenames, command outputs, aliases, depending on the shell. For the following documentation we will be dealing with primarily Bash shell, the currently most common shell on Linux distributions

Command Substitution

Many commands that take arguments ignore standard input (eg. ls, file, echo, cd, rm). This is a problem for passing around results between commands

A conundrum: ls prints the list of file names to standard output, file though takes filenames as arguments and determines the file type. So how do we get the output of ls as an input for file ?

This won't work:

ls | file

So how do we do this ? When the shell encounters a string between backquotes such as `cmd` it executes cmd and replaces the backquotes string with the standard output of cmd, with any trailing newlines deleted. So for out ls and file problem we can solve it like this:

file `ls`

In this scenario ls will be executed first, which will then be passed as parameters to the file command, getting file information about the current directory

Standard output of any command can be made into arguments to any other command like this:

$ date
#will print: Thu May 10 11:33:46 PDT 2012

$ echo The date is `date`
#will print: The date is Thu May 10 11:34:33 PDT 2012

$ echo The date is `data| awk '{print $2, $3 ",", $6}'`
#will print: The data is May 10, 2012

Defining Variables

Shell variables can hold single string values, created with = and accessed $. NOTE they are ONLY string variables. Also spacing matters here. There can be not spaces between the variable name the '=' sign and the value being assigned.

Variable concatenation is also possible using a special syntax. You can concatenate one variable into the other by surrounding the one variable in {}

Examples

$ ub=/usr/bin
$ cd $ub;pwd

This example assigned the /usr/bin folder to the variable ub and then cd's into the variable and then asks for the current directory. Note here that we are assigning variables directly in the console. You can create and use variable both in script files and in the console. You variables will only last in console until the shell terminal is closed.

Note in the above aswell the commands are seperated with a semi-colon. This allows you to make multiple sequential commands on the same line / same command execution

$ curse='@$*&>##! is a legalized virus!'
$ echo $curse
@$*&>##! is a legalized virus!

The above example shows how to include whitespace or matecharacters into assignments. Note that the difference is surrounding the string with single quotes

$ dir=`pwd`

Using the techniques in the previous section here we can store the output of a command into a variable

$ one=1

$ echo $onetwo  #does not work - variable does not exist

$ echo ${one}two #value of variable one, text two
# will print: 1two

Metacharacter Suppression

With shell using so many characters with secondary special meanings (eg /, $,`, \) what do you do if you want literaly these items in your strings ? Using Metacharacter Suppression you can cancel out these second meanings and use these characters for thier literal use

The following rules work for character suppression:

• Single quotes suppress the special meaning of ALL metacharacters
• Double quotes suppress the special meaning of all metacharacters EXCEPT \ (backslash), $ (dollar sign) and ` (backquote)
• Backslash suppresses the special meaning of the immediately following character (except between single quotes - because then it aswell is cancelled out)

Basically you can think of it as the same rules to PHP. Single quotes mean whatever is between them will be taken literally. Double quotes will attempt to resolve any variables or special meaning characters it finds within them, and backslash will escape the character, unless that creates a meaning within itself

Note that >, <, |, spaces and tabs are also metacharacters as they seperate arguments.

Examples

$ echo hello | wc -c         # one argument to echo
# will print: 6

$ echo hello '|' wc -c       # four arguments to echo
# will print: hello | wc- c

$ echo '.....hello | wc -c'  # one argument to echo
# will print: .....hello | wc -c

Patterns for grep and sed often contain metacharactersl its a good habit to surround arguments with single quotes

$ grep '|' chap2

$ sed 's?>/>>/g' foo/bar/readme

Shell Functions

A shell function associates a name with a list fo commands. When the name is given as a command, the list of commands is executes. All functions must be declared before they can be used. The syntax is:

function_name(){
    Commands
}

OR

function_name(){
    Commands
    return $TRUE
}

Example

$ mydate() { date | awk '{ print $2, $3 ",", $6 }'; }
$ mydate
#will print: May 10, 2012

Function Arguments

You can't pass arguments to shell functions. The actual () doesn't mean anything other then help shell identify a function. You can actualy put any parameters in it. You can pass parameters though through positional parameters using $1, $2, .... , $9 like syntax.

Example

nd(){
    cd $1
    PS1="`pwd` $ "
}

#calling the function in shell
$ nd /bin
# will print: /bin $

Positional Parameters and Special Variables

Arguments to a shell script are accessed by position parameters. There are some additional ways to access these parameters

• $1, $2, ...., $9 - Arguments 1 through 9
• $0 - Will be the script file's name
• $* - All Positional paramters (starting from 1)

Some other special variables include

• $# - Number of positional parameters to the script
• $$ - Shell prcess ID number
• &? - Exit status of last command

Shell Scripts

A sequence of commands stored in an ordinaty text file is called a shell script. The .(dot) build-in causes the shell to execute commands from a script

Example

$ cat f_defs
echo Defining nd and mydate
nd() { cd $1; PS1="`pwd` $ "; }
mydate() { date | awk ' { print $2, $3 ",", $6 }'; }


$ . f-defs
Defining nd and mydate
$ nd /bin
/bin $

Alternatively, shell scripts can be executed by setting the execute bit on the file

The awk Programming Language

awk is named after is authors: Al Aho, Peter Weinberger and Brian Kernighan.

It is a full programming language supporting:

• build-in and user-defined variables
• arithmetic and string operations
• loops and branches
• build-and user-defined functions

awk's simplest and most common use is to select and change the order of fields. awk teats each input line as a sequence of fields delimited with whitespace. Fields can be selected by number and printed:

$ awk '{ print $3 }'
UNIX is Linux
Linux
Linux is UNIX
UNIX
^d

Using single quotes makes the print action test a single argument to awk, with no metacharacter interpretation

awk can be used to reformat output of other commands:

$ date
# will print: Thu May 10 13:28:57 PDT 2012

$ date | awk '{ print $2 $3 $6 }'
# will print: May102012

$ date | awk '{ print $2, $3, $6 }'
# will print: May 10 2012

$ date | awk '{ print $2, $3, ",", $6 }'
# will print: May 10 , 2012

$ echo ipsos custodies? | awk '{ print "Quis custodiet ipsos " $2 }'
Quis custodiet ipsos custodies?

awk's field seperator can be changed with the -F options. For example -F: will make colon the identified field seperator. Then we could use this to print out all of the login names of all users in the passwd file on linuz as follows:

$ awk -F: '{ print $1 }' /etc/passwd
# prints list of all users

$ grep /home /etc/passwd | awk -F: '{ print $1 }'
# prints all users who have /home as part of thier login directory (essentialy finding all actual users)

shift and set Build-In Commands

The shift command shifts position parameters

# build shift_test script
$ cat shift_test
echo $0 $1 $5 $9 " Args:" $*, No: $#
shift
echo $0 $1 $5 $9 " Args:" $*, No: $#

# use shift_test script
$ shift_test A B C D E F G H I J
shift_test A E I Args: A B C D E F G H I J, No: 10
shift_test B F J Args: B C D E F G H I J, No: 9

Note that by shifting it is truncating off the first letter, like JavaScript arra's shift function. Note that the indexs though have not changed, only the value they point to has

The set command sets the positional parameters to its arguments. Essentially it overwrites the positional parameters with the returned values from whatever command is passed to set:

# build set_test script
$ cat set_test
date
set `date`
echo The date is $2 $3, $6

# execute set_test script
$ set_test
Thu May 10 13:42:46 PDT 2012
The date is May 10, 2012

The for Loop

General Form:

for var [in val_list]
do
    command1
    command2
    commandN
done

For example to display a string 10 times:

#!/bin/bash
for i in 1 2 3 4 5
do
echo "Hello $i World"
done

Shell also allows you to set a step value, allowing you to count by 2 or 3 or even backwards. The following example has default setp value of 1

#!/bin/bash
for i in {1..5}
do
echo "Hello $i World"
done

We can also specifty step arguments using the {START...END...INCREMENT} syntax:

# Only works for Version 4.0+
#!/bin/bash
echo "Bash version ${BASH_VERSION}..."
for i in {0..10..2}
do
echo "Hello $i World"
done

See in this example it will start at 0, end at 10 and increment by 2 as specified in the {0..10..2} of the for loop of the above example

If no value list is given, $* (the positional parameter list) is assumed:

for varname
do
echo $varname; echo “${varname}: printed”
done

Output:

# sh varprint.sh foo bar foobar
foo
foo: printed
bar
bar: printed
foobar
foobar: printed

The case Branch

Simply, this is the shell equivelent of Case or a Switch statement

General form:

case word in
pattern) commands ;;
esac

Note: the double semi-colons are NOT a typo. Nor is the single ')' closing bracket

In the above example 'word' is usual a variable substitution. It uses filename expansion-like metacharacters with | for "OR"

A case branch can be the body of a for loop, for example:

for i
do
    case ${i} in
        win[0-7])    echo "${i}: Windows" ;;
        linux|unix)  echo "${i}: Operating System" ;;
        *)           echo "${i}: Unknown" ;;
    esac
done

Output:

# sh word_names.sh win0 unix mac
win0: Windows
unix: Operating System
mac: Unknown

The if/else Branch

General Form

if commands
then 
    commands

    #else if condition
    [elif commands 
     then 
        commands]

    # else condition
    [else 
        commands]
fi

Note all of the else if and else commands are inside the if command which is closed with fi

The commands following if are executed. An exit status of 0 from the last command means true, non-zero means false. The if construct is typically used in conjunction with the test command which is covered in detail in the next section

The test Command

The test command evaluates an expression and yield an exit code of 0 for true, and non-0 for false

General form:

test expression

OR 

[expression]

There is also a new form used on Bash and Korn shells referred to as "new test" and use double brackeds ( expression ) instead of single given in the above example

test implements the old, portable syntax of the command. In almost all modtern shells '[' is a synonym for test (but requires a final argument of ']')

'[[' is a new improved version of it, and is a keyword, not a program. This makes it easier to use, as shown in the examples below:

Examples

if [ -z "$variable" ]
then
    echo "variable is null!"
fi

if [ ! -f "$filename" ]
then
    echo "not a valid, existing filename: $filename"
fi

----------------------------------------------------------------------------------------------------------------------

if [[ ! -e $file ]]
then
    echo "directory entry does not exist: $file"
fi

if [[ $file0 -nt $file1 ]]
then
    echo "file $file0 is newer than $file1"
fi

In the case of [[ glob expansion (pattern matching) will be done and therefor arguments need not be quotes. Because [[ is built in on to the shell and does not have legacy requirements, you don't need to worry about word splitting on variables that evaluate to a string with spaces. Therefor, there is not need to put the variables in double quotes.

Test has numerous primitives for determining features of files and strings

File Comparisons

Syntax	Use
-f <file>	True if the file exits, is regular file
-d <file>	True if the file exists and is a directory
-x <file>	true if the file exists and is executable
-s <file>	True if the file exists and has a file size large then 0

String Comparisons

Syntax	Use
s1 = s2	True if String 1 (s1) is identicle to String 2 (s2)
s1 != s2	True if String 1 (s1) is not identicle to String 2 (s2)

Integer Comparisons

Syntax	Use
n1 -eq n2	True if Integer 1 (n1) is equal to Integer 2 (n2)
n1 -ne n2	True if Integer 1 (n1) is not equal to Integer 2 (n2)
n1 -gt n2	True if Integer 1 (n1) is greater then Integer 2 (n2)
n1 -ge n2	True if Integer 1 (n1) is greater then or equal to Integer 2 (n2)
n1 -lt n2	True if Integer 1 (n1) is less then Integer 2 (n2)
n1 -le n2	True if Integer 1 (n1) is less then or equal to Integer 2 (n2)

Logical Operators

Syntax	Use
thing1 -a thing2	True if thing1 AND thing2 both return True
thing1 -o thing2	True if thing1 OR thing2 return True
thing1 ! <another-param> thing2	Inverses the meaning of the <another-param>. Is logical equivelent for NOT

There are also short-circuited Logical Operators. These can be used with test but order matters. These operators can also be used outside of if statements as short-hand of executing commands based on the short-circuited logic aswell

Short-Circuited Logical Operators

Syntax	USe
command1 && command2	True if both commands return True (exit status 0). If command1 does not return exit status 0, command2 will not be run
command1	command2	True if both commands return True (exit status 0). If command1 returns with exit status 0 (true), command2 will not be run

A good reference for all of the operator options is here : http://tldp.org/LDP/abs/html/comparison-ops.html

Combination Examples

#example with AND
test -f temp -a -x temp

#example with OR
[ -if temp -o -d temp]

#example with NOT
test ! -d temp

#example with groupings. NOTE WHITESPACING
[ ${i} = 'ready' -a \( -f temp -o -d temp -o -d tempt \) ]

Examples

Simple example using the if and test commands

if [ -f ${i} ]
then
    echo ${i} is a regular file
    else
        echo ${i} is not a regular file
fi

The next example shows a complete shell program that illustrates the use of "if", "test" and shell functions:

#!/bin/bash
usage(){
    echo "Usage: $0 username"
    exit 1
}

userlogged(){
    if (who | grep $1 > /dev/null)
    then 
        echo $1 is logged on
        else 
           echo $1 is not logged on
    fi
}

# call usage() function if filename not supplied
if test $# != 1
then 
    usage
    else 
        userlogged $1
fi

Using short-circuited logical operators we can simulate if and else statements. See the if else statement below:

if [ -f unixfile ]
then
    rm unixfile
    else
        echo "unixfile was not found, or is not a regular file"
fi

We can simulate this with short-circuited logical operators as:

[ -f unixfile ] && rm unixfile || echo "unixfile was not found, or is not a regular
file"

In addition, multiple commands can be executed based on the results of 1 command by incorporating braces and semi-colons:

command1 && { command2 ; command3 ; command4 ; }

If the exit status of command1 is true (zero), commands 2,3 and 4 will be performed

while and until Loops

General sytnax for the while loop is:

while commands
do
    commands
done

Example

while true
do
    sleep 60
    who
done

The General syntax for the until loop is:

until commands
do
    commands
done

Example

until who | grep Bill
do 
    sleep 30
done
echo Bill is logged in!

Filtering with Loops and Branches

Loops and branch output may be filtered.

Example

$ cat sys_hogs
for user in `awk -F: '{ print $1 }' /etc/passwd`
do
    if [ ${user} != 'root' -a ${user} != 'aman' ]
    then
        home=`awk -F: "/^${user}:/ { print \\$6 }" /etc/passwd`
        usage=`du -s ${home} | awk '{ print $1 }'`
        echo ${usage} ${user}
    fi
done | sort -nr | head | awk '{ print $2 ":", $1 }'

Complex scripts often require debugging

$ sh -x sys_hogs  # trace execution

Redirecting Standard Error

The standard input, output and error streams are associated with file descriptors :

• standard input - 0
• standard output - 1
• standard error - 2

• < redirects 0 (standard input)
• > and >> redirect 1 (standard output)
• Preceding > or >> with 2 redirects standard error

myprog < foo > foo.out 2> foo. err

• Standard error may be combined with standard output

myprog < foo > foo.allout 2>&1

The read Command

The read command is a simple command that allows for user input. The read command waits for input and then assigns it to variables passed as parameters to the read command

$ read line
This is a test

$ echo $line
This is a test

$ read word rest
This is another test

$ echo $word
This

$ echo $rest
is another test

Note that read typical will read in the phrase and then the return key as a second input. If you have multiple reads after one another you may end up with the return key as the input to the second read. The best way to avoid this is to assign two variables to the read command. 1 will take the input, the 2nd will take the return key, which at that time you can discard

You should also note that when read is given multiple variables as parameters to store the input text, read will put each individual word in its own variable using spaces to delimit the start and end of a word, until the last variable where it will place the rest of the read input.

Arithmetic with expr

The shell has no built-in arithmetic operations. The expr command provides arithmetic and other facilities for use in shell scripts or other contexts. expr takes an arithmetic expression as arguments and prints the value to standard output; metacharacters must be suppressed:

$ expr 7 \* \( 2 + 3 \)
35
$ a=1; a=`expr $a + 1`; echo $a
2