Bash Cookbook

Introduction

Bash is a very versatile language, one that gets neglected often and gets crafted quickly and hurriedly as a glue between the application layer and the operating system shell / runtime environment. In my experience, learning bash is never a bad thing.

One should not rely on bash to create an enterprise application, but where the need arise a good developer should be able to craft a well-written bash script that is easy to read, easy to maintain, and doesn’t violate good programming practices (where possible)

Basics

Moving around the filesystem

The cd command is used to move around directories.

cd /absolute/path # brings you to a dir relative to the root (/) fs
cd relativePath   # brings you to a dir relative to your current dir
echo $PWD         # env variable that tells you where you are
cd -              # goes to the previous directory you were at

Hello World

Bash is a scripting language. So really you can program in it without having to write any file. Hello world is as simple as typing the following in the console:

echo "Hello World"

To store it in files, we create a file like so (for example, save this as helloWorld.sh)

#!/bin/bash
echo "Hello World"

Once the file is created, it needs to be made executable.

chmod 700 helloWorld.sh

And then it can be run like this:

./helloWorld.sh

Variables

Variables are weakly typed, they can be number of strings.

# set a variable
varname="value"
# print a variable
echo "varname's value is $varname"

Default variables:

# command line parameters (argv)
$1 $2 $3
# number of command line parameter passed (argv.length)
$#
# return code
$?

String Manipulation

A practical example is extracting filename, directory name, and extension from a given file path. Example illustrates this quite well:

# given this file path
thePath="/opt/someDir/someCoolFile.tar.gz"
 
# doing this...          # will give you...
dirPath=${thePath%/*}    # /opt/someDir
fileName=${thePath##*/}  # someCoolFile.tar.gz
fileBase=${fileName%%.*} # someCoolFile
fileExt=${fileName#*.}   # .tar.gz

So how does this work anyway?

${variable%pattern}  # Trim the shortest match from the end
${variable##pattern} # Trim the longest match from the beginning
${variable%%pattern} # Trim the longest match from the end
${variable#pattern}  # Trim the shortest match from the beginning

Functions

#function declaration
function aFunction()
{
   param1=$1
   param2=$2
   param3=$3
   echo "$param1 is $param2 but not $param3"
   return 0
}
 
#function invocation
aFunction "param1" "param2" "param3"

Commands parameters (arguments)

Commands parameters, or arguments are accessed with numbered variable like $1 (first argument) and $2 (2nd argument). See example script below:

example: helloargs.sh – basic numbered args

#!/bin/bash
echo "hello args a1=$1 a2=$2 a3=$3"

execution example:

$ ./helloargs.sh "hello" "world" "foo"
# should prints 'hello args a1=hello a2=world a3=foo

example: afunc.sh: – args in function exceeding $9

Important note: command args beyond $9 must be enclosed by curly brackets.

#!/bin/bash
function aFunc()
{
   echo $1 $2 $3 $4 $5 $6 $7 $8 $9 ${10} ${11}
}
 
# pay close attention to the use of ${10} and ${11}
# instead of $10 and $11
aFunc ".1" ".2" ".3" ".4" ".5" ".6" ".7" ".8" ".9" "a" "b"

execution example:

$ ./afunc.sh
# should print: .1 .2 .3 .4 .5 .6 .7 .8 .9 a b

Basic Loops and control

# loop by count
for i in {1..100}; do
    echo "my count is $i"
done
 
# dynamic result from command result
for f in $(ls); do
    echo "file: $f"
done
 
# loop forever until terminated, sleep 5
# seconds in between each echo
while [ true ]; do
    echo "I will run forever until ctrl-C"
    sleep 5
done

loop each line in file

Loop each line in a file
This will print each line in a file:

while read line
do
    echo $line;
done < file

GHASSTLY DUCK  (cmd building block)

BASH as a scripting language in itself does not offer much utilities to do any kind of useful activity. What it provides is an environment and framework by which all available commands and executable on the machine can collaborate together.

In my opinion the true power in bash lies in being able to mix and match various executables regardless of their language or origin (C, C++,Java, python) usually via stdout and pipe. Therefore the more commands you learn, the more you can do in bash.

The GHASSTLY DUC stands for the most basic set of commands: grep, head, awk, sed, sort, tail, ls, yes, date, uniq, cut and kill.

This section will cover basic usage of each of these. Their combination is virtually limitless – just by knowing these set of commands you can already solve A LOT of problem.

• grep – useful for filtering and searching and is very efficient. grep is used when working with a list of things.
• head – fetch the first n lines of a file (esp useful on really large files)
• awk – slice and dice a set of data into various columns and fields
• sort – useful for sorting
• sed – in-memory data stream modification
• tail – fetch the lastn lines of a file (esp useful on really large files)
• ls – list files in directory
• yes – say ‘yes’ very useful for automating cmds with y/n prompt.
• diff – spot differences between two files or directory. often used in conjunction with sort and uniq
• uniq – removes duplication and output distinct lines
• cut – extract a column subset from a dataset
• kill – useful for process lifecycle control, kill is also used to terminate rogue process as well as sends signals.

Processes and Lifecycle Management

Process management, signals, child process, lifecycle, and so on

Special Variable

The following are a listing of special variables related to processes and lifecycle.

$$        PID of current process
$PPID     PID of parent of current process
$!        PID of last backgrounded process (as mentioned above).

Spawn child process and get PID

Subprocess & PID, example using $! variable

. /fire/off/some/commands/somewhere.sh
childpid=$!
echo "child pid of spawned command is $childpid"

What you’ll want to do usually is spawning a child process, and continuously monitor the runtime status of that child process and exit only when the child process has finished running:

runChildProcess &
childpid=$!
# enter a loop that monitor childpid presence, and exit
# only when child pid is gone
while [ true ]; do
    ps -lfp $childpid;
    if [ $? == 0 ];
        echo "child is still running..."
    else
        echo "child pid is terminated"
        break
    fi
done
echo "parent process completes execution"

List child processes of a given PID

To check if a single PID is still present or not,

pid=$0 # obtain PID from param
ps -lfp $pid
if [ $? == 0 ]; then
    echo "process $pid is up"
else
    echo "process $pid is no longer running"
fi

Script derived from this Expert Exchange discussion

# Function Definition
function printPIDFamily() { [ -n "$1" ] && echo $1; for p in `pgrep -P $1`; do printPIDFamily $p; done; }
 
# Function use:
printPIDFamily

Kill a PID along with associated child

(Re)use the function printPIDFamily above,

printPIDFamily | xargs kill

Register signal handler

signal registry is handled using trap [see here]
This is an example of registering a handler for SIGTERM. You can read more about SIGTERM on Wikipedia , along with links to other signals.

on_die()
{
   echo "Dying..."
   # do cleanup here as necesary #
   # and then exit!
   exit 0
}
 
# register on_die on TERM signal (SIGTERM)
trap 'on_die' TERM

Commands incovation

A bash “commands” can either be:

1. A true bash-specific commands (such as fg and bg),
2. System executables (such as ls).
3. Functions defined from within a script.

Direct command invocation

Commands can either be invoked directly

function helloWorld()
{
    echo "hello world, I am $1"
}
 
# simple invocation: calls 'date' system execs
date 
 
# calling a function, but we know this already
# this will print 'hello world, I am zen sugiarto'
helloWorld "zen sugiarto"

Dynamic command invocation

Continuing from the example below, we can also call the helloWorld function that was just declared in this way:

funcName="helloWorld"
$funcName "dynamic zen" 
 
# the above is equivalent to calling the helloWorld function
# directly and passing 'dynamic zen' as the parameter. This means
# that the printed result will be 'hello world, I am dynamic zen'

A word of caution

I will put this in blockquote to put the emphasis:

A bash command can invoke any program, such as Java, Python, C and so on.
Java, Python and C can also make system calls, e.g. it is possible to invoke ‘ls’ from Java, Python and C.
However it is not possible to call functions declared within the script from these programs.

I will illustrate this with a combination of bash script and python program. Suppose, we have this test.sh script:

#!/bin/bash
function helloWorld()
{
    echo "hello world 1"
}
function butHelloWorld2()
{
    echo "this will be called anyway"
}
python aProgram.py
# call butHellloWorld2 anyway
butHelloWorld2

And of course, this file aProgram.py:

import os
import subprocess
echo "the 2 attempt below will not be successful"
os.system('helloWorld')
subprocess.call("helloWorld",shell=True)

If we then call the script:

$ ./test.sh

The result will NOT print “hello world 1″. Meaning that it is impossible for aProgram.py to call the bash function helloWorld() that way. But since we can make system call, we can always call the script from within the python program in this manner:

subprocess.call("./test.sh",shell=True)

Now, because of the way test.sh is scripted, butHelloWorld2 will be called regardless. This is the only safe and sure way to call a bash script function from inside a non-bash script program (such as a Python program, as illustrated above).

BASH Skeleton Script

This is the skeleton code for 90% of my scripts. Use them at your own risk

The Script

#!/bin/bash
# you need to update the above shebang line
# with the location of your bash exec.
# find her with 'which bash'
 
#--- commands ---#
 
# comment
function command1()
{
   echo "execute command1 $1 $2"
   return 0
}
 
function command2()
{
   echo "execute command2"
   return 0
}
 
#--- main block ---#
 
# attempt to invoke the first param as the command
# and subsequent param as parameters for that command
# Note: Bash can only go up to $9 so we use ${10}..${12}, and so on
$1 "$2" "$3" "$4" "$5" "$6" "$7" "$8" "$9" "${10}" "${11}" "${12}"
result=$?
if (( $result == 127 )); then
   echo "ERROR: command $1 is not implemented"
   exit 127
fi
exit $result

Usage

When you invoke them like this

$ ./script.sh command1 "param1" "param2"

the script will print:

executing command1 param1 param2

As you can see, it doesn’t even try to check which function to call, it passes through the first parameter ($1) as the function name to call, and passes the rest of the parameters as a parameter for the first function invoked. No if-else, just straight off the bat.

If the function is not available, return code will be 127, in which case we display an error message saying, the command hasn’t been implemented yet.