LINUX - SHELL

This section contains details of SHELL programming and its underlying concepts. It also provides some additional resources for practice.

Introduction

Linux is an open source operating system that manages system’s hardware and resources (CPU, memory and storage). An operating system connects the software and hardware of the system to make them function. It handles the communication between the physical resources and the software. Linux came in mid-1990s and it has been widely used today in smartphones, cars, devices, appliances, etc.

Kernel: Kernel is the core component of Linux. It communicates with the system’s hardware while managing resources of the system. Kernel is the lowest level of the OS and handles memory, process and file management (CPU, RAM, I/O).

System User Space: This works as an administrative layer for system level tasks such as installation and configuration. This is a system library which can access kernel modules. It consists of an init system, shell, command line, daemons (background services), processes that run in the background, desktop environment (users interact with the desktop environment such as GNOME), and user interface which is responsible for displaying the graphics on the screen (Xserver).

Applications: Software using which users can perform tasks. Linux OS provides a lot of apps in its store to install. For example, the Ubuntu software center provides apps that can be installed with a click! Command-line can be used to access the system directly.

Why do we use Linux? Linux is a free, open-source operating system. It is considered the most reliable and secure platform. Linux can be used, run, modified, and redistributed by any user. Anyone can contribute to its source code and redistribute it.

When you work with Shell Scripts in UNIX or Linux, you always must consider the type of SHELL that you are using. It can be a BashShell, a KornShell, a C Shell, and so on. The reason is that the syntax is a little different in the handling of variables and functions etc. As part of the examples used here, we use BashShell.

Installation

Linux:

Check shell type.
You can start working if it is bash.
Otherwise, install bash using sudo apt install bash.

Mac:

mac comes with zsh shell type.
Install bash via Homebrew.

Windows:

Access Windows Microsoft Store.
Search for Ubuntu App.
Click on Install and we end up getting a SHELL Terminal.

Shell is a Linux command line interpreter which provides an interface between the user and the kernel. The Shell allows users to execute commands. It translates the command and sends it to the system. There are different types of shells such as Bourne (sh), Bourne again (Bash), Korn (Ksh), C-style (Csh, Tcsh) shell, etc.

A shell can differ from the other shell in terms of syntax. We are going to talk about bash shell throughout the course. To check which shell our current system has, do the following:

echo $SHELL

Here, SHELL is a variable. To see the path to the parent directory of the system:

echo $HOME

echo - It prints the text to the terminal - echo "Hello Linux!"
man - Displays the user manual of any command we pass as an argument. For example, man echo will display the complete manual of command echo
date - It displays the current date and time.
- Options: -u (to print the date and time in UTC),
- --date="string" (output based on the given string).
date
date -u
date --date="tomorrow"

Navigating SHELL

pwd - It prints the path of the current directory.
cd - It is used to move from one directory to another.
ls - List all the files in the current directory. Options: -l (to list all the files with details such as permissions, size, etc.)
which - It locates the command and prints its path. which python which ls which date

We use nano editor to create shell scripts with the SHELL. We usually save the shell scripts using .sh extension as it is a commonly used convention. .sh is primarily for UNIX/Linux to understand that this is a script and does not hold a similar prominence for file extension as in regard to Windows OS. By including #!/bin/bashon top of the script, we let the machine know that the script was written in BASH Shell standards. Below is the Hello World bash script –script1.sh. The line starting with # is a comment line.

#!/bin/bash
#Display a message
clear
echo "Hello World"

Ctrl+O to SAVE the file. It will write the file into the file system. Ctrl+X to exit the editor. To execute this script ./script1.sh needs to be called from the shell prompt. To execute a file, we need to have the required permissions on the file. ls -l command will give us the permission matrix on the files available in the respective directory. ls -a will list all files along with the hidden files which are set with (.) at the beginning of their name.

Permissions set -rw-r--r--[File/Directory][Owner][Owning Group][EveryoneElse]. The permission set, we've got read and write for the owner. We've got read for the owning group. So as the owner is root. The owning group is also called root and then we've got read for everyone else. What we don't have is execute for anybody. So, for example, if we type in the change mode command chmod +x script.sh to add the execute permission, now if we do a ls -l, the permission set will be updated to -rwxrwxrwx. Notice that we've got an x now for the owner, for the owning group, and for everybody. At the beginning of the permissions set hyphen - indicates a file and d indicates directory permission.

Environment Variables

An Environment variable is a dynamically named value that can affect the way of running processes on a computer. They are part of the environment in which a process runs.

echo $PATH – to print the path value of all the system-defined environment variables
env– to print all the environment variables available in the system

Input-Output Variables

A fundamental aspect of working with Shell scripts is working with input, output, and returning those results. Below is the script for script2.sh written in vi editor. There are different other editors like nano, vim, gedit etc. When we execute this script as ./script2.sh, we get to enter project code and we get the value echo on-screen.

#!/bin/bash
clear
echo
#Continue after reading 4 chars
read -n 4 -p "Enter Project Code:" project_code
echo
echo "Retrieving data for Project"  $project_code
echo

Variables are case-sensitive in BASH. To unset a value associated with the variable, use UNSET keyword. By default, all the declared variables with values will be un-set as soon as we close the BASH automatically. However, if we wish to set the value permanently available every time you login to BASH, go to .bashrc file to declare the variable and set value to it. This file will be called as part of the startup script. Few examples for variables

pwd –Current/Previousworking directory
echo "Current Usr:" $USER – to print the user logged in
Below code snippet is a way to declare and call the integer type variables.

declare -i intvar
intvar=345
echo $intvar

Below code snippet is a way to declare and call the string variables

declare -l rovar="Hyderabad"
echo $rovar

echo ${fakevar –"This is a test"} – If this returns blank then fakevar doesn’t exists
echo ${#rovar} – gives the count of elements in the rovar
echo ${rovar#*r} – truncates data until r and displays the rest of the value

Passing and Using Arguments in Shell Scripts

Below script args.sh which defines $1 and $2 variables. Values need to be passed while executing the script as follows ./args.sh 31 27

#!/bin/bash
echo 
if  test  "$1" = "" 
    then 
        echo "No first value supplied"
        exit
fi
if  test  "$2" = ""
    then
        echo "No second value supplied"
        exit
fi
clear
echo
echo "Sum of  values:" $1+$2='$1'+'$2'

Some in-built UNIX/Linux special variables

$0 - The filename of the current script.
$# - The number of arguments supplied to a script.
$* - If a script receives two arguments, $* is equivalent to $1 $2
$? - The exit status of the last command executed
$$ - The process number of the current shell
$! - The process number of the last background command.
The command-line arguments $1, $2, $3, ...$9 are positional parameters, with $0 pointing to the actual command, program, shell script, or function and $1, $2, $3, ...$9 as the arguments to the command

Input-Output Redirection

In BASH, > symbol prints data as output from the file and <symbol takes data from the file as input. Pipe |symbol is used to provide a response of one command to another. This will help us to move the content of one command to another command. Examples for redirection are illustrated below:

touch labs.txt– for creating a blank file called labs.txt. Open and add data using nano editor
cat labs.txt > /dev/stdout – to print the data as the output on Standard Output i.e. Screen
sort < labs.txt – to print the data in labs.txt in a sorted fashion
grep -i "t" < labs.txt – finds all the strings/elements which has "t"
find /<foldername>/ -size +800c – finds files in where there are more than 800 characters
ls | wc -l – takes the output of the first command, passes it to the second command, and returns the result.
wc -l names.txt | tee -a file2.txt – file2.txt contains the output of the first command

Controlling and Manipulating the Shell Script

In BASH, we can move a process/script execution to background or foreground based on the time consumed by the script to run on open SHELL. There could a long-running scripts which might take too much time. Such scripts can be moved to the background. To demonstrate this, go to the root folder and run find command. This will list all directories available on your root. Ideally, we use Cntrl+S to pause, Cntrl+Q to resume and Cntrl+Cto exit from a long-running script.

Type jobs to know the list of active scripts running at a given point in time.

cat /dev/random > /dev/null – is a random number with no response
cat /dev/random > /dev/null& will run the editor and type text and stop
fg <jobnumber> - to resume the background running job on to screen
bg <jobnumber> - to move the job to background out of the screen
kill <jobnumber> - to stop the job

Working with Built-in Variables

The command env gives us all the built-in variables used by the BASH Shell. HOSTTYPE, LANG, NAME, USER, SHELL, LOGNAME, PATH, LS_COLORS etc. are a few of the built-in variables.

ls /bin - gives us all the files under /bin folder location.
echo $? – to get the exit status of the last executed command.
- If it returns 0, then the last executed command was successful.
- Any non-zero value is bad news.
ls /fake– will throw an error whenever there is no fake folder.
- Now echo $? will return 1 - which is an error as the last executed threw an error.

An example file to use built-in variables in a script - machine_stats.sh The script will return 0 as output value when executed as ./machine_stats.sh as it was expecting an argument. Now pass ./machine_stats.sh 1, we get output value as 1 as we passed a value along with the file during its execution.

#!/bin/bash
clear
echo "Computer Name:" $HOSTNAME
echo "Currently Logged in user:" $USER
echo "Number of this script:" $0
echo "Number of parameters passed to this script:" $#
echo

Manipulating Variables

While executing the below example script file output.sh, we do see that the values of variables are being manipulated and stored in an external file. The command cat <hostname>_stats.txtwill give us the intended output from the script. Use rm _stats.txt to delete the file.

clear
echo
read -p "Enter the City Name:" cityname
echo "Computer:" $HOSTNAME >> $HOSTNAME"_stats.txt"
echo "City:" $cityname
echo "Linux Kernel info:" `uname -a` >> $HOSTNAME"_stats.txt"
echo "Shell version:" $BASH_VERSION >> $HOSTNAME"_stats.txt"
clear
echo
echo $HOSTNAME"_stats.txt file written successfully."
echo

A variable has a scope or limit of access based on where it can be operated. While executing the below script call_export_var.sh file, we used servername variable from this file and called it in a subfile called export_var.sh which was declared in the same script. By calling ./call_export_var.sh, we do not see the variable name as output. However, if we call ./export_var.sh script file, we do see the variable name being displayed from call_export_var.sh file. The variables have been manipulated in such a way that they can be declared in one file and can be called in another script as this defined the scope. To do so we need to use a command called export within the parent script so that this exported variable can be used in any other child script.

#Filename: call_export_var.sh
servername="Prod1"
export servername
./export_var.sh

#Filename: export_var.sh
clear
echo
echo "The Servername  is"$servername
echo

To understand the scope better, follow below scope.sh. The variables in the functions are localized until they are called outside. If func2 was called before func1 declaration, it returns no results. Thus, func2 should be called after func2 logic. Same with the case of func1

func1()
{
#declare sets the variable scope to local within this function
#declare  classname="BTech First Year"
classname="BTech First Year"
}
func2()
{
func1
echo $classname
}
func2

By calling the scope.sh script file as . ./scope.sh is called dotsouring, we can use the $classname variable outside the scope.sh file as dotsouring helps us re-use it in multiple places across the bash. Here the variable is declared internally in a script file and the output value is being used across the shell. declare command needs to be used to ensure that it is used in such a way.

Formatting Output Variables

BASH allows output variable formatting. Follow the below examples:

echo "Kevin said "Hello World"" - we get Kevin said Hello World
echo "Kevin said \"Hello World\"" - we get Kevin said "Hello World"
echo "Stock Price is $500" - we get Stock Price is 500
echo "Stock Price is $500" - we get Stock Price is $500
date - gives us Thu Jan 3 22:18:42 DST 2019
date +'%Y-%m-%d - gives us 2018-04-09
When datevar = date +'%Y-%m-%d,then echar $datevar - gives us 2018-04-09.

` backtick – is used to store results for the command in the variable

printf "%s\n" $costcenter - %s for string and \n is the newline character
printf "%.3s\n" $constcenter .3s\n - displays only the first 3 strings
When numvar=5.5, echo $numvar- gives 5.500000 with 6-digit precision
printf "%f\n" $numvar- %f for floating-point number
When numvar=5673, printf "%d\n" $numvar- %d for integer

Creating Functions

The function is a code snippet that is defined and used to re-run or re-use a logic. The below script func_lib.sh contains two functions userinfo() and ipinfo()which are called while running the script file as ./func_lib.sh. By dotsouring ../func_lib.sh, we can make the ipinfo function run outside the script as an individual function as we have raised the scope of this function from local to global.

function userinfo()
{
echo "Current username:" $USER
echo "User Home directory path:" $HOME
}

function ipinfo()
{
ip_var=`hostname -I | awk '{ print $1}'`
echo "IP ADDRESS:" $ip_var
}
userinfo
ipinfo

Math Expressions

In a Shell script, variables are by default treated as strings but not as numbers. This creates challenges for performing the math operation in shell scripts. However, there are few good commands which help us perform arithmetic operations in Shell.

+ add, -subtract, * multiply, / divide, ++ increment by 1, --decrement by 1, % modulus

let – it helps us perform the arithmetic operation through the command line. Below letmath.sh contains all its variations of let command

#!/bin/bash
#Basic Arithmetic using Let command
echo "Basic Arithmetic using Let command"
let a=15+20
echo "a= 15+20 =" $a #35
let "b=29*20"
echo "b= 29*20 =" $a #580
let a++
echo "a++ =" $a #36
let "x=$1+30"
echo "x= $1+30 =" $x  #30 + first command line argument
let u=16/4
echo "u= 16/4 =" $u #4
let y=4/16
echo "y =4/16 =" $y #ceil the value to smallest number

Expr – it is like let command. However, instead of saving the result to a variable, it prints the answer. Unlike let command you don't need to enclose the expression in quotes. You must provide spaces between the items of the expression. It is also common to use expr within command substitution to save the output into a variable. exprmath.sh contains all its variations of expr command

#!/bin/bash
# expr with space does give the value as output
expr 23 + 29  #52

# expr with no spaces just prints the string
expr  23+29 #23+29

# expr with double quotes just prints the string
expr  "23+29" #23+29

# expr with escape character (backslash) will give us multiply operator. 
# ‘*’directly does work here.
expr 5 \* $1 # prints based on the argument passed during execution

# we get syntax error if we give * directly
expr 5 * $1  #expr: syntax error

# modulus operation -remainder when the first item is divided by second
expr 21 % 2 #1

#expr with asubstitute in order to display it in a variable
a=$( expr 43 -5 )
echo $a #38

Double Parenthesis (( )) –It is a way to handle arithmetic operations by passing the expression within the parenthesis. doublemath.sh contains all the variations below:

#!/bin/bash
a=$(( 21 + 26 ))
echo $a #47

c=$(( 49-3))
echo $c #46

b=$(( a * 12))
echo $b #564

x=$(( $a  / 2 ))
echo $x #23

(( c++ ))
echo $c #47

#adding 3 values to c
(( c += 3 ))
echo $c #50

bc (Basic Calculator) –This is a plugin in BASH to run basic calculator application. Use the command apt-get install bc to install the basic calculator. By just running bc command, the basic calculator will be opened in interactive mode. We can run the bc command in non-interactive mode using | symbol. bc helps us to handle the floating-point value-based calculations. Non-Interactive mode - echo “579*2.5” | bc

Below is an example script for usingbc command within a script. Floating points are handled without any issues using bccommand.

#!/bin/bash
clear
echo
read -p “Enter hours worked:” hoursworked
read -p “Enter hourly wage:” hourlywage
echo

grosspay= `echo “$hoursworked*$hourlywage” | bc`
echo “Gross pay is: \$” $grosspay

Conversion Numbers between bases –We sometimes try to convert the numbers from base-2 to base10 or base-16 or binary based on our need while working with numeric data. Using BASH, the below file binary.shcan be used to convert our input base-2 IP Address value to Binary values. Here the input is - 192.168.1.1 and output is - 11000000 10101000 1 1

#!/bin/bash
clear
echo
read -p “Enter IPv4 address:” ip_address
for ((i=1; i<5;i++))
do
    octet[$i] = `echo $ip_address | cut -f$i-d "."`
    #array of i value to fetch the values from IPv4 address
    #Your Input ExampleIP192.168.1.1
    #cut command to cut IP address into parts using$icounter
    #d is a delimiter for cut. "." is our delimiter here.
done
for ((i=1;i<5;i++))
do
    current_bin_val=`bc <<< "obase=2; ${octate[$i]}"`
    #Logic to convert current octet to binary 
    #bc command to convert octate arrange $i value
    full_bin_val=$full_bin_val" "$current_bin_val
    #add binary values in loop
done 
echo
echo "Binary Value of IP address:" $full_bin_val
#binary value output for your input IP 11000000 10101000 1 1

test Command-This command checks file type and compare values on the command line and returns the success flag based on true or false condition.

test 27 -gt 2 && echo "Yes" – Returns Yes, as 27 is greater than 2
test 27 -lt 2 && echo "Yes" –Returns nothing as we have not defined false condition
test 27 -lt 2 && echo "Yes" || echo "No" – Returns No, as 27 is not less than 2
test 25 -eq 25 && echo "Yes" || echo "No" - Returns Yes, as 25 is equal to 25
test 15 -ne 10 && echo Yes || echo No – Returns Yes, as 15 is not equal to 10
Below cod
e snippet returns relevant echo based on the presence of the respective file.
test -f /etc/resolv.conf && echo "File /etc/resolv.conf found." || echo "File /etc/resolv.conf not found."

Working with Arrays

The Array is a type of variable which is the collection of multiple items of the same type. The counter in the array starts with zero –0. Declared as depts= (“cse” “cnd” “ece”) as depts being the name of the array. To fetch the output - echo $depts. Below are the variations on working with arrays.

echo $depts – will return 0th value or first value in the array i.e. cse
echo ${depts[]}–returns error as bad substitution
echo ${depts[2]} – returns third value i.e.
echo ${depts[3]} – returns nothing as there is no value for dept[3]
depts[3] = "research" – to insert the fourth element in the dept array
echo ${#depts[*]} – returns the count of elements in the array i.e. returns
echo ${depts[*]:2:1} – skips the first two elements and only prints the next one value i.e. returns
echo ${depts[]/research/phd} – replaces the “research” element in the array with “phd” on runtime. However, echo ${depts[]}still shows research as the change is only for runtime.
depts2 = echo ${depts[*]/research/phd}- to store the changed deptsarray into depts2 array. echo ${depts2[*]} returns values with phd in the fourth position.
echo sort <<<"${depts[*]}" - triple redirection to sort a text in array argument but not in a file argument.

If and Else Conditions

If and Else helps us handle and judge the conditions on data, we manage. Below are a few examples of Syntax. Highlighted are the required syntax constructs of if & else.

#!/bin/bash
if test $USER := "root" 
#to open if. Used $USER env variable to check user
then
{
clear
echo "Your are not root"
}
else
# to open else
{
clear
echo "Your are root"
}
fi#close the if

Running If and Else on command like as below. As soon as you close if using fi keywords, the if and else logic will get executed directly.

root@ubuntu1:/scripts# numdays=400
root@ubuntu1:/scripts# echo $numdays
400
root@ubuntu1:/scripts# iftest "$numdays" -gt 100
>then
>{
>echo "Over Budget!"
>}
>fi
Over budget!
root@ubuntu1:/scripts#

Looping Structure - for

For loop is used to repeat a task in a specific number of times we would like to process something. Below for.sh is an example to demo a FOR loop with conduction using INkeywordto list of .sh files in each sub directory based on the total size of the file. The below script will print the size of all the .sh files in a list and then gives us the sum of the total size. Highlighted are the required syntax constructs for the For Loop.

#!/bin/bash
clear
#declaring variables so that can be used
totalsize=0
currentfilesize=0

#currentfile is a temporary variable used for counter
for currentfile in/scripts/*.sh 
do
        currentsize=`ls -l $currentfile | tr -s " " | cut -f5 -d " "`
    #Gets file size and cuts it with space delimiter
    #tr command to squeeze the space on the file name spaces
        let totalsize=$totalsize+$currentsize
    #calculate the totalsizeecho $currentsize
done
echo
echo "Total space used by Script files is:" $totalsize

Below is an example for FOR loop using an array.The Array is a variable contains multiple items of the same type. For example, cities=("Chennai", "Hyderabad") is an array with two items. echo $cities will print the first value in the cities array i.e. Chennai. Highlighted are the required syntax constructs for theFor Loop.@ value in the array to call each element based on the increment value of i.

echo ${cities[0]} #return Chennai
echo $(cities[1]} #will return "Hyderabad"
root@Ubuntu:/scripts# for((i=0;i<${#cities[@]};i++));do
>echo ${cities[$i]}
>done
ChennaiHyderabad

More examples - For loop

#!/bin/bash
echo "Starting FOR loop"
for val in 0 1 2 3 4 5
do
echo "Printing Value:" $val
done
echo " Finished FOR loop"

#!/bin/bash
echo "Starting FOR loop"
for files in $PWD/*.sh
do
echo "Printing File names:" $files
done
echo "Finished FOR loop"

#!/bin/bash
echo "Printing FOR loop"
for x in {1..5}
do
echo "Printing $x times"
done
echo "Finished FOR loop"

#!/bin/bash
echo "Starting FOR loop"
for j in {0..10..3}
do
echo "Printing $j value"
done
echo "Finished FOR loop"

#!/bin/bash
echo "Starting FOR loop"
for a in $(seq 1 4 20)
do
echo "Printing FOR value $a times"
done
echo "Finished FOR loop"

#!/bin/bash
echo "Starting FOR loop"
for ((i=1;i<10;i++))
do
echo "Printing $i entry in the loop"
done
echo "Finished FOR loop"

Looping Structure - While

Unlike a FOR loop, when we use WHILE loop –we don’t necessarily know how many times we are going through the loop. While.sh is an example script that prompts the user to select an option, so while the user doesn’t choose to exit and live in the loop. Highlighted are the required syntax constructs of WHILE Loop.

#!/bin/bash
choice="0"
while (( $choice != "1"))
do
    clear
    echo
    echo "Please select and Option" 
    echo
    echo "1 –Exit"
    echo
    read choice
done

The script waits for the input choice from the user. Until the input is given as1 the loop is not broken. The loop will execute initially as the choice value is zero by default and waits for the input, which will be in the loop unless the user gives the value as 1.

Break and Continue inside Loop

Depending on the code, we might wish to break out of the loop or continue the loop so that we go back to loop statements and run them again. Below is an example for a break and continue. Unless we type exit, we will not be able to break the loop, else it still continues to run the same while again and again.

 #!/bin/bash 
 while true
 do
 clear
 echo "To leave, Type exit" 
 echo 
 read -p "What did you say?" choice
 if test $choice = "exit"
 then
 {
 break
 }
 else
 {
 continue
 }
 done

#!/bin/bash
while true
do
echo "Welcome NEO - The Matrix has you"
echo "Choose wisely - Red Pill or Blue Pill?"
echo
read -p "What is your choice? - Type Red or Blue: " pill
if test $pill = "Blue"
then
{
break
}
else
{
echo "Welcome to Matrix. Thank you for choosing us"
continue
}
fi

Case Statements

Case statements help us to test the value of something for multiple possible items, instead of using a bunch of If-else, we can use Case statements. Below is an example case.sh for your reference. Highlighted are the constructs used in the case statements.

#!/bin/bash
clear
echo "City Type" 
echo 
read -p "Enter your City:" city 
case $city in "Hyderabad") city_type ="Tier 2";;
# Pipe Symbol is a OR command
"Bengaluru" |"Mumbai") city_type="Tier 1";;
"New Delhi") city_type= "Capital";;
# here star is to consider anything else
*)clear; echo "Invalid City $city";exit;;
esac
#case in reverse to close the case statement
clear
echo "City Typefor $city has been set to $city_type"
echo

Additional Resources

Access Full-length Text Book for various other concepts here - Classic Shell Scripting
All Shell Types (Download links) - https://www.freebsd.org/ports/shells.html
A-Z Bash Commands - https://ss64.com/bash/
More Commands - https://docs.cs.cf.ac.uk/notes/linux-shell-commands/

PreviousNetworking NextPractice Questions

Last updated 3 years ago

Was this helpful?