IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 27

Review

Using Braces, { } , with Variables
Special Parameters
$ - The PID of the Current Shell Process
! - The PID of the Last Process Put into the Background
? - The Exit Status
Positional Parameters
# - The Number of Command Line Arguments
0 - The Pathname of the Script
1 - n - The Command Line Arguments

New Material

Expanding Null and Unset Variables
:- Use a Default Value
:= Assign a Default Value
:? Print an Error Message
The [[ ]] Command
Who Uses These Features?
&& and || as Command Separators
An Introduction to AWK
Running awk
AWK Syntax
Blank Pattern
Regular Expression Patterns
BEGIN and END Patterns
Comparison Patterns
Range Patterns
AWK Fields
Contract for the Web

Microphone

Final Exam

The final exam will be held on Tuesday, May 14th from 3:00 - 6:00 PM.

The exam will be given in this room.

If for some reason you are not able to take the Final at the time it will be offered, you MUST send an email to me before the exam so we can make alternative arrangements.

60% of the points on this exam will consist of questions from the Ungraded Class Quizzes.

You do not need to study a Class Quiz question if the topic is not mentioned in either the Midterm or Final review.

The other questions I will make up specifically for this exam.

For these questions you will have to know

Absolute and relative pathnames
The PATH system variable
Access permissions
Redirection
Metacharacters
Utilities

The last class on Thursday, May 2nd, will be a review session.

You will only be responsible for the material in that review session, which you will find here, and the review for the Midterm, which you will find here.

Although the time alloted for the exam is 3 hours, I would expect that most of you would not need that much time.

The final is a closed book exam.

To prevent cheating, certain rules will be enforced during the exam.

Remember, the Midterm and Final determine 50% of your grade.

Course Evaluation

At the end of each semester we offer you the opportunity to say what you think about this course.

What have I done right?

What have I done wrong?

What can I do better?

You are not asked for you name.

So the submissions are anonymous.

I will not see your responses until after I have submitted grades for this course.

We collect this feedback through Course Evaluations.

I will use what you say to make this course better.

To complete the course evaluation, use the following link .

You have until Friday, May 24th, to submit the evaluation.

Today's Class

The material we cover today will not be on the final.

Instead, I'll talk about some advanced features of Linux/Unix that you should be aware of.

Questions

Are there any questions before I begin?

Review

Using Braces, { } , with Variables

If we try to concatenate the value of a variable with a string ...

we can run into problems

$ dir=hw

$ echo The directory is $dir11

The directory is

Bash did not see the variable dir next to the string "11"
Instead, it saw a new variable, dir11, which was not defined
Since this variable was not defined ...
it has no value
To concatenate the value of a variable with a string ...
that does not beging with . or / ...
we need to use braces, { }
The braces surround the name of the variable
They set off the name of the variable from surrounding text
```
$ echo The directory is ${dir}11
The directory is hw11
```
The opening brace comes right after the dollar sign, $

Special Parameters

Special parameters are shell variables whose values are automatically set by Bash
The parameters contain information about the current shell environment
They are very useful when writing shell scripts

$ - The PID of the Current Shell Process

The special parameter $ contains the process ID (PID) of the current shell
If you echo the $ at the command line

You will get the process ID of your current shell

$ echo $$
6834

$ ps
  PID TTY          TIME CMD
 6834 pts/1    00:00:00 bash
 7024 pts/1    00:00:00 ps

Notice that the value returned is the same as that of the shell
Notice also that $ is the name of the parameter
So I had to put a $ in front of it to get its value

! - The PID of the Last Process Put into the Background

! contains the process ID (PID) of the last process put into the background
```
$ sleep 60 &
[1] 7347

$ echo $!
7347
```

You can use ! to kill a job that you just ran

$ ./bother.sh > /dev/null &
[1] 5274

$ jobs
[1]+  Running                 ./bother.sh > /dev/null &

$ echo $!
5274

$ kill $!
[1]+  Terminated              ./bother.sh > /dev/null

$ jobs

$

? - The Exit Status

The ? special parameter returns the exit status of the last command

$ pwd
/home/it244gh/it244/work

$ echo $?
0

$ ls asdfasd
ls: cannot access asdfasd: No such file or directory

$ echo $?
2

The fist command succeeds and returns an exit status of 0
While the second fails and returns a non-zero exit status

You can set the exit code in a shell script by using the exit built-in

$ cat exit.sh
#! /bin/bash
#
# demonstrates the use of the exit command with a status code

exit 2

$ ./exit.sh 

$ echo $?
2

The exit status is also called the condition code or the return code

Positional Parameters

Positional parameters give the value of the command line arguments ...
to a shell script
They can also be used with functions

# - The Number of Command Line Arguments

The # special parameter contains the number of command line arguments

$ cat arg_count.sh 
#!/bin/bash
#
# Prints the number of arguments sent to this script

echo This script received $# arguments

$ ./arg_count.sh foo bar bletch
This script received 3 arguments

This parameter allows you to check that your script ...
gets the command line arguments it needs to run properly

0 - The Pathname of the Script

The 0 positional parameter contains the full pathname used to call the script

$ cat command_name.sh 
#!/bin/bash
#
# prints the pathname by which called this script

echo This script was called using the pathname $0

$ ./command_name.sh 
This script was called using the pathname ./command_name.sh

$ /home/ghoffman/examples_it244/command_name.sh 
This script was called using the pathname /home/ghoffman/examples_it244/command_name.sh

You should use this parameter when creating a usage message
But you should use it with the basename command ...
and command substitution ...
to remove the path part of the pathname
```
echo Usage: $(basename $0) ...
```

1 - n - The Command Line Arguments

The numbered positional parameters return command line arguments ...
to a script ...
or to a function

Here is an example

$ cat print_positionals.sh 
#!/bin/bash
#
# Prints the value of the first four positional arguments

echo 0: $0
echo 1: $1
echo 2: $2
echo 3: $3

$ ./print_positionals.sh foo bar bletch
0: ./print_positionals.sh
1: foo
2: bar
3: bletch

If there is no corresponding command line argument ...
the parameter will have no value

Attendance

New Material

Expanding Null and Unset Variables

If a variable is not set, its value is the empty string
```
$ echo $foo

$ echo --${foo}---
-----
```
Bash provides three ways to deal with this
- Use a default value for the variable just for this evaluation of the variable
- Use a default value and assign that value to the variable
- Display an error message
These features are useful when reading input from a user ...
because you cannot assume the user will give your script a useful value

:- Use a Default Value

When a script uses read to get input from a user ...
but the user enters no value before hitting Enter ...
you can supply a default value to the variable as follows
```
${VARIABLE_NAME:-DEFAULT_VALUE}
```

Let's look at an example

$ unset filename

$ echo $filename

$ ls -l ${filename}.txt
ls: cannot access .txt: No such filename or directory

$ ls -l ${filename:-foo}.txt
-rw-rw-r-- 1 ghoffman crontab 0 2012-04-25 13:27 foo.txt

When I tried to use the value of filename with ".txt" ...
I only got ".txt"
But when used with :-
filename was temporarily assigned the value "foo"
But this value does not stick
If I used the value of filename again
But did not use :-
I would get the empty string
```
$ echo $filename

$
```

:= Assign a Default Value

:- is useful ...
but the value does not stick
I can use the following construct to set a permanent value
```
${VARIABLE_NAME:=DEFAULT_VALUE}
```
For this construction we use an equal sign, =
Instead of a dash, -

Here is an example

$ unset filename

$ echo $filename

$ ls -l ${filename:=foo}.txt
-rw-rw-r-- 1 ghoffman grad 0 Nov 18 16:45 foo.txt

$ echo $filename
foo

:? Print an Error Message

Giving a variable a default value if it is not set is useful
But what if you could not think of a good default value?
In that case, you might want to use the following construction
```
${VARIABLE_NAME:?ERROR_MESSAGE}
```
With this construction no value is assigned to the variable ...

but the script will exit with an error message

$ cat var_error.sh
#! /bin/bash

# demonstrates the use of :?

unset arg # make sure there is no value in the variable arg

read -p "Value: " arg

echo About to echo the value of arg using :?
echo 'The value of arg is' ${arg:?'arg not set. exiting script'}
# otherwise                                                                      
echo Exiting normally

$ ./var_error.sh 
Value: foo
About to echo the value of arg using :?
The value of arg is foo
Exiting normally

$ ./var_error.sh 
Value: 
About to echo the value of arg using :?
./var_error.sh: line 9: arg: arg not set. exiting script

The [[ ]] Command

Bash provides [[ ]] as an alternative to the test command
It has more operators than test
This new command has the following format
```
[[ EXPRESSION ]]
```
It returns an exit status of 0 if EXPRESSION is true
Otherwise it returns an exit status of 1
This new command uses the same operators when evaluating numbers ...

but it has some new string operators

Operator	Condition Tested
<	The first string comes alphabetically before the second string
>	The first string comes alphabetically after the second string

Here are some examples

$ [[ "aa" < "ab" ]]; echo $?
0

$ [[ "ab" < "ab" ]]; echo $?
1

$ [[ "ab" < "abc" ]]; echo $?
0

$ [[ "ab" > "aa" ]]; echo $?
0

$ [[ "abb" > "ab" ]]; echo $?
0

This new command also has different logical operators

Logical Operator	[ ]	[ [ ] ]	Example
AND	-a	&&	[[ $value -gt 1 && $value -lt 4 ]]
OR	-o	\|\|	[[ $value -lt 2 \|\| $value -gt 4 ]]

The logical operators && for AND || for OR ...
are the same as those use in the C programming language
This new conditional command also allows you to use ( ) ...

to group expressions

$ numb=9

$ [[ ($numb -gt 1) && ($numb -lt 10) ]]; echo $?
0

$ numb=11

$ [[ ($numb -gt 1) && ($numb -lt 10) ]]; echo $?
1

With the new conditional command you can use wildcards to match a pattern

$ [[ artist = a* ]] ; echo $?
0

$ [[ aa = a? ]] ; echo $?
0

$ [[ aab = a?? ]] ; echo $?
0

But this only works but this only works ...

if the pattern is to the right of =

$ [[ artist = a* ]] ; echo $?
0

$ [[ a* = artist ]] ; echo $?
1

Who Uses These Features?

I would never use the features I listed above
Because I can do the same thing in a much easier way ...
by using a scripting language like Python
But if your job had you writing Bash scripts frequently ...
you might remember these features ...
and use them when appropriate

&& and || as Command Separators

On the command line && is the logical AND operator ...
and || is the logical OR
These operators are command separators just like ;
But these operators have a special property that can be very useful
This has to do with how these logical operators work
Let's say we have two logical propositions
Each of which can be true or false
To see how the value of each proposition effects the result ...
we draw a truth table
The value the first proposition, p, appears in the first column
The value of the second proposition, q, appears in the second column

And the last column contains the result when the operator is used

p	q	p AND q
true	true	true
true	false	false
false	true	false
false	false	false

The only way that p and q can be true
Is if both p and q are true
This means that if p is false
There is no way that p and q can be true
In such situations, many computer languages use something called short circuit evaluation
Here is how this works in Unix
If two Unix commands are separated by && ...
and running the first commands gives a non-zero status code ...
the second command will not be run
So if we write
```
$ cd dir1 && rm *
```
Unix will not run rm if cd fails
We can use this command separator when we want to run the second command ...
only if the first command succeeds
A similar situation applies to the || command separator

The truth table for OR looks like this

p	q	p OR q
true	true	true
true	false	true
false	true	true
false	false	false

If the first value is true, p OR q must be true
If we separate two commands with ||
The second command will only run
If the first command fails
I use this feature in all my test scripts

I will write command like

grep ls $script  || echo ERROR: Could not find script

The second command that prints the error message will only run ...
If grep cannot find what it is looking for

An Introduction to AWK

AWK is a pattern-scanning and processing language
It searches one or more files for specific patterns
And then does something if a match is found
AWK is actually a programming language
Though it is very different from other programming languages
Languages such as Java, C and Python are procedural languages
AWK is a data-driven language
AWK can be used to generate reports or filter text
AWK gets its name from the computer scientists who created it
Alfred V. Aho, Peter J. Weinberger and Brian W. Kernighan
The GNU project has created an interpreter for the AWK language
called gawk
This is what Ubuntu uses
But you can also use awk at the command line

Due to some clever aliasing

$ which awk
/usr/bin/awk

$ ls -l /usr/bin/awk
lrwxrwxrwx 1 root root 21 Jul 24 12:12 /usr/bin/awk -> /etc/alternatives/awk

$ ls -l /etc/alternatives/awk
lrwxrwxrwx 1 root root 13 Jul 24 12:26 /etc/alternatives/awk -> /usr/bin/gawk

Running `awk`

There are two ways to run an AWK program
- Enter the AWK program on the command line
- Run the AWK program from a file
You run an AWK program from the command line like this
```
awk [OPTIONS] [PROGRAM] [FILE_LIST]
```
Notice that everything is optional
The PROGRAM consists of a series of AWK program lines
If no file is specified, input is taken from standard input
To run an AWK program in a file we use this format
```
awk [OPTIONS] -f PROGRAM_FILE [FILE_LIST]
```
When run with the -f option
awk will read the program from the file specified
If the AWK program file has a hashbang line
```
#! /usr/bin/awk
```
And is executable
You can run the program from the command line by typing its name

AWK Syntax

The syntax of an AWK program line is simple
```
PATTERN '{ ACTION }'
```
awk understands this line to mean the following
For every line in the file that matches PATTERN
Perform ACTION
There are five types of patterns
- Blank
- Regular expressions
- Special patterns
- Comparisons
- Ranges

Blank Pattern

Blank patterns match all lines in the files

If we have the following file

$ cat cars.txt 
plym    fury    1970    73  2500
chevy   malibu  1999    60  3000
ford    mustang 1965    45  10000
volvo   s80     1998    102 9850
ford    thundbd 2003    15  10500
chevy   malibu  2000    50  3500
bmw     325i    1985    115 450
honda   accord  2001    30  6000
ford    taurus  2004    10  17000
toyota  rav4    2002    180 750
chevy   impala  1985    85  1550
ford    explor  2003    25  9500

We can use the blank pattern to print the entire file

$ awk '{print}' cars.txt 
plymt   fury    1970    73  2500
chevy   malibu  1999    60  3000
ford    mustang 1965    45  10000
volvo   s80     1998    102 9850
ford    thundbd 2003    15  10500
chevy   malibu  2000    50  3500
bmw     325i    1985    115 450
honda   accord  2001    30  6000
ford    taurus  2004    10  17000
toyota  rav4    2002    180 750
chevy   impala  1985    85  1550
ford    explor  2003    25  9500

The AWK program line must be contained within single quotes
Since no pattern was contained in the program above ...
all lines in the file were matched ..
so all lines were printed

Regular Expression Patterns

Regulars expression are patterns enclosed within forward slashes, /
The patterns are written using ordinary characters and meta-characters

Here is how we can find all lines that contain "ord"

$ awk '/ord/ {print}' cars.txt
ford    mustang 1965    45  10000
ford    thundbd 2003    15  10500
honda   accord  2001    30  6000
ford    taurus  2004    10  17000
ford    explor  2003    25  9500

The regular expression meta-characters work a little differently ...
than they do in Unix
They also use additional meta-characters
For example ^ matches the beginning of a line

To find all cars made by Ford we could use

$ awk '/^f/ {print}' cars.txt 
ford    mustang 1965    45  10000
ford    thundbd 2003    15  10500
ford    taurus  2004    10  17000
ford    explor  2003    25  9500

Regular expressions are very powerful
But it takes some time to learn how to use them properly

BEGIN and END Patterns

Any action following the BEGIN pattern ...
is executed before any lines in the file are matched
Any action following the END pattern ...
is executed after lines in the file are matched

Here is an example

$ cat begin_end.awk
BEGIN {print "About to start processing lines"}
/ord/ {print}
END   {print "No more lines to process"}

$ awk -f begin_end.awk cars.txt
About to start processing lines
ford    mustang 1965    45  10000
ford    thundbd 2003    15  10500
honda   accord  2001    30  6000
ford    taurus  2004    10  17000
ford    explor  2003    25  9500
No more lines to process

Comparison Patterns

The comparison pattern uses comparison operators
And match any line where the expression is true

We can use this to find all lines with more than 25 characters

$ awk 'length > 25 {print}' cars.txt 
ford    mustang 1965    45  10000
ford    thundbd 2003    15  10500

When the length function is called with no arguments ...
it returns the number of characters on the line ...
including the spaces that separate the individual fields of the file

Range Patterns

Two AWK patterns separated by a comma form a range
To print all lines in cars.txt between "bmw" and "ford"

We can use the following

s$ awk '/bmw/, /ford/ {print}' cars.txt 
bmw 325i    1985    115 450
honda   accord  2001    30  6000
ford    taurus  2004    10  17000

If no line matches the expression after the comma ...

all lines after the first selected line are printed

$ awk '/bmw/, /xxx/ {print}' cars.txt
bmw     325i    1985    115 450
honda   accord  2001    30  6000
ford    taurus  2004    10  17000
toyota  rav4    2002    180 750
chevy   impala  1985    85  1550
ford    explor  2003    25  9500

We can also use boolean, or conditional, expressions in a range

$ awk 'length == 23, length == 24 { print }' cars.txt 
volvo   s80         1998    102     9850
ford    thundbd     2003    15      10500
chevy   malibu      2000    50      3500
bmw     325i        1985    115     450
honda   accord      2001    30      6000
ford    taurus      2004    10      17000
toyota  rav4        2002    180     750

AWK Fields

Perhaps the most useful AWK feature, to a casual user, are the fields
Fields are chunks of text separated by whitespace
The file cars.txt contains four fields
- Manufacturer
- Model
- Year
- Mileage in thousands of miles
- Price
To refer to the value of a field
You put a $ in front of the field number

So if I wanted to see all the models in cars I would write

$ awk '{print  $2 }' cars.txt 
fury
malibu
mustang
s80
thundbd
malibu
325i
accord
taurus
rav4
impala
explor

This is the only feature of AWK that I use
Let me show you how
There is a server for a non-profit that I used to monitor
I needed to check disc usage

But the output of df is a little hard to read

$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/sda1                         12G  3.5G  7.7G  32% /
none                             4.0K     0  4.0K   0% /sys/fs/cgroup
udev                             2.0G  4.0K  2.0G   1% /dev
tmpfs                            396M  536K  395M   1% /run
none                             5.0M     0  5.0M   0% /run/lock
none                             2.0G     0  2.0G   0% /run/shm
none                             100M     0  100M   0% /run/user
blade82:/disk/sd0h/home/ghoffman 7.9G  4.5G  3.4G  58% /home/ghoffman
mx1:/disk/sd1e/spool/mail        4.0G  2.9G  1.1G  73% /spool/mail

All I wanted was the date, the space used and the percent

I can do this with the following script

$ cat check_space.sh
#! /bin/bash
#
#  returns disc usages statistics

date=$(date +%Y-%m-%d)
space_avail=$(df -h | grep /dev/sda1 | awk '{print $3}')
percent=$(df -h | grep /dev/sda1 | awk '{print $4}')
echo "$date  $space_avail  $percent "

$ ./check_space.sh
2014-12-08  7.7G  32%

I use AWK to cherry pick just the fields I want

Class Exercise 27

Today's Class Exercise is unusual.

I am asking you to recreate the script I used to test Class Exercise 25.

Let me explain how this script works.

Contract for the Web

Tim Berners-Lee is the man who created HTML
HTML stands for Hypertext Markup Language
It is the format that is used to create web pages
So you might say that Tim Berners-Lee is the man who invented the World Wide Web
He created the web to allow the free exchange of information
In the early days of the web it looked like something that would change the world for the better
Now it is debatable whether the web does more harm than good
Bigots of various sorts use the web and its platforms to spread hatred
Foreign governments spread lies to undermine our democracy
Crackpot ideas proliferate
- The earth is "flat"
- Vaccines cause autism
All this has been hard for Tim Berner-Lee to watch
Rescue the web he has launches an initiative called the Contract for the Web
It is the result of many months of consultation with many people in government, academia, and not-profits
Here are it's fundamental principles
- Everyone can connect to the Internet
- Keep all of the Internet available, all of the time
- Respect and protect people's fundamental online privacy and data rights
- Make the Internet affordable and accessible to everyone
- Respect and protect people's privacy and personal data to build online trust
- Develop technologies that support the best in humanity and challenge the worst
- Be creators and collaborators on the Web
- Build strong communities that respect civil discourse and human dignity
- Fight for the Web
Tim has asked all of us to endorse the Contract
I have done this on the Contract Web site
I would urge all of you to do so too
Rescuing the web from it's most toxic elements will take some time
But all of us must do what we can to make the web a better place
If you are not part of the solution, you are part of the problem

Review

New Material

Final Exam

Course Evaluation

Today's Class

Questions

Review

Using Braces, { } , with Variables

Special Parameters

$ - The PID of the Current Shell Process

! - The PID of the Last Process Put into the Background

? - The Exit Status

Positional Parameters

# - The Number of Command Line Arguments

0 - The Pathname of the Script

1 - n - The Command Line Arguments

Attendance

New Material

Expanding Null and Unset Variables

:- Use a Default Value

:= Assign a Default Value

:? Print an Error Message

The [[ ]] Command

Who Uses These Features?

&& and || as Command Separators

An Introduction to AWK

Running awk

AWK Syntax

Blank Pattern

Regular Expression Patterns

BEGIN and END Patterns

Comparison Patterns

Range Patterns

AWK Fields

Class Exercise 27

Contract for the Web

Class Exercise

Running `awk`