IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 21

Microphone

Class Quiz

You can connect to Gradescope to take weekly graded quiz today during the last 15 minutes of the class.

Once you start the quiz you have 15 minutes to finish it.

You can only take this quiz today.

There is not makeup for the weekly quiz because Gradescope does not permit it.

Homework 10

Homework 10 is posted here.

It is due this coming Sunday at 11:59 PM.

Questions

Are there any questions before I begin?

Tips and Examples

Don't Use Commands You Copied from Google

I sometimes see students using commands that they must have copied from Google
I can tell, because they use command I have not taught
Sometimes they find an alternate way of doing what I ask that works
Just as often it does not work and students lose points
When I see this, I know that students have been lazy
All problems in this course can be solved using what I have taught you
To complete the homework assignments for this course ...
you only need to read the Class Notes
But some students are lazy
Rather than reading the Class Notes they go to Google
But what you get from a Google search may not be what is needed
Worse, it may give you commands you do not understand
You should never use commands you do not understand
Working in Unix is not magix ...
where you utter a magic incantation
Working in Unix involves understanding what you are trying to do ...
and the tools you have to make it happen
Googling is no substitute for thinking
Students would be better off if they simply took the time ...
to read the Class Notes

An Interesting Use of the `source` Built-in

The source command is a built-in ...
that runs a bash script inside your current shell
If you change a startup file ...
and want the changes visible in your current shell ...
you simply run
```
source ~.bash_profile
```
The other day I encountered problem that was best handled by using source
It my IT 116 class I wrote a homework assignment ...

where students had to create a conversion table like this

Fahrenheit	Kelvin
----------------------
20		266
21		267
22		268
23		268
24		269
25		269

Students were told to make sure the numbers aligned with the column labels
I wrote a script that runs every student script ...
to test for this alignment
But it did not work
The output of the script showed the numbers not aligned
But when I ran the scripts individually they were OK
Since a script wouldn't work I decided to try a function
For some reason the function worked where the script did not

Here is text of the function

h7 () 
{ 
    for id in $s1cl;
    do
        echo $id;
        python3 $id/hw7.py < <(echo 20; echo 25; echo 260; echo 265);
        read -p "Next? ";
    done
}

It uses loops which we will talk about soon
This problem had been bugging me for some time
So I wanted to be sure to document this fix for next semester
But how?
I decided the the simplest way was to create the text file h7.sh ...
that had the function definition I would normally type on the command line

Here is the text of the file

cat h7.sh
h7 () 
{ 
    for id in $s1cl;
    do
        echo $id;
        python3 $id/hw7.py < <(echo 20; echo 25; echo 260; echo 265);
        read -p "Next? ";
    done
}

Where does source come into this story?
Next semester when I need to have this function available ...
I simply go to the directory where this file lives ...
and run
```
source h7.sh
```
This will load the function into my current shell ...
where I can use it to score the assignment

Review

History

The history mechanism is what lets you use up arrow, ↑, ...
to run a previous command
The history mechanism maintains a list of the command lines you type
These command line entries are called events

To view the history list, use the history command

$ history
  2  exit
  3  cd
  4  cd it244/work
  5  pwd
  6  rm -rf *
  7  cd ~/it244/work
  8  pwd
  9  cp ~ghoffman/examples_it244/bother.sh .
 10  ls /home/ghoffman/examples_it244
 11  cp ~ghoffman/examples_it244/bother.sh .
 12  ./bother.sh
 13  ./bother.sh &
 14  jobs
...

If you run history without an argument it shows all events
By default, this list contains 500 values
Running history followed by a number ...

will limit the events shown

 $ history 10
  498  ps
  499  exit
  500  exit
  501  history
  502  cd
  503  cd it244
  504  cd work
  505  ls
  506  history
  507  history 10

Notice that there is no - in front of the number

Using the History Mechanism

You can rerun a previous command using an exclamation mark, ! ...

in front of an event number

$ !517
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Notice that this prints out the old command line before running it
There must be no space between the ! and the number or you will get an error
```
$ ! 517
517: command not found
```
You can follow ! with a letter

Which will return the last command beginning with that letter

$ !e
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Using `fc` to Edit and Run an Old Command

The utility fc (fix command) allows you to edit a previous command line ...
and then run it
fc is useful when you want to run a previous command line entry ...
but need to make some modifications
fc is a built-in, so it executes quickly
When run with no arguments fc will bring up an editor window ...
contain your last command line
You can then
- Modify the command in the editor window
- Save your changes
- Execute the modified command when you quit
Running fc with an event number will put that command in the editor window
fc can also be used to view the history list
When run with the -l option, fc will list the last 16 command lines
You can also tell fc to list all command lines ...
starting with a certain event number

You do this by running fc -l followed by a space and a number

$ fc -l 522
522  echo $PATH
523  traceroute -a standford.edu
524  echo $PATH
525  echo $PATH
526  echo $PATH
527  fc -l

You can also use fc -l to specify a range of events

To do this follow fc -l with two numbers

$ fc -l 522 525
522  echo $PATH
523  traceroute -a standford.edu
524  echo $PATH
525  echo $PATH

The Readline Library

The Readline library is a collection of procedures written in the C programming language
The Readline library was created by the GNU project
When you do command line editing ...
you are using the Readline library
Any program running under Bash and written in C can use the Readline library
There are two modes available in the readline library
- emacs mode
- vim mode
pe15 is configured to use emacs mode by default
That is the mode we have been using

Here are some of the more useful commands for the emacs version of the readline library

Command	Meaning
Control A	Move to the beginning of the line
Control E	Move to the end of the line
Control U	Remove everything from the text entry point to the beginning of the line
Control K	Remove everything from the text entry point to the end of the line
→	Move the text entry point one character to the right
←	Move the text entry point one character to the left
↑	Recall the previous command line entry in the history list
↓	Recall the following command line entry in the history list

Readline Completion

The Readline library provides a completion mechanism
Type a few letters and hit Tab ...
and Readline completion will try to supply the rest
There are three forms of completion provided by the readline library

Pathname Completion

Type the first few letters of a pathname ...
then hit Tab
The Readline Library will finish as much of the pathname as it can
If it cannot get the full pathname ...
hit Tab again to see the possible completions
```
$ ls hw[Tab][Tab]
hw2/ hw4/ hw5/ hw6/
```

Command Completion

The Readline library will complete the name of a command for you
Begin typing a command then hit Tab
If there is more than one possible completion ...
you will hear a beep
If you hit Tab a second time ...
you will see a list of possible completions

Variable Completion

The Readline library will also complete variable names when using their value
It knows when you are using a variable ...
when you type a dollar sign, $
Type a few characters of the variable name and hit Tab
The Readline library will try to complete the variable name

Aliases

An alias is a couple of characters that Bash replaces with a command
They are shortcuts to run a command
To define an alias, you use the alias command
alias uses the following format in Bash
```
alias ALIAS_NAME=ALIAS_VALUE
```
There must be no spaces on either side of the equal sign, =
If the value assigned to the alias has spaces ...
it must be quoted
```
alias  la='ls -a'
```
If you run alias with no arguments ...

it will list all aliases currently defined

$ alias
alias bin='pu $bin'
alias binl='ls $bin'
alias ck755='ls -l *.sh | tr '\''-'\'' '\'' '\'' | grep '\''rwxr xr x'\'''
...

If you follow alias with the name of an alias ...
it will print the definition
```
$ alias ll
alias ll='ls -l'
```
In Bash, an alias cannot use positional parameters

But you can follow the alias with an argument

$ ll agrs.sh
total 4.0K
-rwxr-xr-x 1 ghoffman faculty 70 Apr  8 14:12 agrs.sh

Instead of allowing aliases to accept arguments Bash has functions
Functions in Bash can consist of many commands
And you can use positional parameters with each of these commands
Aliases are not global
They only work in the shell in which they are defined

Single Quotes Versus Double Quotes in Aliases

There are two types of quotes in Bash
- Single quotes - ' '
- Double quotes - " "
You must use quotes, or a backslash, \ when the value has whitespace
The whitespace characters are
- Spaces
- Tabs
- Newlines (carriage returns)
Single quotes turn off all special meanings of characters
```
$ echo 'My name is $name'
My name is $name
```
Variables are not evaluated when they are enclosed in single quotes
But double quotes allow you to use get the value of a variable using $
```
$ echo "My name is $name"
My name is Glenn Hoffman
```
Usually you want to use single quotes, ', when defining an alias

Functions

A function is a collection of commands that is given a name
Functions can be run anywhere you happen to be in the filesystem
Because functions are stored in the process RAM of your shell ...
not on the disk ...
they run quickly since no subprocess must be created
Functions, unlike aliases, can use command line arguments
They use the same positional arguments that shell scripts use
Functions differ from shell scripts in a number of ways
- They are stored in memory (RAM), rather than in a file on disk
- The shell preprocesses the function so it can execute more quickly
- The shell executes the function in it's own process
So functions run much faster than shell scripts
Functions are local to the shell in which they are defined
They are local, not global ...
so they do not work in sub-shells

Functions definitions have the following form

FUNCTION_NAME ()
{
    COMMAND
    COMMAND
    ...
}

Once you type the final } the definition is complete

You can define a function on a single command line

$ echo3 () { echo $1; echo $1; echo $1;} 

$ echo3 foo
foo
foo
foo

Functions, like aliases, only work in the shell in which they are defined

Attendance

New Material

Bash Features and Options

Bash is a program you can run at the command line ...
just like other programs such as ls or cat
And just like those other programs, Bash has options
These options change the way that bash operates when it runs
You can give Bash an option on the command line ...
just like you do when you run ls -l

You can see these options if you run bash --help

$ bash --help
GNU bash, version 5.0.17(1)-release-(x86_64-pc-linux-gnu)
Usage:	bash [GNU long option] [option] ...
	bash [GNU long option] [option] script-file ...
GNU long options:
	--debug
	--debugger
	--dump-po-strings
	--dump-strings
	--help
	--init-file
	--login
	--noediting
	--noprofile
	--norc
	--posix
	--pretty-print
	--rcfile
	--restricted
	--verbose
	--version
Shell options:
	-ilrsD or -c command or -O shopt_option		(invocation only)
	-abefhkmnptuvxBCHP or -o option
Type `bash -c "help set"' for more information about shell options.
Type `bash -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.

bash home page: <http://www.gnu.org/software/bash>
General help using GNU software: <http://www.gnu.org/gethelp/>

Options used on the command line ...
are only in effect while that instance of Bash is running
You to make an option work in your current your login shell
You have to call set
set is a builtin
So you cannot call in in your .bash_profile
The single letter options to set can be found by running help set
There is only 1 bash option that I have every used
You can use bash -x to run a script
If you do, Bash will print each line of the script ...
before executing that command

Here is an example

$ cat find_scripts.sh
#! /bin/bash

cd $1
ls -1 | grep .py

$ bash -x find_scripts.sh $s1fl
+ cd /home/ghoffman/course_files/it116_files
+ ls -1
+ grep .py
anon_awards.py
average.py
cheer_default.py
...

The + is used to distinguish the command line ...
from its output
Shell features and options will not be on the final

Processing the Command Line

The shell can modify what you enter at the command line
It does this to provide features like aliases
To do this properly the shell must modify the command line in a specific order
Otherwise, things could become terribly confused
There are 10 different ways in which the shell can change the command line
The order in which the shell performs them is as follows
1. History Expansion
2. Alias Substitution
3. Brace Expansion
4. Title, ~, Expansion
5. Parameter and Variable Expansion
6. Arithmetic expansion
7. Command substitution
8. Word splitting
9. Pathname expansion
10. Process substitution

History Expansion

The first substitution Bash performs is history expansion
This feature allows you to rerun an old command using the event number

Simply type an exclamation mark, ! followed by the event number

$ history 5
  540  cat output.txt 
  541  echo "Go Red Sox" > output.txt
  542  cat output.txt 
  543  echo foo
  544  history 5

$ !543
echo foo
foo

Alias Substitution

After history expansion, Bash performs alias substitution

$ alias ll='ls -l'

$ ll
total 2
lrwxrwxrwx 1 it244gh man    34 Sep  6 21:09 it244 -> /courses/it244/s19/ghoffman/it244gh
drwxr-xr-x 2 it244gh ugrad 512 Oct 27 09:16 work

The shell replaces the alias with the command assigned to it

Brace Expansion

After alias substitution, Bash performs brace expansion
This feature allows you to create several strings all at once ...
using a pattern
The pattern is formed by typing some initial characters ...
then putting a number of different characters inside braces ...
followed by some additional characters
The braces contain strings of characters separated by commas
The shell expands a brace by creating multiple strings
One for each string contained inside the braces

If I wanted to create "foo" files numbered 1 to 5, I would write

$ touch foo{1,2,3,4,5}.txt

$ ls
foo1.txt  foo2.txt  foo3.txt  foo4.txt  foo5.txt

The shell expanded the braces to create one command line argument ...
for each string inside the { }
The strings inside the braces can contain one or more characters
But each string must be separated from the others by a comma
```
$ touch {a,ab,abc}.txt

$ ls
abc.txt  ab.txt  a.txt
```
There should not be any unquoted spaces or tabs within the braces

If there is, the expansion will not work properly

$ touch {b , bc, b c d}.txt

$ ls -l
total 0
-rw-r--r-- 1 it244gh ugrad 0 Nov 14 10:37 ,
-rw-r--r-- 1 it244gh ugrad 0 Nov 14 10:37 b
-rw-r--r-- 1 it244gh ugrad 0 Nov 14 10:37 {b
-rw-r--r-- 1 it244gh ugrad 0 Nov 14 10:37 bc,
-rw-r--r-- 1 it244gh ugrad 0 Nov 14 10:37 c
-rw-r--r-- 1 it244gh ugrad 0 Nov 14 10:37 d}.txt

Tilde, ~, Expansion

After brace expansion, Bash performs tilde expansion
Bash replaces a tilde, ~ ...
with the the absolute pathname of a home directory
If ~ has nothing after it ...
it is replaced with the absolute pathname of the user's home directory
```
$ echo ~
/home/ghoffman
```
If it is followed by a username ...
it is replaced with the absolute pathname of that user's home directory
```
$ echo ~it244gh
/home/it244gh
```
If there is no matching username, nothing happens
No expansion is performed
```
$ echo ~xxx
~xxx
```
There are two other tilde expansions
- ~+
- ~-
Bash replaces ~+ with the pathname of the current directory
```
$ pwd
/home/it244gh/work

$ echo ~+
/home/it244gh/work
```

Bash replaces ~- with the pathname of the previous directory

$ pwd
/home/it244gh/work

$ cd

$ pwd
/home/it244gh

$ echo ~-
/home/it244gh/work

Parameter and Variable Expansion

After tilde expansion ...
Bash performs parameter and variable expansion
```
$ echo $SHELL
/bin/bash

$ echo $?
0
```
Notice that this expansion comes after alias expansion
So you can use variables and parameters when defining aliases

Arithmetic Expansion

After parameter and variable expansion ...
Bash performs arithmetic expansion
Unix treats everything on the command line as text ...
unless told otherwise
```
$ echo 5 + 4
5 + 4
```
Arithmetic expansion allows Bash to interpret characters as numbers
And to perform ordinary arithmetic upon them
But it does more than that
Bash treats everything inside $(( )) as an arithmetic expression
It performs the calculation indicated
Then it replaces the entire construct with the number calculated
```
$ echo $(( 5 + 4 ))
9
```
The rules for evaluating arithmetic expressions are the same as for C
They are mostly what you would expect

You can use variables within an arithmetic expression

$ a=5

$ b=3

$ echo $a $b
5 3

$ echo $(( $a - $b ))
2

Arithmetic expansion recognizes the following operators

Operator	Description
+	Addition
-	Subtraction
*	Multiplication
/	Division
%	Remainder
**	Exponentiation
==	Equality
!=	Inequality
>	Greater than
>=	Greater than or equal
<	Less than
<=	Less than or equal
\|\|	Logical OR
&&	Logical AND
!	Logical N0T
=	Assignment
++	Increment
--	Decrement

You do not have to use a $ to get the value of a variable
```
$ echo $a $b
5 3

$ echo $(( a * b ))
15
```

Command Substitution

After arithmetic expansion, Bash performs command substitution
Command substitution uses the following format
```
$(COMMANDS)
```
Where COMMANDS are any valid Unix commands
The commands inside the ( ) are run in a subshell
The output of that command replaces the construction on the command line
If I wanted to set a variable to the current time and date

I could use

$ today=$(date)

$ echo $today
Tue Oct 25 17:00:07 EDT 2011

There is an alternate format for command substitution
You can place the command within back tics, ` `

Like this

$ ls -l `which bash`
-rwxr-xr-x 1 root root 954896 2011-03-31 17:20 /bin/bash

In this command line, Bash first runs
```
which bash
```
In a subshell
Then it replaces the construction ...
with the output of which
ls can now take /bin/bash as its argument

Command substitution can be used inside double quotes

$ echo "Today is $(date +'%A, %B %d, %Y')"
Today is Wednesday, November 13, 2013

You type back tic by pressing the ~ key
The back tics do not work in the TC shell
I do not use back tics
Because they are easily mistaken for the single quotes, '

Word Splitting

After command substitution, Bash performs word splitting
Word splitting is parsing the command line into tokens
Tokens are strings of characters usually separated by spaces
I have found a small script which demonstrates word splitting

By showing all command arguments

$ args.sh foo bar bletch 
3 args: [foo] [bar] [bletch]

Word splitting does not work when strings are quoted

 args.sh 'foo bar bletch' 
1 args: [foo bar bletch]

$ args.sh "foo bar bletch"
1 args: [foo bar bletch]

But it does work with command substitution

$ args.sh $(date)
6 args: [Wed] [Apr] [12] [16:19:17] [EDT] [2017]

Or with variable substitution

$ cheer="Let's go Red Sox!"

$ args.sh $cheer
4 args: [Let's] [go] [Red] [Sox!]

Sometimes you want another character other than space ...
to separate tokens
You can do this using the Unix keyword variable IFS
IFS stands for Internal Field Separator
If you give IFS a value
Word splitting will use this character to parse tokens

Look what happens when I set IFS to :

$ args.sh $PATH
1 args: [/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/oracle/12.1/client64/bin:/home/ghoffman/bin/shell:/home/ghoffman/bin/python/umb:/home/ghoffman/bin:/home/ghoffman/bin/shell:/home/ghoffman/bin/python/umb]

$ IFS=:

$ args.sh $PATH
15 args: [/usr/local/sbin] [/usr/local/bin] [/usr/sbin] [/usr/bin] [/sbin] [/bin] [/usr/games] [/usr/local/games] [/snap/bin] [/usr/lib/oracle/12.1/client64/bin] [/home/ghoffman/bin/shell] [/home/ghoffman/bin/python/umb] [/home/ghoffman/bin] [/home/ghoffman/bin/shell] [/home/ghoffman/bin/python/umb]

Word splitting will not be on the final

Pathname Expansion

After word splitting, Bash performs pathname expansion
This is also known as globbing
Pathname expansion is where you use meta-characters ...
to specify one or more pathnames
The metacharacters are used to create patterns
The patterns are called ambiguous file references
The metacharacters are
- ?
- *
- [ ]

Here are some examples

$ ls t*
test1.txt  test2.txt  test3.txt

$ echo t*
test1.txt test2.txt test3.txt

Process Substitution

After pathname expansion, Bash performs process substitution
Process substitution allows you to create a file on the fly
A command is run in a sub-shell ...
and the output of the command is treated like a file
This "file" can be used as an argument on the command line
Process substitution uses the following format
```
<(COMMAND)
```
The output of COMMAND is placed in a Unix structure called a named pipe

We can use process substitution to compare the contents of two directories

$ diff -y <(ls -1 tia777/ce) <(ls -1 jgreen/ce)
ce1                             ce1
ce10                            ce10
ce11                                  <
ce2                             ce2
ce3                             ce3
ce4                             ce4
ce5                             ce5
ce6                             ce6
ce7                             ce7
ce8                             ce8
ce9                             ce9

Here we have two ls commands each running in their own subshell
We can then run diff to look for differences in these two "files"
Process substitution will not be on the final

Tips and Examples

Review

New Material

Class Quiz

Homework 10

Questions

Tips and Examples

Don't Use Commands You Copied from Google

An Interesting Use of the source Built-in

Review

History

Using the History Mechanism

Using fc to Edit and Run an Old Command

The Readline Library

Readline Completion

Pathname Completion

Command Completion

Variable Completion

Aliases

Single Quotes Versus Double Quotes in Aliases

Functions

Attendance

New Material

Bash Features and Options

Processing the Command Line

History Expansion

Alias Substitution

Brace Expansion

Tilde, ~, Expansion

Parameter and Variable Expansion

Arithmetic Expansion

Command Substitution

Word Splitting

Pathname Expansion

Process Substitution

Class Exercise

Class Quiz

An Interesting Use of the `source` Built-in

Using `fc` to Edit and Run an Old Command