Review

• Running a Command in the Background
• Jobs
• Moving a Job from the Foreground into the Background
• Aborting a Background Job
• Pathname Expansion
• The ? Meta-character
• The * Meta-character
• The [ and ] Meta-characters
• Built-ins
• Creating Startup Files
• Redirecting Standard Error
• Shell Scripts
• Making a Shell Script Executable
• Specifying Which Version of the Shell Will Run a Script
• Comments in Shell Scripts
• Separating and Grouping Commands
• Continuing a Command onto the Next Line
• Shell Variables
• Local Variables
• Global Variables
• Keyword Shell Variables
• User-created Variables
• Positional and Special Parameters
• Quoting and the Evaluation of Variables
• Processes
• Process Structure
• Process Identification
• Executing a Command
• The Readline Library
• Readline Completion
• Aliases
• Functions
• Shell Modification of the Command Line
• Alias Substitution
• Brace Expansion
• Tilde, ~, Expansion
• Parameter and Variable Expansion
• Command Substitution
• Pathname Expansion
• Shell Script Control Structures
• The if ... then ... Construct
• test
• The test Operators
• Using [ ]
• The if ... then ... else ... Construct
• The if ... then ... elif ... Construct
• for ... in ... Loops
• for Loops
• while Loops
• continue
• break
• read Command

Final Exam

The final exam will be held on Tuesday, May 14th from 3:00 - 6:00 PM.

The exam will be given in this room.

If for some reason you are not able to take the Final at the time it will be offered, you MUST send an email to me before the exam so we can make alternative arrangements.

60% of the points on this exam will consist of questions from the Ungraded Class Quizzes.

You do not need to study a Class Quiz question if the topic is not mentioned in either the Midterm or Final review.

The other questions I will make up specifically for this exam.

For these questions you will have to know

• Absolute and relative pathnames
• The PATH system variable
• Access permissions
• Redirection
• Metacharacters
• Utilities

Today's will be a review session.

You will only be responsible for the material in today's class and the review for the Midterm, which you will find here.

Although the time alloted for the exam is 3 hours, I would expect that most of you would not need that much time.

The final is a closed book exam.

To prevent cheating, certain rules will be enforced during the exam.

Remember, the Midterm and Final determine 50% of your grade.

Course Evaluation

At the end of each semester we offer you the opportunity to say what you think about this course.

What have I done right?

What have I done wrong?

What can I do better?

You are not asked for you name.

So the submissions are anonymous.

I will not see your responses until after I have submitted grades for this course.

We collect this feedback through Course Evaluations.

I will use what you say to make this course better.

To complete the course evaluation, use the following link .

You have until Friday, May 24th, to submit the evaluation.

No Class Next Week

A normal semester has 28 classes.

But this semester has 29.

I do not have materials for a 29th class.

Next Tuesday I will have Zoom Office Hours, from 9 AM to 5 PM,instead of a class.

The Zoom session will be conducted from my home in Somerville.

Not my office on the UMB campus.

Questions

Are there any questions before I begin?

Attendance

Review

Running a Command in the Background

• Normally, when you run a command you have to wait for it to finish
• Such commands are said to be running in the foreground
• Unix gives you a way to get the command prompt back right away
• You can run the command in the background
• The background job loses it's connection to the keyboard ...
• and the shell will give you a prompt immedidately
• The shell will tell you when the background job has finished
• Every time a program runs, a process is created
• The process has access to system resources
• Such as memory (RAM) and a connection to the filesystem
• Unix, like most OSs, is a multitasking operating system
• This means you can have more than one process running at a time
• To run a command in the background ...
• type an ampersand, & , at the end of the command line

  $ sleep 5 &
  [1] 17895
  $

• The number in square brackets is the job number
• The second number is the process ID

Jobs

• Every time you type some commands and hit Enter you are creating a job
• A job can be a single command
• A pipeline is also a job
• A pipeline is a collection of commands joined by pipes
• Each command will generate its own process
• But the collection of all the separate processes is a single job
• Each process in a pipeline will have its own process ID
• So as the pipeline progresses, the currently running process will change
• But the job number does not change
• The job is the collection of all processes created by an entry at the command line
• You can have multiple jobs running at the same time
• But only one job can be in the foreground at any one time
• Every process has a process ID number
• And every job has a job number
• When you tell the shell to run a job in the background ...
• it returns two numbers

  $ sleep 5 &
  [1] 7431
  $

• The job number is enclosed in brackets and comes first
• The second, larger, number is a the process identification number
• If there is more than one process in the job
• It is the process ID of the first one
• The process identification number is also known as the PID
• When the job finishes, the shell prints out a message

  [1]+  Done                  sleep 5

• The message does not appear the moment the job finishes
• The shell waits for the next time you hit Enter
• Then it prints the message after the output from the command
• To see the status of all background jobs run the jobs command

  $ sleep 10 & sleep 10 & sleep 10 &
  [1] 13256
  [2] 13257
  [3] 13258
  
  $ jobs
  [1]   Running                 sleep 10 &
  [2]-  Running                 sleep 10 &
  [3]+  Running                 sleep 10 &

Moving a Job from the Foreground into the Background

• There can only be one foreground job
• But there can be many background jobs
• Unix will let you move a job from the foreground to the background
• To do this, you must first suspend the foreground job
• A suspended job is not dead
• It is in a state of suspended animation
• You can reactivate it later
• To suspend a foreground job you must use the suspend key sequence
• On our systems you suspend a job by hitting Control Z
• After you do this, the shell stops the current process
• It also disconnects it from the keyboard
• To make a suspended job run in the background use bg
• bg stands for background
• Once placed in the background, the job resumes running
• If more than one job is suspended you must give bg the job number

Aborting a Background Job

• There are two ways to abort a background job
• You can bring a background job to the foreground ...
• using the fg (foreground) command
• Once you have the job in the foreground
• You can abort it using Control C
• If there is only one background job
• Run fg without an argument
• If there is more than one background job ...
• fg needs the job number as an argument
• You can also terminate any job using the kill command
• But to use kill you must tell it what to kill
• The usual way to do this is to give kill a process ID
• You can also use the job number with kill
• But you must precede a job number with a percent sign, %
• You can get the job number by using the jobs command

Pathname Expansion

• Pathname expansion expands a few letters into an entire pathname
• It also allows you to specify more than one file or directory
• Pathname expansion uses characters with special meaning to the shell
• These special characters are called meta-characters
• Meta-characters are also sometimes called wildcards
• They allow you to specify a pattern
• The pattern is called an ambiguous file reference
• When you hit Enter the shell replaces the pattern with pathnames that match
• The shell then runs this altered command line
• You can use as many meta-characters as you want to form a pattern
• Pathname expansion is different from pathname completion
• Which you get by hitting Tab

The ? Meta-character

• The ? meta-character stands for a single instance ...
• of any character
• ? can be used with any command
• Even those that don't normally deal with files

  $ echo dir?
  dir1 dir2 dir3 dir4

• ? does not match a leading period in a filename
• You must explicitly enter a leading period, . 
• When specifying an "invisible" file

The * Meta-character

• * will match any number of characters in a pathname
• It will even match no characters
• * can be used with any command
• Even those that don't normally deal with files

  $ echo dir*
  dir dir1 dir10 dir2 dir3 dir4

• * cannot be used to match the initial period, . ...
• in a "hidden" filename
• A . (dot) followed by a * ...
• will match all hidden files and directories

The [ and ] Meta-characters

• The square brackets, [ and ], are also meta-characters
• They work somewhat like ?
• They only match a single character in a pathname
• But the character must match one of the characters within the brackets
• You can use the bracket meta-characters with any program
• You can use a range to avoid listing all characters
• A range consist of first and last characters separated by a dash, -
• The sequence is specified in alphabetical order
• The square brackets provide another shortcut
• An initial exclamation mark, ! or a caret, ^ ...
• changes the meaning of the square bracket
• Now it matches any character that is not included within the brackets

Built-ins

• Not all commands can be found on disk as executable files
• Some are actually written as part of the shell
• Such commands are called built-ins
• When you run a built-in the shell does not have to create a new process
• Instead the command runs in the same process as the shell
• This makes execution faster

Creating Startup Files

• A startup file contains Unix commands that are run just before you get a prompt
• The startup file normally used by Bash is .bash_profile
• This file must be placed in your home directory
• The commands in this file are run after you enter your password ...
• but before you get a prompt

Redirecting Standard Error

• Standard error is the data stream where error messages are sent
• Redirecting standard error allows a program to separate its output stream ...
• from its error messages
• To redirect standard error, you place a 2 in front of the >

  $ ls xxxx
  ls: cannot access xxxx: No such file or directory
  
  $ ls xxxx 2> error.txt
  
  $ cat error.txt 
  ls: cannot access xxxx: No such file or directory

• 2 is the file descriptor for standard error
• Unix also gives you a way to redirect both standard output and standard input ...
• to the same file
• You can do this using the ampersand and greater than symbols together, &>

Shell Scripts

• A shell script can use any shell feature ...
• that is available at the command line
  • Ambiguous file references using the metacharacters ?, * and [ ]
  • Redirection
  • Pipes
• But not those features which are provided by the Readline Library
  • Command line editing (arrow keys, control key combinations)
  • Pathname completion (hit Tab to get more of a filename)
  • The history mechanism (up arrow to recall previous command line)
• Unix also provides control structures
  • Conditionals
  • Loops

Making a Shell Script Executable

• You must have both read and execute permission ...
• to run a shell script ...
• without calling Bash
• Normally you would give a shell script file 755 permissions
• The owner can read, write and execute
• The group and everyone else can read and execute

Specifying Which Version of the Shell Will Run a Script

• When the shell runs a shell script it creates a new subshell
• This subshell runs inside the script process
• Normally this sub-shell will be the same shell version ...
• as your current shell
• A script can use the hashbang line ...
• to specify which shell version to use to run the script
• The hashbang line must be the first line of the script
• The first two characters on the line ...
• must be a hash symbol, # ...
• followed by an exclamation mark, !
• After these two characters, you need to have the absolute pathname ...
• of the version of the shell which will run the script

Comments in Shell Scripts

• Scripts have to be read by the people
  • Who write the program
  • Who maintain the program
  • Who use the program
• To make it clear what is happening inside a program
• You need to add comments to the code
• Anything following a hash mark, # , is a comment
• Except for the hashbang line

Separating and Grouping Commands

• You can enter many commands on a single command line
• if you separate them with a semi-colon, ;

  $ echo Here are the contents of my home directory ; ls ; echo
  Here are the contents of my home directory
  error.txt  foo  it244  work

• When you hit Enter each command is executed ...
• in the order it was typed at the command line

Continuing a Command onto the Next Line

• A command line can be as long as you like
• If your command goes beyond the edge of your window ...
• it will wrap to the next line
• But the line wrap can occur in the middle of a word ...
• which can make it hard to read
• If you want to break the command at a specific point
• You can type a backslash, \ ...
• followed immediately by the Enter key

  $ echo A man \
  > A plan \
  > A canal \
  > Panama
  A man A plan A canal Panama

• After hitting \ and newline ...
• the shell responds with the greater than symbol, >
• This is the secondary prompt
• The shell is telling you it expects more input

Shell Variables

• A variable is a place in memory with a name ...
• that holds a value
• To get the value of a variable
• Put a dollar sign, $, in front of its name
• Some variables are set and maintained by the shell itself
• They are called keyword shell variables
• Or just keyword variables
• Other variables are created by the user
• They are called are called user-created variables
• The environment in which a variable can be used is called the scope
• Shell variables have two scopes
  • Local
  • Global

Local Variables

• Local variables only exist in the shell in which they are defined
• To create a local variable, use the following format

  VARIABLE_NAME=VALUE

• There should be no spaces on either side of the equal sign
• Variables are local unless you explicitly make them global
• If the value assigned to a variable has spaces or tabs ...
• you must quote it ...
• or put a backslash in front of each whitespace character
• If you want to use the value of a variable inside quotes
• You must use double quotes
• When you run a shell script, the local variables will not be visible ...
• because the script is running in a sub-shell ...
• and the local variables are not defined there

Global Variables

• Global variables are defined in one shell ...
• and keep their values in all sub-shells created by that shell
• The env command, when used without an argument ...
• prints the values of all global variables
• To make a variable global use the export command

  export VARIABLE_NAME=VALUE

Keyword Shell Variables

• Keyword shell variables have special meaning to the shell
• By convention, the names of keyword variables are always capitalized
• Most keyword variables can be changed by the user
• This is normally done in the startup file .bash_profile

User-created Variables

• User-created variables are any variables you create
• By convention, the names of user-created variables are lower case
• User-created variables can be either local or global in scope

Positional and Special Parameters

• Positional and special parameters are variables set by Unix
• Positional parameters allow a shell script ...
• to get arguments from the command line
• Whenever you run a script each token on the command line ...
• is assigned to a positional parameter
• The first token is assigned to positional parameter 0
• This parameter contains the pathname of the script
• Each succeeding token on the command line ...
• is assigned to the next positional parameter

  $ cat print_positionals.sh
  #!/bin/bash
  #
  # Prints the value of the first four positional arguments
  
  echo
  echo 0: $0
  echo 1: $1
  echo 2: $2
  echo 3: $3
  
  $ ./print_positionals.sh foo bar bletch
  0: ./print_positionals.sh
  1: foo
  2: bar
  3: bletch

• Positional parameters are the usual way input is given to a script
• The special parameter is # contains the number of arguments

  $ cat print_arg_numbers.sh 
  #!/bin/bash
  #
  # Prints the number of arguments sent to this script
  
  echo
  echo This script received $# arguments
  
  $ ./print_arg_numbers.sh foo bar bletch
  This script received 3 arguments

Quoting and the Evaluation of Variables

• Whenever the value of a variable contains spaces or tabs
• You must quote the string or escape the whitespace character
• There are three ways this
  • Single quotes (' ')
  • Double quotes (" ")
  • Backslash (\)
• Everything surrounded by single quotes ...
• appears in the value of the variable exactly as you entered it
• But inside double quotes you can get the value of a variable ...
• by typing $ in front of a variable name
• The backslash, \ , only effects the character immediately following it

Processes

• A process is a running program
• Unix is a multitasking operating system
• So many processes can run at the same time
• The shell runs in a process like any other program
• Every time you run a program a process is created
• Except when you run a built-in

Process Structure

• Processes are created in a hierarchical fashion
• When the machine is started, there is only one process
• This process is called either init ...
• or systemd
• init or systemd then creates other processes ...
• which provide the services running on the machine
• These child processes can create other processes
• init or systemd have a PID (Process ID) of 1
• Each is the ancestor of every other processes that ever runs on the machine

Process Identification

• Each process has a unique Process ID (PID) number
• ps -f displays a full listing of information ...
• about each process running for the current user

  $ ps -f
  UID        PID  PPID  C STIME TTY          TIME CMD
  it244gh  26374 26373  0 13:41 pts/5    00:00:00 -bash
  it244gh  27891 26374  0 13:57 pts/5    00:00:00 ps -f

• The UID column shows the user's Unix username
• The PID column is the process ID of the process
• The PPID column is the process ID of the parent process
• The process that created this process
• The CMD column gives the command line that started the process

Executing a Command

• When you type something on the command line
• The shell asks the operating system to create a process for the program
• Then it sleeps ...
• waiting for the child process to finish
• When it finishes it sends back an exit status
• Then the shell wakes up and gives you a prompt

The Readline Library

• The readline library is a collection of procedures ...
• which let you edit the command line
• When you use Control key combinations on the command line ...
• you are using the readline library
• Any program written in C can use the readline library
• Here are some of the more useful commands

  Command	Meaning
  Control A	Move to the beginning of the line
  Control E	Move to the end of the line
  Control U	Remove everything from the text entry point to the beginning of the line
  Control K	Remove everything from the text entry point to the end of the line
  →	Move the text entry point one character to the left
  ←	Move the text entry point one character to the right
  ↑	Recall the previous command line entry in the history list
  ↓	Recall the following command line entry in the history list

Readline Completion

• The readline library provides a completion mechanism
• Type a few letters of something ...
• and readline completion will try to supply the rest
• There are three forms of completion provided by the readline library
  • Pathname completion
  • Command completion
  • Variable completion

Aliases

• An alias is a string that you type on the command line
• When you hit Enter the shell replaces it with a command
• To define an alias, you use the alias command
• alias uses the following format in Bash

  alias ALIAS_NAME=ALIAS_VALUE

• There must be no spaces on either side of the equal sign, =
• If the value assigned to the alias has spaces, it must be quoted
• If you follow alias with the name of an alias ..
• it will print the definition

  $ alias ll
  alias ll='ls -l'

• In Bash, an alias cannot accept an argument
• Aliases are not global
• They only work in the shell in which they are defined

Functions

• A function is a collection of commands that is given a name
• A function is like a shell script that lives in RAM
• They can accept arguments from the command line
• They use the same positional arguments that shell scripts use
• Functions differ from shell scripts in a number of ways
  • They are stored in memory (RAM), rather than in a file on disk
  • The shell preprocesses the function so it can execute more quickly
  • The shell executes the function in it's own process
• Functions are local to the shell in which they are defined
• They do not work in sub-shells
• Functions definitions have the following form

  FUNCTION_NAME ()
  {
      COMMANDS
  }

Shell Modification of the Command Line

• But before the shell executes your command line
• If first looks to see if it needs to make changes
• The shell can rewrite the command line before executing it
• It does this to implement features of the shell
• Like command substitution and pathname expansion
• There are 10 different ways in which the shell can modify the command line
• You are only responsible for the following

Alias Substitution

• When Bash performs alias substitution ...
• it replaces the name of the alias ...
• with the value of the alias

Brace Expansion

• Braces, { }, allow you to create several strings all at once
• Braces can appear with strings of characters in front or behind
• The braces contain strings of characters separated by commas
• The shell expands a brace by creating a string ...
• for each string contained within the braces
• If I wanted to create 5 foo files I could use braces expansion as follows

  $ touch foo{1,2,3,4,5}.txt
  
  $ ls
  foo1.txt  foo2.txt  foo3.txt  foo4.txt  foo5.txt

Tilde, ~, Expansion

• Bash replaces ~ with the absolute address of your home directory
• If the ~ is followed by a user name
• Bash substitutes the absolute address of the home director ...
• for that user

Parameter and Variable Expansion

• After tilde expansion, Bash performs parameter and variable expansion

  $ echo $SHELL
  /bin/bash

• When you type a $ followed by the name of a variable ...
• Bash replaces this with the value of the variable

Command Substitution

• In command substitution, a command is run in a sub-shell
• The output of that command replaces the command substitution text
• Command substitution uses the following format

  $(COMMAND)

• Where COMMAND is any valid Unix command

Pathname Expansion

• Pathname expansion is where a pattern using metacharacters
• If replaced with one or more pathnames
• The meta-characters are
  • ?
  • *
  • [ ]

Shell Script Control Structures

• Control structures are Unix statements that change the order of execution ...
• of commands within a program or script
• There are two basic types of control structures
  • Loops
  • Conditionals

The if ... then ... Construct

• The first conditional is the if ... then statement
• It has the format

  if TEST_COMMAND
  then
      COMMAND_1
      COMMAND_2
      ...
  fi

• The TEST_COMMAND can be any Unix command
• then and fi mark off the code block
• These commands are only executed if TEST_COMMAND returns a status code of 0
• The most common command used with if is test

test

• The test command evaluates a logical expression ...
• and returns a status code of 0 if the expression is true

The test Operators

• test has a number of operators
• The operators test for different conditions
• Some operators work only on numbers

  Operator	Condition Tested
  -eq	Two numbers are equal
  -gt	The first number is greater than the second
  -lt	The first number is less than the second

• test uses the different operators when comparing strings

  Operator	Condition Tested
  =	When placed between strings, are the two strings the same
  !=	When placed between strings, are the two strings not the same

• Note that test uses symbols (=) when comparing strings
• But letters preceded by a dash (-eq) when comparing numbers

Using test in Scripts

• Bash provides a synonym for test
• A pair of square brackets, [ ] ...
• that contain a logical expression
• You can write either

  test $number1 -gt $number2

  or

  [ $number1 -gt $number2 ]

The if ... then ... else ... Construct

• The another conditional is the if ... then ... else ... construct
• Which has the following format

  if TEST_COMMAND 
  then
      COMMAND_1
      COMMAND_2
      ....
  else
      COMMAND_A
      COMMAND_B
      ...
  fi

• If TEST_COMMAND returns an exit status of 0 ...
• the commands between then and fi are run
• Otherwise the commands between else and fi are executed

The if ... then ... elif ... Construct

• The if ... then ... elif ... construct allows you to create nested if statements

  if TEST_COMMAND_1
  then
      COMMAND_1
      COMMAND_2
      ...
  elif TEST_COMMAND_2
  then
      COMMAND_A
      COMMAND_B
  ...
  ...
  [else
      COMMAND_N1
      COMMAND_N2
      ...]
  fi

• elif stands for "else if"
• Notice that elif must be followed by then
• The else section is optional

for ... in ... Loops

• The for ... in loop has this format

  for LOOP_VARIABLE in LIST_OF_VALUES
  do
      COMMAND_1
      COMMAND_2
      ...
  done

• The block of loop commands start after the do ..
• and end just before the done
• With this loop, Bash
  • Assigns the first value in the LIST_OF_VALUES to the variable specified by LOOP_VARIABLE
  • Executes the block of commands between do and done
  • Assigns the next value in the LIST_OF_VALUES to the LOOP_VARIABLE
  • Executes the commands between do and the done again
  • And so on until each value in LIST_OF_VALUES has been used

for Loops

• The for loop has a simpler structure than the for ... in ... loop

  for LOOP_VARIABLE
  do
      COMMAND_1
      COMMAND_2
      ...
  done

• The difference between the two for loops
• Is where they get the values for the loop variable
• The for ... in ... loop gets its values
• From the list that appears right after the in keyword
• The simple for loop gets its values from the command line
• This for loop can have different values each time it is run

while Loops

• The first two for loops keep running ...
• until all values supplied to them are used
• A while loop continues running ...
• as long as the command following the keyword while ...
• returns a status code of 0
• while loops have the form

  while TEST_COMMAND
  do
      COMMAND_1
      COMMAND_2
      ...
  done

continue

• Normally, a loop will execute all the commands
• Between do and done
• If you want to skip some part of the code block ...
• call continue ...
• and the following statements will be skipped
• The script then returns to the top of the loop ...
• and begins the next iteration
• continue does not cause the script to break out of the loop
• It merely stops the execution of the loop code ...
• for one iteration

break

• If you want to jump completely out of the loop
• Call break

read Command

• The read command sets a variable ...
• to a value entered by the user
• When Bash is running a script and comes across a line like this

  read value

• Bash does the following
  • It waits for the user to enter some text
  • It assigns the text entered by the user to the variable
    whose name follows the read command
• You can use the -p option to read
• to have read print a prompt

  read -p "Please enter a number: " num