IT 117: Intermediate Scripting

IT 117: Intermediate Scripting
Class 9

Microphone

Questions

Are there any questions before I begin?

Solution to Homework 3

I have posted a solution to homework 3 here.

Let's take a look.

Homework 5

I have posted homework 5 here.

It is due this coming Sunday at 11:59 PM.

Tips and Examples

Never Assume Your Code Works

In the American justice system a defendant is presumed innocent ...
until he or she is proven guilty
In other words you are innocent until you have been shown to be guilty
With code the opposite is true
You should always assume your code is not working ...
until you can test it to prove otherwise
This means your work is not done once you have written the code
It is only finished when you can prove it works
This is done by testing the code
One technique for writing program actually creates the test ...
before writing the code

An Example of Evaluation in a Boolean Context

Headers for if statements and while loops need boolean values
If given another kind of value ...
the interpreter will automatically convert it to boolean
This feature can be used to write shorter, cleaner code
Let me give you an example
I wrote a Python script called find_student.py
When I run it with a part of a name ...
it will print information about that student
I can't run it for you in class because it prints the student ID ...
which is confidential information
If a student prefers to be called by something other than their first name ...
I store this value in a field named "preferred"
For most students the value in this field is the empty string
But where it has a value, I want it to appear in parentheses after the full name
Like this
```
Wiliam Campbell (Bill)
```

So my code has the following if statement

if preferred:
   preferred = "(" + preferred + ")"

This is shorter and more readable than

if len(preferred) > 0:
   preferred = "(" + preferred + ")"

The first version is shorter ...
and easier to read

Review

Sets in Mathematics

A set is an unordered collection of distinct objects
You can put anything into a set
The set with nothing in it is called the empty set

Set Membership

If x is contained in set A we say that x is a member of A
In mathematics, this is written
```
x ∈ A
```
This is a boolean expression meaning its value can only be true or false

Subsets and Supersets

Let's say you have two sets, A and B
If all the values inside A are also in B, then A is a subset of B
The situation is shown in the following diagram

In mathematics, the statement that A is a subset of B is written
```
A ⊂ B
```
Again this statement can only be true or false
If one value inside A is not inside B, then A is not a subset of B
Another way to look at this relationship is to say that B is a superset of A
In mathematics this relationship is written
```
B ⊃ A
```

Union of Sets

A set that has all the elements in A and all the elements in B
is called the union of A and B
It is written
```
A ∪ B
```
In the diagram below the union of A and B is shown in red

Intersection of Sets

A set that has all the elements that are both A and in B ...
is the intersection of A and B
It is written
```
A ∩ B
```
In the diagram below, the intersection of A and B is shown in red

Difference between Sets

A set of all the elements in A which are not in B ...
is called the difference between A and B
And is written
```
A - B
```
In the diagram below, the difference between A and B is shown in red

Symmetric Difference between Sets

The set of all elements in A not in B ...
and all the elements of B not in A ...
is called the symmetric difference between A and B
And is written
```
A Δ B
```
In the diagram below the symmetric difference between A and B is shown in red

Creating a Set in Python

To create a set in Python we can use the built-in set function
set takes a single argument
That argument must be iterable
A Python object is iterable if you can use it in a for loop
This means you can use any of the following as an argument to set
- Lists
- Tuples
- Dictionaries
- Strings

Here is an example

>>> num_list = [1,2,3]
>>> num_set  = set(num_list)
>>> num_set
{1, 2, 3}

Set Literals

A list literal is has values, separated by commas ...

inside square brackets

>>> list_1 = [1, 2, 3, 4, 5]
>>> type(list_1)
<class 'list'>

A set literal uses curly braces

>>> nonsense = {'foo', 'bar', 'bletch'}
>>> type(nonsense)
<class 'set'>

The Empty Set

We can use empty square brackets to create an empty list
```
>>> empty = []
>>> type(empty)
<class 'list'>
```
But we cannot use empty curly braces to create a empty set
The empty curly braces denote an empty dictionary
```
>>> empty = {}
>>> type(empty)
<class 'dict'>
```
When the creators of Python got to set literals
They had run out of symbols to enclose the elements
So they had to reuse { }
So how do you create an empty set?
You run set with no arguments
```
>>> set_1 = set()
>>> set_1
set()
```
You cannot create an empty set like this
```
>>> set_2 = {}
```
This creates not a set but a dictionary
```
>>> type(set_2)
<class 'dict'>
```
That is why an empty set
```
empty_set = set()
```
Looks like this
```
>>> empty_set
set()
```

Adding Elements to a Set

Sets are mutable objects so they can be changed at any time
There are two set methods that can be used to add elements to a set
- add()
- update()
add adds a single element to a set
So if we start with an empty set
```
>>> s1 = set()
>>> s1
set()
```

We can use add to add individual elements

>>> s1.add(1)
>>> s1
{1}
>>> s1.add('two')
>>> s1
{1, 'two'}
>>> s1.add((3,3,3))
>>> s1
{(3, 3, 3), 1, 'two'}

If you add an element to a set that is already in the set ...

nothing will change

>>> s1.add(1)
>>> s1
{(3, 3, 3), 1, 'two'}

But it won't raise an exception
The update method adds several elements to a set

I takes one argument which must be iterable

>>> s2 = set()
>>> s2
set()
>>> s2.update([1, 2, 3])
>>> s2
{1, 2, 3}
>>> s2.update('foo')
>>> s2
{1, 2, 3, 'f', 'o'}

Notice that only one 'o' was added to the set ...
and no exception was raised

Removing Elements from a Set

To remove an element from a set use one of two methods
- discard()
- remove()
Both methods take a single argument

The value that is to be removed

>>> numb_set
{1, 2, 3, 4, 5}
>>> numb_set.discard(2)
>>> numb_set
{1, 3, 4, 5}
>>> numb_set.remove(4)
>>> numb_set
{1, 3, 5}

The only difference is what happens ...
when you remove a value that is not in the set

discard will say nothing

>>> numb_set.discard(2)
>>> numb_set
{1, 3, 5}

But remove will raise an exception

>>> numb_set.remove(4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 4

Attendance

New Material

The Size of a Set

One way to compare sets is to compare the number of their elements
Mathematicians call the size of a set its cardinality

The len function gives the size of a set

>>> s1 = {1, 2, 3}
>>> len(s1)
3
>>> s2 = {3, 2, 1}
>>> len(s2)
3
>>> s3 = {'one', 'two', 'three', 'four'}
>>> len(s3)
4

When Are Sets Equal?

Both lists and sets have elements
So we can run the len function on each to get their size
But how to we tell two lists, or two sets, apart?
Lists have two attributes
- Their elements
- The order of the elements
So we can have two lists that have the same elements ...

but are not equal

>>> list_1 = [1, 2, 3]
>>> list_2 = [3, 2, 1]
>>> list_1 == list_2
False

But sets only have one attribute
Their elements

If two sets have the same elements they are equal

>>> s1 = {1, 2, 3}
>>> s2 = {3, 2, 1}
>>> s1 == s2
True

Elements in A Python Set

In mathematics the elements of a set can be anything ...
even other sets
Python is not so open minded
Only immutable values can be elements of a set

So you can create a set of tuples

>>> tuple_set = {(1,2), (3,4), (5,6)}
>>> tuple_set
{(5, 6), (1, 2), (3, 4)}

But you cannot create a set of lists

>>> list_set = {[1,2], [3,4], [5,6]}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

But what if you use variables to define a set?
In that case, Python uses the values of the variables

So the set elements remain constant even if the variables change

>>> a, b, c = 1, 2, 3
>>> A = {a, b, c}
>>> A
{1, 2, 3}
>>> a = 5
>>> A
{1, 2, 3}

`for` Loops with Sets

Sets are iterable
This means that they can be used in a for loop

The general format of a for loop looks like this

for LOOP_VARIABLE in ITERABLE_OBJECT:
    STATEMENT
    ...

The loop variable is set to each element of the set in turn

This continues until all values in the object are used once

>>> s1 = {1, 2, 3, 4, 5}
>>> for number in s1:
...     print(number)
... 
1
2
3
4
5
>>> s2 = {'one', 'two', 'three', 'four', 'five'}
>>> for number in s2:
...     print(number)
... 
two
five
three
one
four

The elements of a set have no inherent order
We can create elements in a set in a certain order
But there is no guarantee that will be their order in the loop
In the case of s1 the order was the same
But not in s2

This is not the case with lists where order matters

>>> list_1 = [1, 2, 3, 4, 5]
>>> for number in list_1:
...    print(number)
... 
1
2
3
4
5
>>> list_2 = [5, 4, 3, 2, 1]
>>> for number in list_2:
...    print(number)
... 
5
4
3
2
1

Testing for Set Membership

To test whether a particular value is contained in a list ...

you can use the in operator

>>> list_1
[5, 4, 3, 2, 1]
>>> 7 in list_1
False
>>> 5 in list_1
True

We can use the same in operator ...

to test whether a value is a member of a set

>>> s1
{1, 2, 3, 4, 5}
>>> 7 in s1
False
>>> 8 in s1
False
>>> 3 in s1
True

To test whether a value is not inside a group ...

we can use the not in operator

>>> 8 not in s1
True
>>> 3 not in s1
False

Although not in are two words ...
it is a single operator

Union of Sets in Python

You can combine two sets to create a new set using the union operation
The union of two sets is a new set consisting of all the elements of both sets
Of course the new set will have no duplicates
In mathematics this is written
```
A ∪ B
```

We can form the union of two sets in Python by using the union method

>>> A = {1, 4, 8, 12}
>>> B = {1, 2, 6, 8}
>>>  A.union(B)
{1, 2, 4, 6, 8, 12}

The union operation is symmetrical
This means that
```
A ∪ B
```
is the same as
```
B ∪ A
```
So it doesn't matter what set object we use when running the method
```
>>> B.union(A)
{1, 2, 4, 6, 8, 12}
```
In addition to the union method on the set object ...
there is also a union operator
The union operator gives the same results as the union method
```
>>> A | B
{1, 2, 4, 6, 8, 12}
```
The union operator is also symmetric
```
>>> B | A
{1, 2, 4, 6, 8, 12}
```

Intersection of Sets in Python

The intersection of two sets ...
has only those elements which are inside both sets
In mathematics, this is written
```
A ∩ B
```

Sets in Python have an intersection method

>>> A
{8, 1, 12, 4}
>>> B
{8, 1, 2, 6}
>>> A.intersection(B)
{8, 1}

Intersection is also symmetrical, so
```
A ∩ B = B ∩ A
```
So we can get the same results by running the intersection method ...
on either object
```
>>> B.intersection(A)
{8, 1}
```
Python also has an intersection operator,
```
>>> A & B
{8, 1}
```
It is also symmetrical
```
>>> B & A
{8, 1}
A & B == B & A
True
```

Difference between Sets in Python

Another way to form a new set ...
is to take the difference between the sets
The difference between set A and set B ...
contains all elements in A that are not in B
This is written
```
A - B
```

In Python, we can use the set difference method

>>> A
{8, 1, 12, 4}
>>> B
{8, 1, 2, 6}
A.difference(B)
{12, 4}

Set difference is not a symmetric operation
```
A - B ≠ B - A
```
So the difference method is not symmetric
```
>>> B.difference(A)
{2, 6}
```
Python also has a set difference operator, -
```
>>> A - B
{12, 4}
```

which is also not symmetric

>>> A - B == B - A
False
>>> A - B != B - A
True

Symmetric Difference between Sets in Python

Lets say we have two sets, A and B
The symmetric difference between A and B ...
consists of all the elements of A that are not in B ...
and all the elements of B that are not in A
In mathematics, this is written
```
A Δ B
```
We can take the symmetric difference in Python

using the symmetric_difference method

>>> A
{8, 1, 12, 4}
>>> B
{8, 1, 2, 6}
>>> A.symmetric_difference(B)
{2, 4, 6, 12}

The symmetric difference operation is symmetric
```
A Δ B = B Δ A
```
So we can run the method on either set object ...
and get the same results
```
B.symmetric_difference(A)
{2, 4, 6, 12}
```
Python also has a symmetric difference operator, ^
```
A ^ B
{2, 4, 6, 12}
```
which is also symmetric
```
>>> A ^ B == B ^ A
True
```

Augmented Set Operators

We now have a new set of operators that work on sets

Operator Operation

| union

& intersection

- difference

^ symmetric difference
In some sense, they are like the arithmetic operators +, -, *, / ...
because they work on two values ...

Operator	Operation
\|	union
&	intersection
-	difference
^	symmetric difference

and create a new value

>>> s1 = {1,3,5}
>>> s2 = {2,4,6}
>>> s1 | s2
{1, 2, 3, 4, 5, 6}

All the numeric operators have augmented assignment operators ..
that we can use to shorted assignment statements
```
>>> n1 = 3
>>> n2 = 5
>>> n1 += n2
>>> n1
8
```
Python has augmented set operators ...
for all the set operators
So instead of writing
```
s1 = s1 | s2
```
We can write
```
>>> s1 |= s2
>>> s1
{1, 2, 3, 4, 5, 6}
```

Subsets and Supersets

If all the elements of A are also in set B
Then A is a subset of B
We can tell if one set is a subset of another by using the issubset method

If we have two sets

>>> A = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
>>> B = {1, 3, 5, 7, 9}

We can ask if one set is the subset of another like this
```
>>> A.issubset(B)
False
>>> B.issubset(A)
True
```
Python also provides the subset operator, <=
```
>>> A <= B
False
>>> B <= A
True
```
If all the elements of the set B are contained in A then A is a superset of B
We can ask if one set is a superset of another using the issuperset method
```
>>> A.issuperset(B)
True
>>> B.issuperset(A)
False
```
The superset operator is >=
```
>>> A >= B
True
>>> B >= A
False
```

Disjoint

If two sets have no element in common they are said to be disjoint
The isdisjoint method of a set object ...

will tell you if two sets are disjoint

>>> odds = {1, 3, 5, 7, 9}
>>> evens = {2, 4, 6, 8, 10}
>>> evens.isdisjoint(odds)
True

Since this condition is symmetric ...
we can run the method on either set object
```
>>> odds.isdisjoint(even)
True
```

The clear Method

The clear method removes all elements from a set

>>> D = {1, 2, 3, 4, 5}
>>> D
{1, 2, 3, 4, 5}
>>> D.clear()
>>> D
set()

`min` And `max` with Sets

To find the set element with the maximum value you can use the max built-in function
```
>>> B = {1, 3, 5, 7, 9}
>>> max(B)
9
```
To find the set element with the minimum value use the min function
```
>>> min(B)
1
```
These functions will not normally work on sets ...

where the elements have no natural ordering

>>> s1 ={1, "q", 5, "kk", 87}
>>> max(s1)
Traceback (most recent call last):
  File "<;stdin>;", line 1, in <module>
TypeError: '>' not supported between instances of 'str' and 'int'
>>> min(s1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'str' and 'int'

When Sets Are Best

I would guess that most of you think you will never need to use sets
But there are certain situations where they are very useful
Let me show you an example
Let's say you have two files with email address ...
and you know there are duplicates
You want to create a new file that combines the addresses in both files ...
but eliminates duplicates
You might think to use lists

To do this we first have to read in the two files and store each in a list

emails_1 = []
emails_2 = []
for line in email_file_1:
    emails_1.append(line.strip())
for line in email_file_2:
    emails_2.append(line.strip())

Now we create a new list which has all the emails from the first list
```
emails_new = emails_1[:]
```
Then we loop through the second list
We do this to look for entries that are not already in the new list

If we find one we have to add it to the news list

for email in emails_2:
    if email not in emails_new:
        emails_new.append(email)

When we run this we get

$ ./merge_emails_1.py
Email file 1: emails_1.txt
Email file 2: emails_2.txt
List 1
Glenn.Hoffman@umb.edu
big_bill@hotmail.com
larry.wall@gmail.com
timsmith@yahoo.com
bigorangeshithead@whitehouse.gov

List 2
Glenn.Hoffman@umb.edu
alanh@hotmail.com
timsmith@yahoo.com
davidm@mac.com
billybob@cs.umb.edu


Merged list
Glenn.Hoffman@umb.edu
big_bill@hotmail.com
larry.wall@gmail.com
timsmith@yahoo.com
bigorangeshithead@whitehouse.gov
alanh@hotmail.com
davidm@mac.com
billybob@cs.umb.edu

You can see the full code for this script here
Not a bad way to go if the lists were small
But what if each list had thousands of addresses?
The merge loop would have to test each one of the entries in the second list
This could take quite a bit of time
If we used sets, it would be much quicker
Not only that, but the code would be much shorter ...
and shorter code is always better than longer code
We replace the loop from the first script with a single line of code
```
emails_new = emails_1.union(emails_2)
```
You can see the code here

Practical Example

Some problems we face in real life are best handled using sets
At UMB, instructors have to send academic status reports for students who are playing sports

Let's say I have a set of student IDs each of my courses

>>> it244 = {"01459659", "00709552", "01565798", "00974687", "01357397", "01107434", 
...          "01516157", "01470962", "01015502", "01283749", "01313387", "01113342", 
...          "01684609", "01018750", "01458680", "01530289", "01600144"}
>>> it116 = {"01276750", "01246473", "01146053", "01361550", "01330451", "01405338", 
...          "01240592", "01328324", "01393077", "00822499", "01158165", "01342910", 
...          "01559794", "01293714", "01352486", "01367216", "01165111", "01617531", 
...          "01485027", "01397047", "01459659"}

Why did I have to enter the IDs as strings instead of numbers?
Most UBM student IDs start with 0
But in Python a number literal cannot begin with 0

If you try Python will object

>>> a = 01684609
  File "<stdin>", line 1
    a = 01684609
               ^
SyntaxError: invalid token

The only way to get around this problem is to enter the IDs as strings
Are there any students who are registered in both courses?
I can use the intersection method to find out
```
>>> it244.intersection(it116)
{'01459659'}
```
Now let me create a new set of all the students in both classes
```
>>> all_students = it244 | it116
```
By using the union operator |, I have eliminated all duplicates

I can see this by taking the length of both sets

>>> len(it244)
17
>>> len(it116)
21
>>> len(all_students)
37

Since 17 + 21 is 38 creating the union of the two sets has eliminated the duplicates
Let's say the Athletic department has sent me the ID's of all students playing sports
They send me one list for each sport
Let's say these set objects are pointed too by the variables basketball and baseball
I need to combine these sets into a new set of all student athletes
Again I use the union operator
```
>>> athletes = basketball | baseball
```
Now I can get the student IDs for all students whose grades I need to report

I do this by creating a new set that is the intersection of all_students and athletes

>>>report_needed = all_students & athletes
>>> report_needed
{'01146053', '01015502', '01367216', '01330451'}

Another Example

I like to give you examples how how Python scripts can be used to automate boring tasks
I try to use examples of scripts that I create to help me with my work
For example, I sometimes need to create a list of names ...
to use as sample data for a homework assignment ...
or an example I use in Class Notes
I cannot use the names of student in my courses due to privacy regulations
In the past I created these list manually ...
but this takes a lot of time to create a long list of names
I decided to write a script to do this for me

The algorithm I used was straight forward

create a tuple of first names
create a tuple of last names
ask the user for the number of names required 
create an empty set full_names
while the length of full_names is less than number required:
   create a full name from a randomly chosen first name and last name
   add this full name to full_names
for each name in full_names:
   print the name

I have to make sure that no duplicates appear in the names I create
By adding each name to a set ...
I can be sure that no duplicate names will be created
You can see the full code for this script here

Sets More Efficient Than Lists

Why use sets if we can use lists?
Lists can do everything sets can do ...
and the elements also have an order
But getting a value from a set is very fast
Because Python implements sets using hash tables
So sets are more efficient than lists ...
for certain operations
If the number of elements in a problem is small, efficiency is not a concern
But what if we were doing something that involved a large number of elements?
Here sets could make things significantly faster

Tips and Examples

Review

New Material

Questions

Solution to Homework 3

Homework 5

Tips and Examples

Never Assume Your Code Works

An Example of Evaluation in a Boolean Context

Review

Sets in Mathematics

Set Membership

Subsets and Supersets

Union of Sets

Intersection of Sets

Difference between Sets

Symmetric Difference between Sets

Creating a Set in Python

Set Literals

The Empty Set

Adding Elements to a Set

Removing Elements from a Set

Attendance

New Material

The Size of a Set

When Are Sets Equal?

Elements in A Python Set

for Loops with Sets

Testing for Set Membership

Union of Sets in Python

Intersection of Sets in Python

Difference between Sets in Python

Symmetric Difference between Sets in Python

Augmented Set Operators

Subsets and Supersets

Disjoint

The clear Method

min And max with Sets

When Sets Are Best

Practical Example

Another Example

Sets More Efficient Than Lists

Class Exercise

Class Quiz

`for` Loops with Sets

`min` And `max` with Sets