1. Great Theoretical Ideas in Computer Science
15-251
Great Theoretical Ideas in Computer Science

2. Cantor’s Legacy: Infinity and Diagonalization
Lecture 23 (April 7, 2009)

3. Ideas from the course A hint of the infinite Induction
Numbers, Number theory and Algebra
Representation
Finite Counting and Probability
Automata and Computation
A hint of the infinite
Infinite row of dominoes
Infinite sums (formal power series)
Infinite choice trees, and infinite probability

4. Infinite RAM Model Platonic Version:
One memory location for each natural number 0, 1, 2, …
Aristotelian Version: Whenever you run out of memory, the computer contacts the factory. A maintenance person is flown by helicopter and attaches 1000 Gig of RAM and all programs resume their computations, as if they had never been interrupted.

5. Ideal Computer is defined as a computer with infinite RAM.
The Ideal Computer: no bound on amount of memory no bound on amount of time
Ideal Computer is defined as a computer with infinite RAM.
You can run a Java program and never have any overflow, or out of memory errors.
We will be talking about ideal computers. Ideal computers differ from real computers in that they have no bound on the amount of memory they can use.
If the notion of a computer with an infinite amount of memory seems unsatisfactory, think of it this way:
Your programs ask for as much memory as they wish. Whenever your computer runs out of memory, the computer automatically contacts the manufacturer, and a technician arrives with additional ram chips for your computer. Your computations continue where they left off, as if nothing had happened.

6. An Ideal Computer It can be programmed to print out:
2: …
1/3: …
: …
e: …
: …
What can these ideal computers do that real computers can’t?
Well, real computers have a finite amount of memory. Add up all the memory they have – hard disk space, RAM, registers, etc. , and they still have only k bits of storage. This means, a real computer can be in only 2^k different states.
Suppose you want to compute the ith digit of pi, for any I. On a real computer, as your computation, runs the computer changes from state to state. Eventually, it will run out of new states and must return to a state indistinguishable from a previous one. Thus, a deterministic program on a real computer can output only repeating decimals.
So, real computers can output some digits of pi, but there will always be some ith digit which cannot be computed.
Because ideal computers have an unlimited amount of memory, they can be in an unlimited number of different states. Thus, we can write a program on an ideal computer that can print the ith digit of pi, for any I. Any digit of pi which interests us will eventually appear in the output.

7. Printing Out An Infinite Sequence..
A program P prints out the infinite sequence s0, s1, s2, …, sk, … if when P is executed on an ideal computer, it outputs a sequence of symbols such that
The kth symbol that it outputs is sk
For every k2, P eventually outputs the kth symbol. I.e., the delay between symbol k and symbol k+1 is not infinite.

8. Computable Real Numbers
A real number R is computable if there is a program that prints out the decimal representation of R from left to right. Thus, each digit of R will eventually be output.
Pi is “computable,” because we can program an ideal computer to output the ith digit for any I.
In general, any real number r is computable, if we can program an ideal computer to print it digit by digit, such that any digit we are interested in will eventually be printed.
For example, to find the 2^50th digit of r, we start the program and count the output digits. Eventually, we’ll the 2^50th digit will appear.
The real numbers we’re familiar with -- pi, e, phi, and the square root of two– are all computable. But in general, are all real numbers computable?
Are all real numbers computable?

9. Are all real numbers describable?
Describable Numbers
A real number R is describable if it can be denoted unambiguously by a finite piece of English text.
2: “Two.”
: “The area of a circle of radius one.”
<think of good segue>
When a finite piece of English text unambigously describes a real number r, we call that number describable. By unambigous, we mean that the text must describe one number and only one number.
For example, we can describe pi as “The area of a circle of radius one.”
This description applies to pi and only to pi – there is no other number that is the area of a circle of radius one.
Our descriptions o f numbers, such as pi, do not necessarily tell us how to compute them. So, it Is not obvious that all describable numbers are computable.
Are all real numbers describable?

10. Is every computable real number, also a describable real number?
And what about the other way?
The moral here is that computer programs can be viewed as descriptions of their output.
A program that computes pi, for example, can be taken as an unambiguous description of the number pi.
Computable R: some program outputs R
Describable R: some sentence denotes R

11. Computable  describable
Theorem:
Every computable real is also describable
Proof: Let R be a computable real that is output by a program P. The following is an unambiguous description of R:
“The real number output by the following program:” P
However, it is true that every computable number is describable.
Let r be a computable real number. Therefore, there is some program P which outputs r. Therefore, we can describe r by saying “The real number output by P.”
That is, since the program outputs exactly r, it can be taken as an unambiguous description of r.

12. MORAL: A computer program can be viewed as a description of its output.
The moral here is that computer programs can be viewed as descriptions of their output.
A program that computes pi, for example, can be taken as an unambiguous description of the number pi.
Syntax: The text of the program
Semantics: The real number output by P

13. Are all reals describable? Are all reals computable?
We saw that computable  describable,
but do we also have describable  computable?
So, it is natural to wonder if all real numbers can be described in English.
<get answers from audience>
Expect: someone will say that you can’t describe an indescribable real, so if he asserts that there is no such thing I wouldn’t be able to supply a counterexample.
Response: You’re correct that I cannot show you a specific indescribable number as evidence that they exist. Your argument, however, leaves open the possibility that I could convince that indescribable numbers exist, without showing you a specific one.
Questions we will answer in this (and next) lecture…

14. Correspondence Principle
If two finite sets can be placed into 1-1 onto (bijective) correspondence, then they have the same size.
Recall the correspondence principle.
Two sets, A and B. Put them into one-to-one, onto correspondence: they have the exact same size.

15. Correspondence Definition
In fact, we can use the correspondence as the definition:
Two finite sets are defined to have the same size if and only if they can be placed into 1-1 onto (bijective) correspondence.
In fact, we will define two sets to have the same size if and only if they can be placed into one-to-one, onto correspondence.
In the case of finite sets, this definition of size matches our intuitive understanding of size.

16. Georg Cantor ( )
Georg Cantor was one of the greatest 19th-century mathematical geniuses.
While his work was in pure mathematics, the theorems Cantor proved and the techniques he developed have, as we’ll see today, profound implications for computer science.
In addition to being a seminal character in mathematics, Cantor was also a very interesting person. Cantor suffered from bouts of severe depression which lasted for weeks. His work in mathematics was not well appreciated during his life, a fact which further fueled his depression.
When Cantor was too depressed to pursue math, he pursued his interests in philosophy in literature.
Cantor’s pet project in literature, unfortunately, was as controversial as his mathematics. He published several papers attempting to prove his pet theory that Shakespeare’s plays were written not by Shakespeare but by Francis Bacon.

17. Cantor’s Definition (1874)
Two sets are defined to have the same size if and only if they can be placed into 1-1 onto correspondence.
One of Cantor’s most important contributions to mathematics was his generalization of the correspondence principle.
Two sets, infinite or finite, have the same size if and only if they can be placed into one-to-one, onto correspondence.
This notion gives us a well-defined understanding of “size” of infinite sets, which is compatible with our intuition for finite sets.
If there exists a bijection between them.

18. Cantor’s Definition (1874)
Two sets are defined to have the same cardinality if and only if they can be placed into 1-1 onto correspondence.
The term “cardinality” is synonymous with “size.”
The cardinality of a finite set is the number of elements it contains.
Sets, infinite or finite, have the same cardinality if and only if they can be placed into one to one onto correspondence.
If there exists a bijection between them.

19. Do N and E have the same cardinality?
The even, natural numbers.
Let’s take a look at two infinite sets, N, the natural numbers, and E the even naturals.
Do N and E have the same size?
Most students will say they clearly don’t have the same size.

20. The attempted correspondence f(x)=x does not take E onto N.
E and N do not have the same cardinality! E is a proper subset of N with plenty left over.
The attempted correspondence f(x)=x does not take E onto N.
Bonzo has a very natural view on the question.
We can take the set of natural numbers, cross off all the naturals, and still be left with an infinite number of odd numbers. So, this means that there are many more natural numbers than even numbers.
In other words, the correspondence which maps the even number x to the natural number x, is one-to-one: all the even naturals get mapped to distinct natural numbers. But the correspondence is not onto: all the odd naturals are left over.

21. E and N do have the same cardinality!
N = 0, 1, 2, 3, 4, 5, … E = 0, 2, 4, 6, 8,10, …
f(x) = 2x is 1-1 onto.
Bonzo’s intuitive argument is very natural, but it’s not a proof.
Odette has a better idea.
In fact, the evens and the naturals DO have the same cardinality, and there is a one-to-one onto correspondence from the naturals to the even naturals which proves it.
We map 0 to 0, 1 to 2, 2 to 4 and so forth, mapping the natural number x to the even natural 2*x. It’s easy to see that this correspondence is one-to-one – every natural gets mapped to a distinct even, and onto—every even is mapped from some natural.

22. This distinction never arises when the sets are finite.
Lesson:
Cantor’s definition only requires that some 1-1 correspondence between the two sets is onto, not that all 1-1 correspondences are onto.
This distinction never arises when the sets are finite.
There’s an important lesson to learn here.
In the case of finite sets, a one to one correspondence between two sets implies that all one-to-one correspondences are onto. That is, no matter how I try to match my fingers up, there are never any left over.
But this is not the case for infinite sets. For two infinite sets A and B, there may many one-to-one correspondences, only some of which are onto.
Fortunately, Cantor’s definition of cardinality requires only that there be some one-to-one correspondence which is onto.
So, showing a one-to-one correspondence which isn’t onto proves nothing about the cardinality of two infinite sets.

23. Cantor’s Definition (1874)
Two sets are defined to have the same size if and only if they can be placed into 1-1 onto correspondence.
One of Cantor’s most important contributions to mathematics was his generalization of the correspondence principle.
Two sets, infinite or finite, have the same size if and only if they can be placed into one-to-one, onto correspondence.
This notion gives us a well-defined understanding of “size” of infinite sets, which is compatible with our intuition for finite sets.

24. You just have to get used to this slight subtlety in order to argue about infinite sets!
If the idea that a one-to-one correspondence between equally-sized sets can leave some elements left over, well….. Tough!
You simply have to get used to this fact in order to reason about infinite sets.

25. Do N and Z have the same cardinality?
Let’s take a look at two more infinite sets:
N, the natural numbers, and Z, the integers.

26. Therefore, there are far more integers than naturals.
No way! Z is infinite in two ways: from 0 to positive infinity and from 0 to negative infinity.
Therefore, there are far more integers than naturals.
Actually, no!

27. N and Z do have the same cardinality!
N = 0, 1, 2, 3, 4, 5, 6 … Z = 0, 1, -1, 2, -2, 3, -3, ….
f(x) = x/2 if x is odd x/2 if x is even

28. Transitivity Lemma Lemma: If f: AB is a bijection, and
g: BC is a bijection.
Then h(x) = g(f(x)) defines a function
h: AC that is a bijection too.
Hence, N, E, and Z all have the same cardinality.

29. Do N and Q have the same cardinality?
= { 0, 1, 2, 3, 4, 5, 6, 7, …. }
Q = The Rational Numbers

30. No way!
The rationals are dense: between any two there is a third. You can’t list them one by one without leaving out an infinite number of them.

31. Don’t jump to conclusions!
There is a clever way to list the rationals, one at a time, without missing a single one!

32. First, let’s warm up with another interesting example:
N can be paired with N×N

33. Theorem: N and N×N have the same cardinality

34. Theorem: N and N×N have the same cardinality… 4 3 2 1
The point (x,y) represents the ordered pair (x,y) 1 2 3 4 …

35. Theorem: N and N×N have the same cardinality… 4 3 2 1 6 7 8 9
The point (x,y) represents the ordered pair (x,y) 3 4 1 2 5 1 2 3 4 …

36. The first few tuples output…
(0,0) (0,1), (1,0) (0,2), (1,1), (2,0) (0,3), (1,2), (2,1), (3,0) …

37. Defining bijection f: N -> N×N
let i := 0; //will range over N
for (sum = 0 to forever) {
//generate all pairs with this sum
for (x = 0 to sum) {
y := sum-x
define f(i) := the point (x,y)
i++; } }

38. Onto the Rationals!

39. The point at x,y represents x/y

40. The point at x,y represents x/y

41. Cantor’s 1877 letter to Dedekind: “I see it, but I don't believe it! ”
Georg Cantor was one of the greatest 19th-century mathematical geniuses.
While his work was in pure mathematics, the theorems Cantor proved and the techniques he developed have, as we’ll see today, profound implications for computer science.
In addition to being a seminal character in mathematics, Cantor was also a very interesting person. Cantor suffered from bouts of severe depression which lasted for weeks. His work in mathematics was not well appreciated during his life, a fact which further fueled his depression.
When Cantor was too depressed to pursue math, he pursued his interests in philosophy in literature.
Cantor’s pet project in literature, unfortunately, was as controversial as his mathematics. He published several papers attempting to prove his pet theory that Shakespeare’s plays were written not by Shakespeare but by Francis Bacon.

42. N, E, Q and Z are all countable.
Countable Sets
We call a set countable if it can be placed into 1-1 onto correspondence with the natural numbers N.
Hence
N, E, Q and Z are all countable.

43. Do N and R have the same cardinality?
R = The Real Numbers

44. No way!
You will run out of natural numbers long before you match up every real.

45. Now hang on a minute!
You can’t be sure that there isn’t some clever correspondence that you haven’t thought of yet.

46. To do this, he invented a very important technique called
I am sure!
Cantor proved it.
To do this, he invented a very important technique called
“Diagonalization”

47. Theorem: The set R[0,1] of reals between 0 and 1 is not countable.
Proof: (by contradiction)
Suppose R[0,1] is countable.
Let f be a bijection from N to R[0,1].
Make a list L as follows:
0: decimal expansion of f(0) 1: decimal expansion of f(1) 2: decimal expansion of f(2) …
k: decimal expansion of f(k)

48. Theorem: The set R[0,1] of reals between 0 and 1 is not countable.
Proof: (by contradiction)
Suppose R[0,1] is countable.
Let f be a bijection from N to R[0,1].
Make a list L as follows:
0: … 1: … 2: … …
k: …

49. Position after decimal point1 2 3 4 …
Index

50. Position after decimal point1 2 3 4 … 5 9 6 8
Index

51. digits along the diagonal1 2 3 4 … d0 d1 d2 d3

52. Define the following real number ConfuseL = . C0 C1 C2 C3 C4 C5 …1 2 3 4 d0 d1 d2 d3 …
Define the following real number ConfuseL = . C0 C1 C2 C3 C4 C5 …

53. L 1 2 3 4 d0 d1 d2 d3 …
Define the following real number ConfuseL = . C0 C1 C2 C3 C4 C5 …
5, if dk=6
6, otherwise
Ck=

54. 5, if dk=6
6, otherwise
Ck= L 1 2 3 4 d1 d2 d3 …
C0d0 C1 C2 C3 C4 …

55. 5, if dk=6
6, otherwise
Ck= L 1 2 3 4 d0 d2 d3 …
C0 C1d1 C2 C3 C4 …

56. 5, if dk=6
6, otherwise
Ck= L 1 2 3 4 d0 d1 d3 …
C C1 C2d2 C3 C4 …

57. Diagonalized! By design, ConfuseL can’t be on the list L!
ConfuseL differs from the kth element on the list L in the kth position.
This contradicts the assumption that the list L is complete; i.e., that the map
f: N to R[0,1] is surjective.

58. The set of reals is uncountable! (Even the reals between 0 and 1.)
An aside:, you can set up a
correspondence between
R and R[0,1] .

59. Hold it!
Why can’t the same argument be used to show that the set of rationals Q is uncountable?

60. The argument is the same for Q until the punchline.
However, since CONFUSEL is not necessarily rational, so there is no contradiction from the fact that it is missing from the list L.

61. Another diagonalization proof
Problem from a final:
Show that the set of real numbers in [0,1] whose decimal expansion has the property that every digit is a prime number (2,3,5, or 7) is uncountable.
E.g., and … are in the set, but
… and … are not.

62. Another diagonalization proof
Show that the set of real numbers in [0,1] whose decimal expansion has the property that every digit is a prime number (2,3,5, or 7) is uncountable.

63. Another diagonalization proof
Show that the set of real numbers in [0,1] whose decimal expansion has the property that every digit is a prime number (2,3,5, or 7) is uncountable.

64. Another diagonalization proof
Show that the set of real numbers in [0,1] whose decimal expansion has the property that every digit is a prime number (2,3,5, or 7) is uncountable.
Assume this set is countable and therefore it can be placed in a list L.
Given L, show how to define a number called Confuse.
Show that Confuse is not in L.
Explain why Confuse not being in L implies the set is not countable.

65. Countable and Uncountable
N, E, Q and Z are all countable sets
R is an uncountable set

66. Back to the questions we were asking earlier

67. Are all reals describable? Are all reals computable?
We saw that computable  describable,
but do we also have describable  computable?
So, it is natural to wonder if all real numbers can be described in English.
<get answers from audience>
Expect: someone will say that you can’t describe an indescribable real, so if he asserts that there is no such thing I wouldn’t be able to supply a counterexample.
Response: You’re correct that I cannot show you a specific indescribable number as evidence that they exist. Your argument, however, leaves open the possibility that I could convince that indescribable numbers exist, without showing you a specific one.

68. Standard Notation = Any finite alphabet
Ex.: {a,b,c,d,e,…,z} or {a,b} or {0,1}
S* = All finite strings of symbols from S including the empty string e

69. Theorem: Every subset S of S* is countable
Try #1:
Sort S alphabetically, map first word to 0, second word to 1, and so on…
What if S = {a, b, aa, bb, aaa, bbb, aaaa, bbbb, …. } ?

70. Theorem: Every subset S of S* is countable
Proof: Sort S by first by length and then alphabetically.
Map the first word to 0, the second to 1, and so on….

71. This sorting on length, and then aphabetically, is similar in spirit to…4 3 2 1 6 7 8 9 3 4 1 2 5 1 2 3 4 …

72. Stringing Symbols Together
= The symbols on a standard keyboard
For example:
The set of all possible Java programs is a subset of S*
The set of all possible finite pieces of English text is a subset of S*

73. The set of all possible Java programs is countable.
Thus:
The set of all possible Java programs is countable.
The set of all possible finite length pieces of English text is countable.

74. There are countably many Java programs and uncountably many reals.
Hence,
Most reals are not computable!

75. There are countably many descriptions and uncountably many reals.
I see!
There are countably many descriptions and uncountably many reals.
Hence:
Most real numbers are
not describable!

76. Are all reals describable? Are all reals computable?
We saw that computable  describable,
but do we also have describable  computable? NO NO
So, it is natural to wonder if all real numbers can be described in English.
<get answers from audience>
Expect: someone will say that you can’t describe an indescribable real, so if he asserts that there is no such thing I wouldn’t be able to supply a counterexample.
Response: You’re correct that I cannot show you a specific indescribable number as evidence that they exist. Your argument, however, leaves open the possibility that I could convince that indescribable numbers exist, without showing you a specific one.

77. Is there a real number that can be described, but not computed?
Wait till the
next lecture!

78. We know there are at least 2 infinities. (the number of naturals,
the number of reals.)
Are there more?

79. Definition: Power Set
The power set of S is the set of all subsets of S. The power set is denoted as P(S).
Proposition:
If S is finite, the power set of S has cardinality 2|S|

80. Theorem: S can’t be put into bijection with P(S)
{A}
{B} B
{C}
{A,B} C
{B,C}
{A,C}
{A,B,C}
Suppose f:S → P(S) is a bijection.
Let CONFUSEf = { x | x  S, x  f(x) }
Since f is onto, exists y  S such that f(y) = CONFUSEf.
Is y in CONFUSEf?
YES: Definition of CONFUSEf implies no
NO: Definition of CONFUSEf implies yes

81. This proves that there are at least a countable number of infinities.
The first infinity is called: 0

82. Are there any more infinities?
0,1,2,…
Are there any more infinities?

83. P(S) is provably larger than any of them.
0,1,2,…
Let S = {k | k 2 N }
P(S) is provably larger than any of them.

84. In fact, the same argument can be used to show that no single infinity is big enough to count the number of infinities!

85. Cantor wanted to show that the number of reals was 1
0,1,2,…
Cantor wanted to show that the number of reals was 1

86. However, he was unable to prove it. This helped fuel his depression.
Cantor called his conjecture that 1 was the number of reals the “Continuum Hypothesis.”
However, he was unable to prove it. This helped fuel his depression.
The problem of the continuum was the source of great vexation for Cantor. Once, he thought he had proved it false, only to uncover his error the following day. And later, Cantor believed he had proved the hypothesis true, but again he quickly discovered a bug in his proof.

87. The Continuum Hypothesis can’t be proved or disproved from the standard axioms of set theory!
This has been proved!

88. Here’s What You Need to Know…
Cantor’s Definition: Two sets have the same cardinality if there exists a bijection between them.
E, N, Z and Q all have same cardinality (and proofs)
Proof that there is no bijection between N and R
Countable versus Uncountable
Power sets and their properties
Here’s What You Need to Know…