1. Streams and Lazy Evaluation in Lisp and Scheme

2. Overview Examples of using closures Delay and force Macros
Different models of expression evaluation
Lazy vs. eager evaluation
Normal vs. applicative order evaluation
Computing with streams in Scheme

3. Streams: Motivation
A stream is “a sequence of data elements made available over time”
E.g.: Streams in Unix
Also used to model objects changing over time without assignment
Describe the time-varying behavior of an object as an infinite sequence x1, x2,…
Think of the sequence as representing a function x(t)
Make the use of sequences (e.g., lists) as conventional interface more efficient

4. Example: Unix Pipes Unix pipes support stream oriented processing
E.g.: % cat mailbox | addresses | sort | uniq | more
Output from one process becomes input to another
Data flows one buffer-full at a time
Benefits:
No need to wait for one stage to finish before another can start;
storage is minimized;
works for infinite streams of data
cat
addr
sort
uniq
more

5. Delay and Force
A closure is a “function together with a referencing environment for the non-local variables”
Closures are supported by many languages, e.g., Python, javascript
Closures that are functions of no arguments are often called thunks
Thunks can be used to delay a computation and force it to be done later (in the right environment!)
Scheme has special built in functions for this: delay and force
> (define N 100)
> N
100
> (define c
(let ((N 0)) (lambda () (set! N (+ N 1)) N)))
> c
#<procedure:c>
> (c) 1 2 3

6. Delay and force (delay <exp>) ==> a “promise” to evaluate exp
(force <delayed object>) ==> evaluate the delayed object and return the result
> (define p (delay (add1 1)))
> p
#<promise:p>
> (force p) 2
#<promise!2>
> (define x (delay (print 'foo)
(print 'bar)
'done))
> (force x)
foobardone
Done
>
Note that force evaluates the delayed computation only once and remembers its value, which is returned if we force it again.

7. Delay and force
We want (delay S) to return the same function that just evaluating S would have returned
> (define x 1)
> (define p (let ((x 10)) (delay (+ x x))))
#<promise:p>
> (force p)
> 20

8. Delay and force
Delay is built into scheme, but it would have been easy to add
It’s not built into Lisp, but is easy to add
In both cases, we need to use macros
Macros provide a powerful facility to extend the languages

9. Macros In Lisp and Scheme, macros let us extend the language
They’re syntactic forms with associated defini-tion that rewrite the original forms before evaluating
E.g., like a compiler
Much of Scheme and Lisp are implemented as macros
Macros continue to be a feature that relatively unique to the Lisp family of languages

10. Simple macros in Scheme
(define-syntax-rule pattern template)
Example:
(define-syntax-rule (swap x y)
(let ([tmp x])
(set! x y)
(set! y tmp)))
Whenever the interpreter is about to eval something matching the pattern part of a syntax rule, it expands it first, then evaluates the result

11. Simple Macros > (define foo 100) > (define bar 200)
> (swap foo bar)
(let ([tmp foo]) (set! foo bar)(set! bar tmp))
> foo
200
> bar
100

12. A potential problem (let ([tmp 5] [other 6]) (swap tmp other)
(list tmp other))
A naïve expansion would be:
(let ([tmp 5] [other 6])  (let ([tmp tmp]) (set! tmp other)       (set! other tmp)) (list tmp other))
Does this return (6 5) or (5 6)?
It returns (5 6) since we have a collision of names with tmp being used in the macro expansion and in the environment

13. Scheme is clever here (let ([tmp 5] [other 6]) (swap tmp other)
(list tmp other))
(let ([tmp 5] [other 6])  (let ([tmp_1 tmp]) (set! tmp_1 other)       (set! other tmp_1)) (list tmp other))
This returns (6 5)
Scheme uses Hygienic macros
Macros whose whose expansion are guaranteed not to accidentally capture variables
Variables in the macro are renamed as needed

14. mydelay in Scheme
(define-syntax-rule (mydelay expr) (lambda ( ) expr))
> (define (myforce promise) (promise))
> (define p (mydelay (+ 1 2)))
> p
#<procedure:p
> (myforce p) 3
#<procedure:p>
Rmydelay returns a thunk: a closure function of no args
myforce just calls the thunk, which evals the delayed computation

15. Evaluation Order
Functional programs are evaluated following a reduction (or evaluation or simplification) process
There are two common ways of reducing expressions
Applicative order
≈ Eager evaluation
Normal order
≈ Lazy evaluation

16. Applicative Order
In applicative order, expressions at evaluated following the parsing tree (deeper expressions are evaluated first)
This is the evaluation order used in most programming languages
It’s the default order for Scheme, in particular
All arguments to a function or operator are evaluated before the function is applied
e.g.: (square (+ a (* b 2)))

17. Normal Order
In normal order, expressions are evaluated only when their value is needed
Hence: lazy evaluation
This is needed for some special forms
e.g., (if (< a 0) (print ‘foo) (print ‘bar))
Some languages use normal order evaluation as their default.
Sometimes more efficient than applicative order since unused computations need not be done
Can handle expressions that never converge to normal forms

18. Motivation for lazy evaluation
Goal: sum primes between two numbers
Here is a standard, traditional version using Scheme’s iteration special form, do
(define (sum-primes lo hi)
;; sum the primes between LO and HI
(do [ (sum 0)
(n lo (add1 n)) ]
[(> n hi) sum]
(if (prime? N)
(set! sum (+ sum n))
#t)))

19. Do in Lisp and Scheme (define (sum-primes lo hi) ;; sum …
(do [ (sum 0)
(n lo (add1 n)) ]
[(> n hi) sum]
(if (prime? N)
(set! sum (+ sum n))
#t)))
sum is a loop variable with initial value 0
n is a loop variable with initial value lo that’s incremented on each iteration

20. Do in Lisp and Scheme (define (sum-primes lo hi) ;; sum …
(do [ (sum 0)
(n lo (add1 n)) ]
[(> n hi) sum ]
(if (prime? N)
(set! sum (+ sum n))
#t)))
The loop terminates when (> n lo) is true
The value returned by the do is sum

21. Do in Lisp and Scheme (define (sum-primes lo hi) ;; sum …
(do [ (sum 0)
(n lo (add1 n)) ]
[(> n hi) sum]
(if (prime? N)
(set! sum (+ sum n))
#t)))
The loop body is a sequence of one or more expression to evaluate

22. Motivation: prime.ss
Here is a straightforward version using the functional paradigm:
(define (sum-primes lo hi)
; sum primes between LO and HI
(reduce + 0 (filter prime? (interval lo hi))))
(define (interval lo hi)
; return list of integers between lo and hi
(if (> lo hi) null (cons lo (interval (add1 lo) hi))))

23. Prime?
Since this function is defined inside prime?, it inherits prime’s envi-ronment, so can access n
(define (prime? n) ;; true iff n is a prime integer (define (unevenly-divides? m) ;; true iff m doesn’t evenly divide n (> (remainder n m) 0)) (andmap unevenly-divides? (interval 2 (/ n 2)))))

24. Motivation
The functional version is interesting and conceptually elegant, but inefficient
Constructing, copying and (ultimately) garbage collecting the lists adds a lot of overhead
Experienced Lisp programmers know that the best way to optimize is to eliminate unnecessary consing
Worse yet, suppose we want to know the second prime larger than a million?
(car (cdr (filter prime? (interval ))))
Can we use the idea of a stream to make this approach viable?

25. A Stream A stream is a sequence of objects, like a list
It can be an empty stream, or
It has a first element and a stream of remaining elements
However, the remaining elements will only be computed (materialized) as needed
Just in time computing, as it were
So, we can have a stream of (potential) infinite length and use only a part of it without having to materialize it all

26. Streams in Lisp and Scheme
We can push features for streams into a programming language.
Makes some approaches to computation simple and elegant
The closure mechanism used to implement these features.
Can formulate programs elegantly as sequence manipulators while attaining the efficiency of incremental computation.

27. Streams in Lisp/Scheme
A stream is like a list, so we’ll need construc-tors (~cons), and accessors (~ car, cdr) and a test for the empty stream (~ null?).
We’ll call them:
SNIL: represents the empty stream
(SCONS X S): create a stream whose first element is X and whose remaining elements are the stream S
(SCAR S): returns first element of the stream
(SCDR S): returns remaining elements of the stream
(SNULL? S): returns true iff S is the empty stream

28. Streams: key ideas
Write scons to delay computation needed to produce the stream until value is needed
and only as little of the computation as needed
Access parts of a stream with scar & scdr, so they may have to force the computation
We’ll always compute the first element of a stream and delay actually computing the rest of a stream until needed by some call to scdr
Two important functions to base this on: delay & force

29. Streams using DELAY and FORCE
;; empty stream is just null (define sempty empty) (define (snull? stream) (null? stream)) ;; scons delays evaluating the cdr (define-syntax-rule (scons first rest) (cons first (delay rest))) ;; scar is just car (define (scar stream) (car stream)) ;; scdr forces the computation to be done first (define (scdr stream) (force (cdr stream)))

30. Consider the interval function
Recall the interval function:
(define (interval lo hi)
; return a list of the integers between lo and hi
(if (> lo hi) null (cons lo (interval (add1 lo) hi))))
Now imagine evaluating (interval 1 3):
(interval 1 3)
(cons 1 (interval 2 3))
(cons 1 (cons 2 (interval 3 3)))
(cons 1 (cons 2 (cons 3 (interval 4 3)))
(cons 1 (cons 2 (cons 3 ‘())))
 (1 2 3)

31. … and the stream version
Here’s a stream version of the interval function:
(define (sinterval lo hi)
; return a stream of integers between lo and hi
(if (> lo hi)
sempty
(scons lo (sinterval (add1 lo) hi))))
Now imagine evaluating (sinterval 1 3):
(sinterval 1 3)
(scons 1 . #<procedure>))

32. Stream versions of list functions
(define (snth n stream) (if (= n 0) (scar stream) (snth (sub1 n) (scdr stream)))) (define (smap f stream) (if (snull? stream) sempty (scons (f (scar stream)) (smap f (scdr stream))))) (define (sfilter f stream) (cond ((snull? stream) sempty) ((f (scar stream)) (scons (scar stream) (sfilter f (scdr stream)))) (else (sfilter f (scdr stream)))))

33. Applicative vs. Normal order evaluation
(car (cdr
(filter prime? (interval ))))
(scar
(scdr
(sfilter prime? (interval ))))
Both return the second prime larger than 10 (which is 13)
With lists it takes about operations
With streams about three

34. Infinite streams We can easily define infinite streams
(define (integers-from n)
(scons n (integers-from (+ n 1))))
(define integers (integers-from 0))

35. Infinite streams
(define (sadd s1 s2) ; returns a stream which is the pair-wise ; sum of input streams S1 and S2. (cond ((snull? s1) s2) ((snull? s2) s1) (else (scons (+ (scar s1) (scar s2)) (sadd (scdr s1)(scdr s2))))))

36. Infinite streams 2 The streams are computed as needed
This works even with infinite streams
Using sadd we define an infinite stream of ones:
(define ones (scons 1 ones))
An infinite stream of the positive integers:
(define integers (scons 1 (sadd ones integers)))
The streams are computed as needed
(snth 10 integers) => 11

37. Sieve of Eratosthenes
Eratosthenes (air-uh-TOS-thuh-neez), a Greek mathematician and astrono- mer, was head librarian of the Library at Alexandria, estimated the Earth’s circumference to within 200 miles and derived a clever algorithm for computing the primes less than N
Write a consecutive list of integers from 2 to N
Find smallest number not marked as prime and not crossed out. Mark it prime and cross out its multiples
Goto 2

38. Finding all the primes XX 5 100 99 98 97 96 95 94 93 92 91 90 89 88 8786 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 60 69 68 67 66 65 64 63 62 61 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 XX X 3 XX 7 XX X 2

39. Scheme sieve (define (sieve S) ; run the sieve of Eratosthenes
(scons (scar S)
(sieve
(sfilter
(lambda (x) (> (modulo x (scar S)) 0))
(scdr S)))))
(define primes (sieve (scdr integers)))

40. Summary Scheme’s functional foundation shows its power here
Closures and macros let us define delay and force
Which allows us to handle large, even infinite streams easily
Streams and stream functions are pre-defined in the racket streams package
Other languages, including Python, also let us do this, but in slightly different ways
See generators and yield