PRACTICAL COMMON LISP Peter Seibel 1

CHAPTER 11 COLLECTIONS 2

COLLECTIONS Common Lisp provides standard data types that collect multiple values into a single object. The basic collection types An integer-indexed array type Including arrays, lists, or tuples A table type Including hash tables, associative arrays, maps, and dictionaries Lisp is famous for its list data structure, but lists are not the only collection type in Lisp. This chapter will focus on Common Lisp’s other collection type: vectors. 3

VECTORS Vectors are Common Lisp’s basic integer-indexed collection, and they come in two flavors. Fixed-size vectors Resizable vectors Fixed-size vectors: the function VECTOR and MAKE-ARRAY VECTOR (vector) → #() (vector 1) → #(1) (vector 1 2) → #(1 2) #(...) syntax can be used to include literal vectors in the code, but as the effects of modifying literal objects aren’t defined. MAKE-ARRAY (make-array 5 :initial-element nil) → #(NIL NIL NIL NIL NIL) MAKE-ARRAY can be used to create arrays of any dimensionality as well as both fixed-size and resizable vectors. The one required argument to MAKE-ARRAY is a list containing the dimensions of the array. 4

VECTORS MAKE-ARRAY is also the function to use to make a resizable vector. A resizable vector is a slightly more complicated object than a fixed-size vector. A resizable vector keeps track of the number of elements actually stored in the vector. This number is stored in the vector’s fill pointer. To make a vector with a fill pointer, you pass MAKE-ARRAY a :fill-pointer argument. For instance, the following call to MAKE-ARRAY makes a vector with room for five elements; but it looks empty because the fill pointer is zero: (make-array 5 :fill-pointer 0) → #() (make-array 5 :fill-pointer 5) → #(NIL NIL NIL NIL NIL) 5

VECTORS VECTOR-PUSH can be used to add an element to the end of a resizable vector. VECTOR-PUSH adds the element at the current value of the fill pointer and then increments the fill pointer by one, returning the index where the new element was added. The function VECTOR-POP returns the most recently pushed item, decrementing the fill pointer in the process. For example, (defparameter *x* (make-array 5 :fill-pointer 0)) (vector-push 'a *x*) → 0*x* → #(A) (vector-push 'b *x*) → 1*x* → #(A B) (vector-push 'c *x*) → 2*x* → #(A B C) (vector-pop *x*) → C*x* → #(A B) (vector-pop *x*) → B*x* → #(A) (vector-pop *x*) → A*x* → #() The vector *x* can hold at most five elements. 6

VECTORS To make an arbitrarily resizable vector, we need to pass MAKE- ARRAY another keyword argument: :adjustable. (make-array 5 :fill-pointer 0 :adjustable t) → #() This call makes an adjustable vector whose underlying memory can be resized as needed. To add elements to an adjustable vector, VECTOR-PUSH- EXTEND will automatically expand the array if we try to push an element onto a full vector—one whose fill pointer is equal to the size of the underlying storage. > (defparameter *x* (make-array 5 :fill-pointer 0 :adjustable t)) *X* > (vector-push-extend 'c *x*) 0 > *x* #(C) 7

SUBTYPES OF VECTOR It’s also possible to create specialized vectors that are restricted to holding certain types of elements. One reason to use specialized vectors is they may be stored more compactly and can provide slightly faster access to their elements than general vectors. For example: strings are vectors specialized to hold characters. (make-array 5 :fill-pointer 0 :adjustable t :element-type 'character) → "" (setf *X* (make-array 5 :initial-element #\a :fill-pointer 3 :adjustable t :element-type 'character) → "aaa" MAKE-ARRAY can be used to make resizable strings by adding another keyword argument, :element-type. This argument takes a type descriptor. 8

VECTORS AS SEQUENCES Vectors and lists are the two concrete subtypes of the abstract type sequence. The two most basic sequence functions are LENGTH: returns the length of a sequence, and ELT: accesses individual elements via an integer index. For example: (defparameter *x* (vector 1 2 3)) (length *x*) → 3 (elt *x* 0) → 1 (elt *x* 1) → 2 (elt *x* 2) → 3 (elt *x* 3) → error ELT is a SETFable place. (setf (elt *x* 0) 10) *x* → #(10 2 3) 9

SEQUENCE ITERATING FUNCTIONS Common Lisp provides a large library of sequence functions. One group of sequence functions allows us to express certain operations on sequences such as finding or filtering specific elements without writing explicit loops. 10

SEQUENCE ITERATING FUNCTIONS For examples: (count 1 #( )) → 3 (remove 1 #( )) → #( ) (remove 1 '( )) → ( ) (remove #\a "foobarbaz") → "foobrbz" (substitute 10 1 #( )) → #( ) (substitute 10 1 '( )) → ( ) (substitute #\x #\b "foobarbaz") → "fooxarxaz" (find 1 #( )) → 1 (find 10 #( )) → NIL (position 1 #( )) → 0 Note REMOVE and SUBSTITUTE always return a sequence of the same type as their sequence argument. 11

SEQUENCE ITERATING FUNCTIONS We can modify the behavior of these five functions in a variety of ways using keyword arguments. For example: The :test keyword can be used to pass a function that accepts two arguments and returns a boolean. (count "foo" #("foo" "bar" "baz") :test #'string=) → 1 The :key keyword can pass a one-argument function to be called on each element of the sequence to extract a key value, which will then be compared to the item in the place of the element itself. (find 'c #((a 10) (b 20) (c 30) (d 40)) :key #'first) → (C 30) To limit the effects of these functions to a particular subsequence of the sequence argument, we can provide bounding indices with :start and :end arguments. 12

SEQUENCE ITERATING FUNCTIONS If a non-NIL :from-end argument is provided, then the elements of the sequence will be examined in reverse order. :from-end can affect the results of only FIND and POSITION. For instance: (find 'a #((a 10) (b 20) (a 30) (b 40)) :key #'first) → (A 10) (find 'a #((a 10) (b 20) (a 30) (b 40)) :key #'first :from-end t) → (A 30) :from-end can affect REMOVE and SUBSTITUTE in conjunction with another keyword parameter, :count. :count is used to specify how many elements to remove or substitute. (remove #\a "foobarbaz" :count 1) → "foobrbaz" (remove #\a "foobarbaz" :count 1 :from-end t) → "foobarbz" 13

SEQUENCE ITERATING FUNCTIONS While :from-end can’t change the results of the COUNT function, it does affect the order the elements are passed to any :test and :key functions, which could possibly have side effects. For example: CL-USER> (defparameter *v* #((a 10) (b 20) (a 30) (b 40))) *V* CL-USER> (defun verbose-first (x) (format t "Looking at ~s~%" x) (first x)) VERBOSE-FIRST CL-USER> (count 'a *v* :key #'verbose-first) Looking at (A 10) Looking at (B 20) Looking at (A 30) Looking at (B 40) 2 CL-USER> (count 'a *v* :key #'verbose-first :from-end t) Looking at (B 40) Looking at (A 30) Looking at (B 20) Looking at (A 10) 2 14

SEQUENCE ITERATING FUNCTIONS Table 11-2 summarizes these arguments. 15

HIGHER-ORDER FUNCTION VARIANTS Common Lisp provides two higher-order function variants that take a function to be called on each element of the sequence. One set of variants are named the same as the basic function with an -IF appended. These functions count, find, remove, and substitute elements of the sequence for which the function argument returns true. The other set of variants are named with an -IF-NOT suffix and count, find, remove, and substitute elements for which the function argument does not return true. (count-if #'evenp #( )) → 2 (count-if-not #'evenp #( )) → 3 (position-if #'digit-char-p "abcd0001") → 4 (remove-if-not #'(lambda (x) (char= (elt x 0) #\f)) #("foo" "bar" "baz" "foom")) → #("foo" "foom") 16

HIGHER-ORDER FUNCTION VARIANTS With a :key argument, the value extracted by the :key function is passed to the function instead of the actual element. (count-if #'evenp #((1 a) (2 b) (3 c) (4 d) (5 e)) :key #'first) → 2 (count-if-not #'evenp #((1 a) (2 b) (3 c) (4 d) (5 e)) :key #'first) → 3 (remove-if-not #'alpha-char-p #("foo" "bar" "1baz") :key #'(lambda (x) (elt x 0))) → #("foo" "bar") REMOVE-DUPLICATES has only one required argument from which it removes all but one instance of each duplicated element. For example: (remove-duplicates #( )) → #( ) 17

WHOLE SEQUENCE MANIPULATIONS Some functions perform operations on a whole sequence (or sequences) at a time. The CONCATENATE function creates a new sequence containing the concatenation of any number of sequences. CONCATENATE must be told explicitly what kind of sequence to produce in case the arguments are of different types. Its first argument is a type descriptor, for example, VECTOR, LIST, or STRING. (concatenate 'vector #(1 2 3) '(4 5 6)) → #( ) (concatenate 'list #(1 2 3) '(4 5 6)) → ( ) (concatenate 'string "abc" '(#\d #\e #\f)) → "abcdef" 18

SORTING AND MERGING The functions SORT and STABLE-SORT provide two ways of sorting a sequence. They both take a sequence and a two-argument predicate and return a sorted version of the sequence. (sort (vector "foo" "bar" "baz") #'string<) → #("bar" "baz" "foo") The difference is that STABLE-SORT is guaranteed to not reorder any elements considered equivalent by the predicate while SORT guarantees only that the result is sorted and may reorder equivalent elements. SORT and STABLE-SORT will destroy the sequence in the course of sorting it. (setf my-sequence (sort my-sequence #'string<)) 19

SORTING AND MERGING MERGE: takes two sequences and a predicate and returns a sequence produced by merging the two sequences, according to the predicate. It’s related to the two sorting functions in that if each sequence is already sorted by the same predicate, then the sequence returned by MERGE will also be sorted. MERGE takes a :key argument. The first argument to MERGE must be a type descriptor specifying the type of sequence to produce. (merge 'vector #(1 3 5) #(2 4 6) #'<) → #( ) (merge 'list #(1 3 5) #(2 4 6) #'<) → ( ) (merge 'vector #(5 3) #( ) #'<) → #( ) 20

SUBSEQUENCE MANIPULATIONS Another set of functions can manipulate subsequences of existing sequences. SUBSEQ: extracts a subsequence starting at a particular index and continuing to a particular ending index or the end of the sequence. For instance: (subseq "foobarbaz" 3) → "barbaz" (subseq "foobarbaz" 3 6) → "bar" SUBSEQ is SETFable. (defparameter *x* (copy-seq "foobarbaz")) (setf (subseq *x* 3 6) "xxx") ; subsequence and new value are same length *x* → "fooxxxbaz" (setf (subseq *x* 3 6) "abcd") ; new value too long, extra character ignored *x* → "fooabcbaz" (setf (subseq *x* 3 6) "xx") ; new value too short, only two characters changed *x* → "fooxxcbaz" 21

SUBSEQUENCE MANIPULATIONS SEARCH: to find a subsequence within a sequence. Like POSITION except the first argument is a sequence rather than a single item. (position #\b "foobarbaz") → 3 (search "bar" "foobarbaz") → 3 MISMATCH: to find where two sequences with a common prefix first diverge. It takes two sequences and returns the index of the first pair of mismatched elements. (mismatch "foobarbaz" "foom") → 3 It returns NIL if the strings match. MISMATCH takes many of the standard keyword arguments :key :test :start1, :end1, :start2, and :end2 A :from-end argument of T specifies the sequences should be searched in reverse order. (mismatch "foobar" "bar" :from-end t) → 3 22

SEQUENCE PREDICATES EVERY, SOME, NOTANY, and NOTEVERY functions iterate over sequences testing a boolean predicate ( 述語 ). The first argument to all these functions is the predicate, and the remaining arguments are sequences. For examples: (every #'evenp #( )) → NIL (some #'evenp #( )) → T (notany #'evenp #( )) → NIL (notevery #'evenp #( )) → T These calls compare elements of two sequences pairwise: (every #'> #( ) #( )) → NIL (some #'> #( ) #( )) → T (notany #'> #( ) #( )) → NIL (notevery #'> #( ) #( )) → T 23

SEQUENCE MAPPING FUNCTIONS MAP: takes an n-argument function and n sequences and returns a new sequence containing the result of applying the function to subsequent elements of the sequences. (map 'vector #'* #( ) #( )) → #( ) (map 'vector #'* #( ) #( ) #( )) → #( ) MAP-INTO places the results into a sequence passed as the first argument. This sequence can be the same as one of the sequences providing values for the function. [8]> (defparameter v (vector 1 2 3)) V [9]> V #(1 2 3) [10]> (setf v (map-into v #'1+ v)) #(2 3 4) [11]> V #(2 3 4) 24

SEQUENCE MAPPING FUNCTIONS For instance, to sum several vectors—a, b, and c—into one: (map-into a #'+ a b c) Break 10 [14]> (defparameter a (vector 1 2 3)) A Break 10 [14]> (defparameter b (vector 1 2 3)) B Break 10 [14]> (defparameter c (vector 1 2 3)) C Break 10 [14]> (map-into a #'+ a b c) #(3 6 9) Break 10 [14]> a #(3 6 9) Break 10 [14]> b #(1 2 3) Break 10 [14]> c #(1 2 3) 25

SEQUENCE MAPPING FUNCTIONS REDUCE maps over a single sequence, applying a two-argument function first to the first two elements of the sequence and then to the value returned by the function and subsequent elements of the sequence. The following expression sums the numbers from one to ten: (reduce #'+ #( )) → 55 (reduce #'intersection ‘((a b c d) (a b e f) (e f b))) → (B) REDUCE is a useful function, whenever we need to distill ( 精鍊 ) a sequence down to a single value. For instance, to find the maximum value in a sequence of numbers, we can write (reduce #'max numbers). 26

EXERCISE (defun merge-sort (list) (if (small list) list (merge-lists (merge-sort (left-half list)) (merge-sort (right-half list))))) (defun small (list) (or (null list) (null (cdr list)))) (defun right-half (list) (last list (ceiling (/ (length list) 2)))) (defun left-half (list) (ldiff list (right-half list))) (defun merge-lists (list1 list2) (merge ‘list list1 list2 #'<)) Can you explain the meaning of each function? 27