F RANZ I NC. An In-Depth Look at Simple Streams By Duane Rettig October, 2002

F RANZ I NC. What is a Lisp Stream? Lisp stream external device internal device

F RANZ I NC. An In-Depth Look at Simple Streams Simple Streams Design Goals What is a Simple Stream? Simple-stream Concepts –Opening, closing, blocking, reading, writing Character Strategies Common Windows Rewrite Further changes in next version (references distributed throughout presentation)

F RANZ I NC. Design Goals CL compliance Bivalence Thin Strategy layer that is uniform Extensibility via specializations/mixins and encapsulations Minimal decision points in critical strategy code Minimal calls to generic-functions External-format ready for 8 and 16 bit lisps

F RANZ I NC. Gray vs. Simple Stream Object functionality buffer API call Strategy Gray Stream Simple Stream Device interface

F RANZ I NC. Dual-channel stream Two octet buffers External-format character translation Ext. format Device methods Out buffer API In buffer External device

F RANZ I NC. Si ngle-channel stream One octet buffer External-format character translation Ext. format Device methods buffer API direction Internal or External device

F RANZ I NC. String stream Zero, one, or two string buffers No external-format translations Plugable character/buffer strategies Device methods Out buffer API In buffer Internal device

F RANZ I NC. Opening a Stream Creates or reuses a stream object Makes or retains a connection to a device Initializes character strategies Marks the stream as open

F RANZ I NC. Open open make-instance (gf)... shared-initialize (gf) shared-initialize :after [Method] device-open [Method]

F RANZ I NC. (Open): Simple encapsulation (setq bun (open "sesame")) (setq big-mac (make-instance 'all-beef :input-handle bun...)) bunbig-mac all-beef sesame

F RANZ I NC. (Open): shared-initialize (defmethod shared-initialize :after ((stream simple-stream) slots &rest initargs) (declare (ignore slots)) (unless (device-open stream initargs) (device-close stream t)) stream)

F RANZ I NC. (Open): device-open At device-open time: –All instance slots have been initialized –connection has been made or will be made by device-open When device-open returns: –If non-nil, the stream is ready for appropriate actions –If nil, then the open has failed.

F RANZ I NC. (Open): device-open device-open must: –Ensure a connection before returning –Ensure buffers are in place –Initialize/reinitialize pointers –Return nil for failure, or non-nil after success device-open must not: –Assume that the stream was closed –close the stream

F RANZ I NC. (Open): subclass example (defclass file-with-header (file-simple-stream) ((header :initform nil :accessor file-header-info))) (defmethod device-open ((stream file-with-header) options) (declare (ignore options)) (let ((success (call-next-method))) (when success (setf (file-header-info stream) (read-file-header stream)) t)))

F RANZ I NC. (Open): device-open :before methods In future releases, for string-simple-streams: –A catch-all primary method returns true for an open stream. –Strategy installation functions are directional and do not override. install-string-character-strategy deprecated, replaced by install-string-{input,output}-character-strategy –All :before methods fire according to CPL, and may shadow default actions.

F RANZ I NC. (Open): device-open: string-input (future) (defmethod device-open :before ((stream string-input-simple-stream) options) (with-stream-class (string-input-simple-stream stream) (let ((string (getf options :string))) (when (and string (null (sm buffer stream))) (let ((start (getf options :start)) (end (or (getf options :end) (length string)))) (setf (sm buffer stream) string (sm buffpos stream) start (sm buffer-ptr stream) end))))) (install-string-input-character-strategy stream) (add-stream-instance-flags stream :string :input :simple))

F RANZ I NC. (Open): device-open: string-output (future) (defmethod device-open :before ((stream string-output-simple-stream) options) (with-stream-class (string-output-simple-stream stream) (unless (sm out-buffer stream) (let ((string (getf options :string))) (if string (setf (sm out-buffer stream) string (sm max-out-pos stream) (length string)) (let ((buflen (max (device-buffer-length stream) 16))) (setf (sm out-buffer stream) (make-string buflen) (sm max-out-pos stream) buflen))))) (unless (sm control-out stream) (setf (sm control-out stream) *std-control-out-table*))) (install-string-output-character-strategy stream) (add-stream-instance-flags stream :string :output :simple))

F RANZ I NC. Closing a Stream Flushes output if any. Breaks connection to device. Secures against accidental future operations. Does not change-class. (defmethod close ((stream simple-stream) &key abort) (device-close stream abort))

F RANZ I NC. Close (defmethod device-close :around ((stream simple-stream) abort) (let (res) (when (pseudo::open-stream-p stream) (unwind-protect (progn (when (output-stream-p stream) (ignore-errors (if abort (clear-output stream) (force-output stream)))) (setq res (call-next-method))) (without-interrupts (pseudo::unset-open-flags stream) (setf (stream-input-handle stream) nil (stream-output-handle stream) nil)) (setf (stream-external-format stream) (find-external-format :void)) res))))

F RANZ I NC. (Close): device-close device-close should: –flush all data (unless aborting) –disconnect handles of any encapsulated streams –call lower-levels to close as necessary device-close should not: –operate on a closed stream –close an encapsulated stream

F RANZ I NC. Blocking Issues with listen –read direction only –covers character availability and not blocking –assumes character as the basic data unit stream-listen (carried over from Gray streams) read-no-hang-p and write-no-hang-p

F RANZ I NC. Blocking styles Non-blocking Blocking B/NB (blocking, then non-blocking) Character: One element Octet: One element

F RANZ I NC. Reading and Writing Mostly symmetrical Device methods obey B/NB discipline

F RANZ I NC. Basic read strategy (block read (when (>= buffpos buffer-ptr) (let ((res (device-read stream nil 0 buffpos blocking))) (when (< res 0) (pseudo::do-error-handling)) (when (= res 0) (if blocking (pseudo::do-eof-handling) (return-from read nil))) (setq buffer-ptr res buffpos 0))) (prog1 (aref buffer buffpos) (incf buffpos)))))

F RANZ I NC. Basic write strategy (block write (when (>= out-pos max-out-pos) (when (> out-pos 0) (erroring-device-write-whole stream nil 0 out-pos t)) (setq out-pos 0)) (setf (aref buffer out-pos) value) (incf out-pos))

F RANZ I NC. Implementing force-output, finish-output Both are implemented using device-write blocking argument determines whether force (nil) or finish (t) device-write buffer argument is :flush (future)

F RANZ I NC. Reading and Writing Sequences read-sequence and write-sequence –are width sensitive and by-element –require blocking semantics –do not return a count read-vector and write-vector –are octet-based –employ B/NB semantics –return a count, 0, or error code

F RANZ I NC. Deprecated generic function: device-extend History and relationship to device-read and device-write –structural differences –purity of reference Problems –one generic-function for two directions –inconsistent interface

F RANZ I NC. Replacement of device-extend Actions :input and :input-check become blocking argument to device-read Actions :output and :output-check become blocking argument to device-write Extra actions come through via device-read and device-write buffer argument

F RANZ I NC. Record orientation Via device-finish-record gf –dual-channel input –string output

F RANZ I NC. Other generic-functions device-buffer-length device-clear-input device-clear-output device-file-length device-file-position

F RANZ I NC. Strategies What is a strategy’s purpose? –To satisfy the high-level requirements of a specified behavior in a uniform manner. Octet strategies are non-programmable. –read-byte, write-byte are cast in concrete. Character strategies are programmable...

F RANZ I NC. Character strategies Programmable, replaceable. External-formats use template functions for dual-channel and single-channel streams. String-streams use simple strategy functions. (future): Subclassed strategy functions can shadow more general ones. Occupy “joint” slots in the stream.

F RANZ I NC. Example strategy set for *terminal-io* cl-user(5): :i #x711e057a = #<terminal-simple-stream [initial terminal io] fd #x711e057a> 0 Class > # 1 j-unread-char -> # 2 j-write-chars -> # 3 j-write-char -> # 4 j-read-chars -> # 5 j-read-char --> # 6 j-listen -----> # src-position-table -> The symbol nil cl-user(6):

F RANZ I NC. Example strategy set for string-output cl-user(9): :i #x719cb5e2 = #<string-output-simple-stream "" pos #x719cb5e2> 0 Class > # 1 j-unread-char -> The symbol nil 2 j-write-chars -> # 3 j-write-char -> # 4 j-read-chars -> The symbol nil 5 j-read-char --> The symbol nil 6 j-listen -----> The symbol nil out-buffer ---> A simple-string (4096) that starts "xxxxxxxxxxxxxxxx” cl-user(10):

F RANZ I NC. Example strategy set for string-input cl-user(14): :i #x719e74aa = #<string-input-simple-stream "abc" pos #x719e74aa> 0 Class > # 1 j-unread-char -> # 2 j-write-chars -> The symbol nil 3 j-write-char -> The symbol nil 4 j-read-chars -> # 5 j-read-char --> # 6 j-listen -----> # src-position-table -> The symbol nil cl-user(15):

F RANZ I NC. j-read-char Args: stream eof-error-p eof-value blocking Implements read-char and read-char-no-hang functionality directly (after argument resolution) blocking argument is nil/true dichotomy

F RANZ I NC. j-write-char Args: character stream Implements write-char directly (after argument resolution) Writing nil as the character argument will flush an external-format’s output state.

F RANZ I NC. j-read-chars Args: stream string search start end blocking Implements read-sequence, read-line blocking argument is nil, :bnb, or true

F RANZ I NC. j-write-chars Args: stream string start end Implements write-sequence, write-string Always blocks (for now)

F RANZ I NC. j-listen Args: stream Implements stream-listen directly Must have a complete character or an error condition in order to return true.

F RANZ I NC. j-unread-char Args: stream relaxed May need to unread multiple characters as part of a composing or encapsulating character. relaxed allows non-error unreads beyond the first unread.

F RANZ I NC. Taking strategies beyond ANS Other strategies can be created for operations that don’t fit into CL standard functionalities. Example: one-buffer ring fifo queue

F RANZ I NC. Common Windows low-level rewrite Source code comparison between Gray and simple streams Demo