Pipes & Filters Architecture Pattern Source: Pattern-Oriented Software Architecture, Vol. 1, Buschmann, et al

Problem You are designing a system that needs to perform several transformations on some data The transformations can be performed sequentially It should be easy to: –Reorder the sequence of the transformations –Add new kinds of transformations –Remove transformations There are different sources of input data (files, network, programs) There are different destinations for output data (files, network, programs)

Solution Organize the system using Pipes & Filters Input data originates from Data Source components Output data is consumed by Data Sink components Each transformation required to convert the input into the output is implemented by a Filter component Data Sources, Filters, and Data Sinks arranged in a pipeline Pipes connect adjacent components, the output of one component becoming the input to the next component

Solution Sources, Filters, and Sinks can be Active or Passive Active components run on their own thread of control Passive components execute only when invoked, directly or indirectly, by an Active component

Dynamic Behavior : Scenario I Active Source / Passive Filter / Passive Sink

Dynamic Behavior : Scenario II Passive Source / Passive Filter / Active Sink

Dynamic Behavior : Scenario III Passive Source / Active Filter / Passive Sink

Dynamic Behavior : Scenario IV Passive Source / Multiple Active Filters / Passive Sink Buffering Pipe

Implementation Divide the system into a sequence of processing stages Define the data format to be passed along each pipe Decide how to implement each pipe connection –The simplest Pipe is direct Read/Write method calls between adjacent filters –If buffering or synchronization is required between filters, actual Pipe objects will be needed to perform these functions

Implementation Design and implement the filters –Passive filters can be implemented as regular methods –Active filters can be implemented as processes or threads Design the error handling –In pipelines with one active component, standard error handling mechanisms can be used (exceptions, return codes, etc.) –In pipelines with multiple active components, how will an error in one thread of control become visible to other threads? –If an error occurs, we could: Tear down the entire pipeline and restart it from scratch, or Restart only the parts that failed and resynchronize the pipeline Setup the pipeline

Known Uses: UNIX Command Pipelines cat fall.txt win.txt | sort | gzip | mail

Known Uses: Image Processing

Known Uses: Compilers

Consequences Very flexible –Filters can be reused and recombined in arbitrary ways –Individual filters can be easily replaced (e.g., plug in a different kind of compression) No intermediate files necessary between processing stages Benefits from efficiencies inherent in parallel processing (multiple active components) Some filters don’t produce any output until they've consumed all of their input (e.g., sort), which is not conducive to parallelism Data transfer and context switching between processes and threads can be expensive