SS ZG653Second Semester, Topic Architectural Patterns Pipe and Filter

SS ZG653Second Semester, PIPES & FILTERS Mud to Structure

SS ZG653Second Semester, Pipes and Filters The pipes and filters architectural pattern provides a structure for systems that process a stream of data. Each processing step is encapsulated in a filter component. Data is passed through pipes between adjacent filters. Recombining filters allows you to build families of related systems.

SS ZG653Second Semester, Pipes and Filters Type of Data Flow Architecture Filter is a component and pipe is a connector Filter has interfaces from which a set of inputs can flow in and a set of outputs can flow out. Incremental transformation of data by successive components. All data does not need to be processed for next filter to start working. Any set of filters may be combined in any order, although reasonable semantics are not guaranteed by this style.

SS ZG653Second Semester, Pipes and Filters Filter Independent entities Does not share state with other filters. Do not know the identity to upstream and downstream filters. Pipes Stateless data stream Source end feeds filter input and sink receives output.

SS ZG653Second Semester, Pipes and Filters

SS ZG653Second Semester, Pipes and Filters Compilers Various “translation” systems

SS ZG653Second Semester, Pipes and Filters – 3 part schema Pattern ContextProcessing Data Streams ProblemSystem that must process or transform a stream of input data. Expect flexibility by exchanging or reordering the processing steps. Possible to build family of such systems. Future enhancements – exchange processing steps or recombination Small processing steps – aid reuse Non adjacent processing steps do not share information Different sources of data exist Store final result in various ways Explicit storage of intermediate results should be transparent to users – is error prone if done by users Multiprocessing the steps SolutionPipes and filters – data source to data sink

SS ZG653Second Semester, Pipes and Filters Filter component is a processing units of the pipeline Enriches, refines or transforms its input data Enriches – computing and adding information Refine – concentrating or extracting information Transforms – delivering data in some other representation Class Filter Responsibility Gets inputs data Performs a function on its input data Supplies output data Collaborators Pipe

SS ZG653Second Semester, Pipes and Filters Pipe denotes connection between filters Data source and first filter Last filter and Data sink Synchronises two active component Class Pipe Responsibility Transfers data Buffers data Synchronizes active neighbors Collaborators Data Source Data Sink Filter

SS ZG653Second Semester, Pipes and Filters Data source represents input to the system Sequence of data of the same structure or type Class Data Source Responsibility Delivers input to processing pipeline Collaborators Pipe

SS ZG653Second Semester, Pipes and Filters Data sink collects results from end of the pipeline Active: pulls results from preceding processing stage Passive: allows preceding filter to push or write the results into it Class Data Sink Responsibility Consumes output Collaborators Pipe

SS ZG653Second Semester, Dynamics Scenario I – Push pipeline [Activity starts with the Data source] – Filter activity started by writing data to the filters – Passive Filter [Use direct calls to the adjacent pipeline]

SS ZG653Second Semester, Dynamics Scenario II – Pull pipeline – Control flow is started by the data sink calling for data

SS ZG653Second Semester, Dynamics Scenario III – Push-pull mixed pipeline

SS ZG653Second Semester, Dynamics Scenario IV – All filters actively pull, compute and push data in a loop – Each filter runs its own thread of control – Filters are synchronised by buffering pipe between them

SS ZG653Second Semester, Implementation #Steps 1Divide the system’s task into a sequence of processing stages 2Define the data format to be passed along each pipe 3Decide how to implement each pipe connection 4Design and implement the filters 5Design the error handling 6Set up the processing pipeline

SS ZG653Second Semester, : Divide the systems tasks into sequence of processing stages Each stage must depend on the output of the predecessor All stages conceptually connected by data flow

SS ZG653Second Semester, : Define data format to be passed along each pipe Define a uniform format results in the highest flexibility because it makes recombination of filters easy Define the end of input marking

SS ZG653Second Semester, : Decide how to implement each pipe connection Decision determines active or passive filter Using a separate pipe mechanism that synchronises adjacent active filters provide a more flexible solution

SS ZG653Second Semester, : Design and implement the filters Depends on – Task it must perform – Adjacent pipe Active or Passive filters Performance and tradeoffs Filter reuse

SS ZG653Second Semester, : Design the error handling Never neglect error handling No global state shared; error handling hard to address Strategies in case of error – depend on domain

SS ZG653Second Semester, : Setup the processing pipeline Use of standardised main program Use of user inputs or choice

SS ZG653Second Semester, Variants Tee and Join pipeline – Filters with more then one input and/or more than one output

SS ZG653Second Semester, Benefits No intermediate files necessary, but possible Flexibility by filter exchange Flexibility by recombination Reuse of filter components Rapid prototyping of pipelines Efficiency by parallel processing

SS ZG653Second Semester, Liabilities Sharing state information is expensive or inflexible Data transformation overhead Error handling

SS ZG653Second Semester, Known Uses Find (at least 2) more popular uses and document them