Network Computing Laboratory A programming framework for Stream Synthesizing Service

Network Computing Laboratory | 2 Korea Advanced Institute of Science and Technology Outline Motivation expected applications Scenario An example application Challenges of application development Approach What to do

Network Computing Laboratory | 3 Korea Advanced Institute of Science and Technology Motivation Stream Synthesizing Services (S 3 ) Receives data streams from heterogeneous sources (context, rate..) Continuously runs over the streaming data Includes complex logic that requires operations beyond declarative queries E.g. Location-based services (LBS), ITS, … Continuously generated as a form of streams Stream Synthesizing Services Stream Synthesizing Services RFID GPS Sensor Net Mobile devices

Network Computing Laboratory | 4 Korea Advanced Institute of Science and Technology Scenario Scenario: A Command Control System Context-aware target prioritization & fire control CCS Locations of adversary objects Radar, sensor net, sonar Armor status, Vehicle condition, Health condition, Location Monitor Command

Network Computing Laboratory | 5 Korea Advanced Institute of Science and Technology At each level of the hierarchy A unique algorithm for target partitioning & fire allocation is implemented Target partitioning & fire allocation changes as Positions of objects change Data : Location stream, Processing : speed & orientation detection Available fire power Data : Physical condition of weapons & soldiers Data : biomedical data, Processing : expert system for diagnosis Scenario Monitor Command

Network Computing Laboratory | 6 Korea Advanced Institute of Science and Technology Challenges of S 3 development(1/3) Control flow management is inherently complex Threaded model, message-passing model are both inappropriate for the development Thread?  Control flow of a tuple is not sequential (condition 의 만족을 위해 waiting 하는 경우가 많다.) E.g. Tuple/Time windows are placed between program modules Thread 로 구현할 경우  logic 과 별개로 condition 의 충족 여부를 checking 하는 복잡 함이 있다. (Example) Message-Passing?  An event-handler may be triggered by a synthesized event Synthesized event: A packaged event formed with events from multiple sources N 개를 묶어서 할 때 뭐가 어렵냐 N 개를 못묶는지 ? Enabling Flexible control flow expression is required

Network Computing Laboratory | 7 Korea Advanced Institute of Science and Technology Synthesized stream 만드는 방법 Multiple source 로부터 나온 stream 을 묶어서 새 input stream 을 만들 때.. 묶는 방법 나누는 법 ??

Network Computing Laboratory | 8 Korea Advanced Institute of Science and Technology.. weather traffic smoothingaggregation Traffic status S A R A A M aggregation Black box 가 낫다

Network Computing Laboratory | 9 Korea Advanced Institute of Science and Technology Challenges of S 3 development (2/3) Before running the logic, complex processing of raw data streams is required Since values and data rates change dynamically over time Operations on data from a single source Buffering: due to the fluctuating data rate Windowing: logics require aggregated data often (time, tuple window) Filtering: logics require selected data according to values Operations on data from multiple sources Scheduling data update: app semantic, performance reason Updating constraints Temporal constraint, ratio constraint( 예제 ), semantic constraints Logic …

Network Computing Laboratory | 10 Korea Advanced Institute of Science and Technology Challenges of S 3 development (3/3) Error inspection and diagnosis is hard due to the long running feature and dynamicity of data Many factors affect the execution of the services Data rate Data values Running time Due to the “time” factor, it becomes harder to find an error & verify the correction of the error A program may have state which is built over time It may take some time to browse the program state which leads to error E.g. a service may start to misbehave after 15 hours of execution Unpredictable, Thus hard to handle exceptional situations

Network Computing Laboratory | 11 Korea Advanced Institute of Science and Technology Stream Synthesizing Services (S 3 ) Stream Synthesizing Services Receives data from multiple sources, exhibits multiple merging stages, and then generates final outputs Data from each source goes through multiple processing steps As time passes (the steps may require different time window) As more data arrives (the steps may require different tuple window) Modules may require data from other sources (or modules)

Network Computing Laboratory | 12 Korea Advanced Institute of Science and Technology Approach Provide a programming model and an execution framework to facilitate the development of S 3 s Let developers to concentrate on service logic by Delegating primitive data processing tasks to the framework Interface which abstracts input data streams Hiding complex execution flow Event-based programming Enabling easy verification of the logic Emulating data sources Marking certain moment of execution (?) Allow execution speed control (?) Programming Framework Programming model Runtime Error diagnosis

Network Computing Laboratory | 13 Korea Advanced Institute of Science and Technology The waterwheel model Waterwheel: logic segment which is triggered as stream arrives A stream creates event & event triggers the logic A service is composed of waterwheels Each waterwheel have different input streams Flexible event definition & management The constructed waterwheels run as streams are pushed from the sources Logic service Logic

Network Computing Laboratory | 14 Korea Advanced Institute of Science and Technology What to do Scenario refinement Realizing the programming model Design the interface which is used for input abstraction Define operations & constraints on multiple sources Define events & define how they could be expressed Define how the waterwheels can be organized into a program Design the runtime Data management Supporting operations Connection between waterwheels (outputs to inputs) Waterwheel scheduling Evaluating events of waterwheels Programming Framework Programming model Runtime Error diagnosis

Network Computing Laboratory | 15 Korea Advanced Institute of Science and Technology What to do Supporting error diagnosis Refine possible requirements Case study Build one or two services Programming Framework Programming model Runtime Error diagnosis

Network Computing Laboratory | 16 Korea Advanced Institute of Science and Technology Scenario fuel Engine temp Tire pressure My position Values from sensors Mobility Map Orientation, speed Aim target Scenario: Military logistics applications Guide an object (e.g. tank) strategically using context information 변화 율 계 산 Vehicle status diagnosis Position of external objects Raw stream pre-processing Complex execution flow Error inspection & diagnosis Black box 가 낫다