Event Based Systems Short intro on Trident Dr. Emanuel Onica Faculty of Computer Science, Alexandru Ioan Cuza University of Iaşi

Trident – general description Offers a higher level of abstraction on top of Storm topologies + some extras: transactional processing - tuples are sent in batches and a successful/failed send send is detected at batch level

Trident – general description state management – a fault-tolerant state is maintained internally in the topology holding information about the batch processing exactly once processing – every batch receives an ID and is processed exactly once by an operator (in case of fail and resending, the operators which already processed it will skip re-processing) filtering and aggregation abstractions

Trident – topology and spouts The Trident topology: TridentTopology topology = new TridentTopology(); Spouts in Trident: Implement the ITridentSpout interface Three types: Non-transactional – same tuple can appear in different batches Transactional – no batches overlapping, and a batch with a given ID will always have the same tuples Opaque – no batches overlapping, but batches composition might vary in time (i.e., when re-emitting failed batches tuples can be split differently between these)

Trident – topology and spouts The simplest spout – for testing purposes: FeederBatchSpout spout = new FeederBatchSpout (ImmutableList.of (“field1",...,“fieldN")); Emitting tuples: spout.feed(ImmutableList.of(new Values("val1", ..., "valN"))); A topology is created through chaining the operations. Some example: TridentState state = topology.newStream(“stream_name", spout).each(new Fields(“field1", “field2"),new Function(), new Fields(“field3")).groupBy(new Fields(“field3")) .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count"));

Trident – the topology chain The data stream is created using newStream("stream_name",spout_instance) . Various operations can further be applied on a stream: each(new Values("selector1","selector2",...,"selectorN"),new SomeFilter("some_args")) //selects a subset of fields and applies the isKeep() method in a filter extending BaseFilter to filter tuples based on the selection; the entire tuple will be forwarded or not each(new Values("selector1","selector2",...,"selectorN"),new SomeFunction("some_args"),new Values("appended1","appended2",...,"appendedN")) //selects a subset of fields and applies the execute() method in a function extending BaseFunction to apply a function on the tuples based on the selection; the entire tuple will be forwarded with possible extra values appended

Trident – the topology chain aggregate(new AggregatePrimitive(),new Fields("aggregated_field")) //aggregates the incoming tuples in a new field using a specific primitive for aggregation (e.g., Count()) groupBy(new Fields(“field_name")) //similar to groupBy in the original Storm topology (shuffle() also exists) parallelismHint(number) //sets the degree of parallelism on the topology up to the first partitioning operation (e.g., groupBy()/shuffle())